xbbg 1.8.0

Added

• Gap-aware subscription delivery (#271). Python, Node, and Rust support opt-in drop_oldest deltas with exact local-eviction counts, and delivery="state"/delivery: 'state' for fair, conflated current images. Delivery envelopes preserve gaps across Arrow and dataframe backends; state rows expose continuity validity and ordered invalidation/recovery without building Arrow on incoming updates or scanning quiet topics. Python converts delivery backends on the consuming event loop, never on native bridge threads. Rust SubscriptionDeliveryMode, SubscriptionGap, and SubscriptionDelivery are #[non_exhaustive].
• Transactional marked reads and drains. Native source receipts survive conversion failure, cancellation, and closing views until publication. Frozen drains retain complete epoch/gap/error ordering for backend retry, without retaining exported Arrow buffers.
• Offline native verification and benchmarks. Feature-gated SDK TestUtil drivers exercise real CPython, V8, and Bun paths; CI excludes the drivers from shipped artifacts. Benchmarks compare default queues with a frozen v1.7.0 source and distinguish full adapters from bare overwrite primitives, with privacy-safe provenance.
• Field metadata refusal diagnostics (#369). FieldInfo results append Bloomberg's field id for each answered field and always retain per-request-id fieldError details under xbbg.field_errors, including on empty results. Native Python tables/batches expose field_errors, pandas preserves xbbg_field_errors, and Node and bounded tool results expose fieldErrors.
• Configurable security key column (#371). An engine setting names the column that holds each row's security: "ticker" (default) or "security", after Bloomberg's securityData.security element. Set it with Python configure(key_column=...)/Engine(key_column=...), Node keyColumn, Rust EngineConfig::key_column (KeyColumn), or MCP XBBG_MCP_KEY_COLUMN. It covers reference, historical, bulk, intraday bar and tick, and BQL results, the recipes and resolvers built on them, and xbbg.testing mock responses; the ext pivot and earnings-header helpers accept either name. Subscription rows keep topic.
• Offline decode and read-latency benchmarks. request_decode_replay replays SDK TestUtil BDH, BDP and BDS responses through the production extractors in every output mode, and subscription_read_latency measures per-update enqueue-to-RecordBatch latency for each read limit under quiet, burst and sustained schedules, both without Bloomberg access.
• Free-threaded CPython wheels (beta, #370). Releases and CI build cp314t/cp315t wheels for Linux x86_64 (manylinux_2_28), Windows x64 and macOS arm64, and the package declares Free Threading :: 2 - Beta. pyarrow, pandas and narwhals are supported there; polars, duckdb and xbbg-langgraph are not, because they or their dependencies publish no free-threaded wheels. An advisory CI job runs the offline suite on 3.14t/3.15t and fails if an import re-enables the GIL. See docs/free-threading.md.
• Bounded native Arrow access (#370). ArrowColumn.logical_nbytes sizes a slice without copying it, ArrowTable/ArrowRecordBatch.metadata_sizes() and .metadata_value(key, max_bytes=None) read schema metadata without copying every value, and ArrowTable.from_pydict() builds a table column-wise.
• Python delivery-latency benchmark. py-xbbg/benchmarks/bench_python_delivery_latency.py measures publish-to-delivery latency for an asyncio consumer on release test-support builds and reports event-loop lag separately; its provenance omits local paths.
• Concurrency testing for free-threaded Python.
  • The free-threaded CI job also runs the offline suite under pytest-run-parallel, with each test body in 8 threads at once (pixi run -e py314t test-parallel locally). Async tests and tests that change process-wide state run in one thread.
  • New stress tests check exact results from many threads at once: one subscription driver, one request driver, subscribe/unsubscribe churn, the first import of the native extension, and configure()/shutdown()/reset(). CI runs them, and the native teardown tests, against the test-support build.
  • A nightly ThreadSanitizer workflow runs the xbbg-async and xbbg-arrow tests, and the native Python tests on a ThreadSanitizer-built free-threaded CPython.
  • docs/free-threading.md lists the settings that apply process-wide and how middleware must behave when several threads run their own event loops.

Deprecated

• The ticker key column (#371). Once identifiers are case-folded it collides with Bloomberg's TICKER field. Python results that use it emit a FutureWarning, Node emits one DeprecationWarning per process (code XBBG_TICKER_KEY_COLUMN), and the MCP server logs a warning at startup. The next major release makes security the default and removes ticker.

Changed

• BREAKING (Rust): SubscribeRequest includes delivery, and OverflowPolicy adds DropOldest. Update full struct literals and exhaustive policy matches. Default sparse-delta behavior and ordinary Python/Node iteration remain unchanged.
• LangGraph snapshots explicitly select a strict policy. Omitting overflow policy selects drop_newest independently of engine defaults; callers can still request block. Snapshot schemas do not advertise lossy policies they cannot represent.
• BREAKING (raw metadata): compact field exceptions (#368). xbbg.field_exceptions now writes a version-1 dictionary envelope with unique full exception payloads and per-security occurrence indices. The engine interns repeated text during extraction and remaps dictionaries by value when merging shards or the requests of a composed recipe; each streamed chunk carries its own dictionary. Public Python/Node exception accessors retain their expanded map shape, and Python decodes it without holding the GIL. Readers continue to accept historical stored maps and name the unsupported version or malformed member they reject. Bounded tool adapters (MCP, Python and Node LangGraph) expand only retained diagnostics and omit unreadable ones as unsupported_value instead of failing the result.
• BREAKING (Rust): shared diagnostic types live in xbbg_async::diagnostics. Import FieldExceptionMeta and METADATA_KEY_FIELD_EXCEPTIONS from that module instead of engine::state; it also provides the compact FieldExceptions codec and field-id refusal metadata.
• BREAKING (output shape): FieldInfo tables gain an id column. bflds() (with fields= or search_spec=), Node fieldInfo(), and FieldInfoRequest results carry Bloomberg's field id (for example PR005) as a trailing fifth column. Code that unpacks these columns positionally must accept it.
• BREAKING (Rust): EngineConfig gains key_column. Full struct literals must set it or use ..Default::default().
• BREAKING (subscriptions): Arrow reads combine already-queued updates, up to 16 per batch, by default. subscription_flush_threshold (Python/Rust) and subscriptionFlushThreshold (Node) default to 16 instead of 1, and flush_threshold/flushThreshold is documented as a maximum. A read still returns as soon as the first update arrives and adds only updates already waiting, so quiet streams keep one-row batches at unchanged latency. One-row batches cost about 3 µs of fixed work per update and, under bursts, let the default 256-update queue overflow and end the stream under drop_newest; in the offline read benchmark a 2,000-update burst drains at about 4 µs median latency instead of 1.3–1.7 ms. Code that assumes one row per Arrow batch, delivery or drain should iterate rows or pass flush_threshold=1/flushThreshold: 1. Scalar tick iteration is unchanged, and LangGraph snapshots keep counting one update per read.
• Long-to-wide pivots keep typed values. ext_pivot_to_wide and Rust pivot_to_wide gather each field column from the value column, so numeric and temporal values keep their Arrow type instead of being rejected; string input produces the same output as before.
• Faster request decoding. Extractors resolve column slots, cached datatypes and interned element names once per request, and scan each securityData entry once instead of probing usually-absent diagnostic names. In offline replay, BDH long_metadata drops from 74 to 54 ms and bulk data from 11.0 to 7.6 ms for the same output; BDP drops from 5.5 to 1.1 ms, mostly because absent-name probes take the SDK's slow path in TestUtil messages (unmeasured on live wire messages). The default long output still formats values as text and changes little.
• Leaner subscription and binding hot paths. Shared feeds find metadata fields and reconcile delayed status once per layout instead of per tick (about 20% faster strict-delta and 30% faster conflated-state publishing in offline replay). Node scalar ticks cross N-API as one column batch with direct BigInts instead of per-tick padded arrays and decimal strings, and Polars results arrive as one native Arrow IPC file instead of a JS decode and re-encode. Native Python tables release the GIL for row work, filter equality with Arrow kernels, compact flat arrays in one copy, and convert to Python objects column by column without per-cell module imports. Recipes resolve typed column views once per batch.
• Async results and exceptions are built off the async-bridge workers (#370). Every async Python API now yields owned native results that pyo3-async-runtimes converts once on its completion thread, exceptions keep all request, SDK-ID and data-loss attributes but are built lazily, and the interpreter-finalization gate no longer attaches. Bridge workers never wait for the GIL, so a busy Python thread no longer stalls unrelated requests, recipes or subscription reads. Marked-delivery receipts are created on that completion thread, and a panic while converting a result raises RustPanic instead of leaving the awaitable pending.
• Data-sized native work releases the GIL (#370). Native Arrow sort, filter, compaction, batch concatenation, presence scans, BDH presentation, generic BQR reshaping and Python row/column table construction; diagnostics metadata of at least 64 KiB; the caller-sized ext_* kernels (futures candidates, contract and ticker filters, column renames, currency and level calculations); field/schema cache clearing, invalidation and list_cached_schemas(); and signal_shutdown(). The ext list kernels no longer copy surviving strings, and the embedded exchange-rule table is parsed once at import.
• BQR shaping and BDH presentation run off the event loop (#370). abqr()/bqr() quote shaping and the abdh()/bdh() presentation options (show_date, sort, PERIODIC/BOTH) run as native kernels on the runtime's blocking pool, so the asyncio loop and other Python threads keep running.
• One BQR shaper and generic reshape for every surface (#370). blp.abqr()/bqr() use the Rust shaper behind xbbg.ext.abqr() and Node bqr(), and ArrowTable.reshape_bqr_generic() runs its reshape. Python's generic (path) output now keeps schema metadata (request IDs, EIDs, pandas attrs); column types of extractor output are unchanged. A field without text keeps the value_num Arrow type, so the shaper no longer turns integers into float64 or drops dates and unsigned values, and reshape_bqr_generic() no longer widens integers to int64. A non-string value_str raises ValueError instead of being misread. The missing-broker-attribution RuntimeError is unchanged.
• Engine teardown no longer runs on Python threads (#370). Releasing the last reference to an engine, or to a subscription or delivery receipt that outlives its engine, hands session stop and runtime shutdown to a native teardown thread; stop still waits for SDK callbacks before destroy. shutdown(), reset() and configure() still return after their engine's teardown, now without holding the GIL, and interpreter exit waits for any outstanding teardown.
• Conversions without pyarrow are column-wise (#370). pandas output, the Polars < 1.3 path and convert_ccy input build columns instead of one dict per row, with the same dtypes and attrs. ArrowTable.to_polars() uses the xbbg.backend conversion, keeping native chunks instead of falling back to rows. Currency adjustment reuses native Arrow inputs without FFI and drains foreign Arrow streams in one detached step on the calling thread, so producer release callbacks never run on async workers.
• SDK discovery no longer imports blpapi (#370). The package is located with find_spec and its version read from distribution metadata, so discovery runs no third-party code. get_sdk_info() is thread-safe and never publishes a result made stale by set_sdk_path()/clear_sdk_path(); a broken blpapi install is listed as a source, and the library search still filters it out.
• Free-threading groundwork (#370). PyEngine and PySubscription are frozen classes, clippy forbids Python::attach/try_attach in the Python bindings, the GIL pin moves into a pixi gil feature with a new py314t environment, and the pixi duckdb floor matches the package's >=1.5.1.
• Faster package import and first request. import xbbg takes about 30 ms, and a new process loads neither importlib.metadata, importlib.abc nor pandas before its first result.
  • xbbg.__version__ and the native-extension guard read a _version.py that setuptools-scm writes at build time, instead of importing importlib.metadata (about 80 ms).
  • SDK discovery on the import path no longer reads distribution metadata; only get_sdk_info() reads the blpapi version.
  • The guard no longer imports importlib.abc (25–50 ms).
  • The Bloomberg SDK library is prepared when xbbg._core first loads instead of on import xbbg. On Windows, the highest-priority library (manual path, blpapi wheel, Terminal DAPI, BLPAPI_ROOT) is loaded by full path, so the process binds that copy and only that copy, and set_sdk_path() called before first use takes effect.
  • Converting results to pyarrow, including the default Narwhals output, reads the Arrow C stream directly instead of calling pa.table, which imported pandas (about 300 ms).
  • The engine opens its request sessions, its subscription sessions and each worker's warm-up services concurrently. On a Desktop API connection, engine startup drops from about 250 ms to 80 ms.
  • Async calls start a missing default engine on a worker thread instead of blocking the event loop.
  • Requests without extra keyword arguments skip the request-schema lookup.
• Benchmark reports can be committed as written. Provenance keeps package versions, the xbbg commit and build profile, and the Python version and GIL mode. It drops paths, host OS and CPU details, compiler flags, environment variables and binary hashes. Reports carry the UTC date instead of a local timestamp, and archive files are numbered within the day. Competitor reports save the exception type instead of Bloomberg's error text, and name a mismatched row and field without its values.
• The free-threaded dev environment builds on Zen 5 Windows hosts. pixi.toml caps Rust below 1.99: rustc 1.99.1 fails to compile libm with -C target-cpu=native there (Missing .seh_endepilogue), and the py314t environment, solved on its own, had picked it up. The lower bound is unchanged.

Fixed

• xbbg.testing.mock_engine works without a Bloomberg connection. bdp() and bdh() resolved field types while building the request, before any middleware ran, so a mocked call still started a real engine (and waited out its connection retries offline). Field types are now resolved in the request terminal, so a middleware that answers the request, such as the mock engine or a cache, never starts an engine or queries //blp/apiflds. Middleware that inspects context.params.field_types sees only caller overrides; RequestContext.field_type_default names the default the terminal will apply.
• convert_ccy() accepts xbbg's default Narwhals results without a warning. Narwhals frames are unwrapped first, so Arrow-backed results take the direct Arrow path instead of reaching Narwhals' pandas check.
• Live tests: the invalid-subscription test bounds its wait, asserts the BlpRequestError and shuts down its engine, so a lost failure fails the test instead of hanging the suite; JS live tests skip saved screens and search domains a login does not have, as the Python suite does.
• The shaping thread-progress test no longer races the kernel. It repeats until the helper thread reaches the event loop while a shaping kernel is still running, instead of failing when the kernel finishes before the OS schedules the helper, as on free-threaded macOS runners.
• Debug builds no longer overflow the async-bridge stack. Unoptimized builds of the Python extension, such as pixi run install produces, crashed with a stack overflow in xbbg.ext.turnover(). pyo3-async-runtimes and the completion gate wrap every async binding in several generic futures, and without optimization each of their frames held copies of the binding future: 280–560 KB per frame for requests and recipes, and about 1.9 MB of a bridge worker's 2 MiB stack for turnover. The binding future is now boxed once before it is wrapped. Release builds were not affected.
• Newer CPython releases resolve to current xbbg, not an older release line. requires-python drops its <3.16 upper bound. pip skips every release whose Requires-Python excludes the running interpreter and installs the newest one that admits it, so a CPython newer than the cap got the newest release without one: 0.7.7 (2022), as Python 3.15 users did until 1.6.0 shipped cp315 wheels, and 0.12.4 from the 0.x line once that release dropped its own cap. pip now selects the current release, building it from source when no wheel exists. Supported versions are unchanged.
• Field validation and type lookup retain refused-id reasons (#369). Validation recognizes Bloomberg field ids as well as canonical mnemonics, lenient warnings and strict errors include Bloomberg's refusal details, and answered fields are also cached under their field id. Subscription type lookup reports each refused id once per engine instead of silently treating it as ordinary missing metadata.
• macOS CI checks the subscription-delivery benchmark. The benchmark's production receiver is boxed during untimed queue setup so the heterogeneous receiver enum does not trigger large_enum_variant; per-update work is unchanged.
• Installed-wheel CI registers native async fixtures explicitly. The SDK-backed subscription fixture works with pytest-asyncio's strict mode when tests are copied outside the source tree, rather than depending on the repository's asyncio_mode="auto" configuration.
• GIL-release tests no longer fail intermittently. The probe behind them repeats the probed call until its helper thread runs, for up to a second, instead of judging a single call. A native kernel that released the GIL but returned before the operating system woke the helper, as filter_eq and has_any_value did on CI runners, was reported as holding it; a call that keeps the GIL still never lets the helper run.
• Concurrent first use of the native extension (#370). Threads that first touch xbbg._core, including through from xbbg import _core, while another thread is importing it now wait for that import instead of raising "Recursive import of _core detected".
• Request middleware changes no longer affect running requests (#370). Each request snapshots the middleware chain, so removing a middleware while another request is awaiting no longer skips or replaces the rest of that request's chain.
• Field types, element routing and schema parsing follow the active engine (#370). The process-wide Python field-type, valid-element and parsed-schema caches are gone. A default type returned during a transient lookup failure is no longer pinned and sent as an override after the engine learns the real type, and scoped engines, invalidate_schema() and clear_schema_cache() take effect immediately.
• Caller-owned results (#370). ta_study_params(), get_sdk_info() and ServiceSchema.from_json() return objects that callers can mutate without changing later results.
• LangGraph native results are bounded before copying (#370). With an xbbg release that provides the bounded native getters, cell sizes are checked before a cell is materialized, and metadata sizes before any value is fetched, so oversized cells or metadata no longer cost a full copy before the result limits apply. With older xbbg releases, which the adapter's declared floor still allows, it copies them as before.
• Runtime backend floors match the declared extras. The pandas and duckdb backend checks now require 2.2.2 and 1.5.1, as pyproject.toml declares.
• Engine.with_config() releases its config before connecting (#370). Reading or setting that config from another thread during startup no longer raises "Already borrowed".
• ext_generate_futures_candidates bounds count (#370). Counts above 10,000 raise ValueError (Node: InvalidArg) instead of preallocating whatever the caller passed, and a corrupt embedded exchange-rule table fails loudly instead of meaning "no rule".
• ColumnSet::finish_with_order appends unlisted columns in first-write order (Rust), as documented. Removing a listed column no longer moves the last column into its place.
• Source builds stay clear of rustc's recursion_depth_exceeding_limit lint. Proving Send for the Python request futures exceeded the default recursion depth. Nightly rustc warns about this (rust-lang/rust#159228) and will make it a hard error, so the Python extension raises its recursion limit to 256.
• Patched MCPB build tooling. The private build tools' lock moves node-forge from 1.4.0 to 1.4.2, inside the ^1.3.2 range that @anthropic-ai/mcpb 2.1.2 requests, fixing the eleven advisories for node-forge 1.4.0 and earlier, among them the X.509 basicConstraints bypass in GHSA-5x38-2vff-x55p, the PKCS#12 MAC forgery in GHSA-jc66-7q78-782g, and the AES-GCM authentication bypass in GHSA-5cwg-6vw7-rj6p. The tools only build and inspect the bundle in the release workflow, and node-forge is not part of it.
• Patched JavaScript dev tooling. Both JavaScript packages override minimatch to 11.0.4, fixing the denial of service through glob-part amplification in GHSA-g3hv-75wg-m249, which has no fixed 10.x release. ESLint 10 and typescript-eslint still request minimatch 10 but use only APIs that 11 keeps. The @xbbg/core lock also moves knip's zod from 4.6.5 to 4.6.7, fixing GHSA-mrhx-c7xc-m3w5. These are development dependencies and are not part of the published packages.

Full Changelog: v1.7.0...v1.8.0