graphsense-lib 2.16.4

[2.16.4] - 2026-09-22

Library

Added

• ZEC transactions now carry a per-pool breakdown of their shielded value. sapling_value_balance, orchard_value_balance and ironwood_value_balance are emitted on every transaction record in zatoshi, positive when value left that pool. They exist because the pool is not recoverable downstream: every shielded input and output is address-less and identical once emitted, and the bundle that produced it is not stored anywhere, so attributing a shielded row to Sapling, Orchard or Ironwood after the fact would need another node read. Two of three pools would be no use either — the remaining rows would stay ambiguous between Sprout and Sapling — so Sapling is reported alongside the two new ones. Nothing consumes them yet: the parquet sink builds its table with pa.Table.from_pylist(mapping=data, schema=...) and logs uncovered fields at debug, and the Cassandra sink prepares its insert from the table's own columns, so both ignore the three keys until the parquet schema declares them. The Cassandra raw schema does not change and the graph output is unaffected.
• spark/build.sbt is now the single source of truth for the Scala job's
  dependencies. DEFAULT_SCALA_JOB_PACKAGES mirrors its compile-scope
  entries so the slim artifact can pass them to spark-submit --packages,
  and nothing enforced that mirroring — the two drifted silently whenever the
  sbt build changed, and a slim run would only find out as a
  NoClassDefFoundError on a cluster. scripts/check_spark_packages.py parses
  the declaration block and either fails (make check-spark-packages) or
  regenerates the lists (make sync-spark-packages), wired into pre-commit
  exactly like update-api-version. It caught a real drift on its first run:
  bcprov-jdk18on had been added to the sbt build and never mirrored. If the
  declaration shape ever changes, the parser exits with an error rather than
  silently matching nothing.
• --exclude-packages on the slim submit path. --packages resolves
  transitively, so an artifact the sbt build excludes came straight back:
  without this the slim path re-resolved bcprov-jdk15on through
  org.web3j:utils, undoing the BouncyCastle swap. The new
  DEFAULT_SCALA_JOB_EXCLUDES is generated from the build's exclude(...)
  clauses by the same script, and is only emitted alongside --packages.

Changed

• DEFAULT_SCALA_JOB_PACKAGES drops org.web3j:core, tracking the same
  removal in spark/build.sbt (see spark/CHANGELOG.md): nothing in the Scala
  job imports it. This list only feeds the slim artifact's --packages;
  production runs fat, where the assembly jar is authoritative.
• The pre-commit gate is now a fast subset; CI runs everything. The hook
  ran the full suite with coverage and no marker filter, so a commit cost ~236s
  — and more than half of that was one test, which resolved a bogus hostname
  (redis://invalid-host) and paid whatever the local resolver charged for a
  dead name: 120s on a box behind a blackholing DNS server, instant on CI. With
  that fixed, make test-fast drops coverage (nothing reads the local report),
  the slow marks and the four testcontainer directories (tests/web,
  tests/db, tests/tagstore, tests/integration — ~80s of a ~95s suite,
  mostly container start/stop, since the fixtures are session-scoped and the
  cost is all-or-nothing). A typical commit is now ~33s / 1307 tests.
  make test-rust joins the hook path-gated on rust/** (0.06s warm) and the
  codegen hook is no longer always_run: 7s of Docker plus the Java
  openapi-generator now fires only for src/graphsenselib/web/, the client
  templates and the compat patches, which are its only inputs. The hook prints
  as pytest: fast subset only (no slow/container tests) rather than
  pytest-check, which read exactly like a full-suite pass. semver-check and
  ensure-env drop their stages: [pre-commit, pre-push]: make dev runs
  pre-commit install, which installs the pre-commit hook only, so the
  pre-push half had never fired. README, CLAUDE.md and
  .github/copilot-instructions.md document what each target covers and what
  it costs -- the latter two had claimed make test-ci skips slow tests.
• CI no longer skips slow tests, and the Spark job no longer path-filters.
  Every CI pytest invocation passed -m "not slow", so the three slow-marked
  tests ran only on a developer's machine — and the fast local gate would have
  left them running nowhere at all. make test-ci now runs the suite in full
  (2326 tests, ~104s), and so does make test-with-base-dependencies-ci (2046
  passed / 26 skipped, ~78s) -- no CI target filters slow any more, so nothing
  can be skipped everywhere at once. spark_tests.yml likewise ran on a path list covering
  spark/, the delta updater and transformed_*.sql; since sbt is not run
  locally at all, that list was the only thing standing between a new file
  feeding the delta/Spark contract and an unnoticed drift, so it now runs on
  every push and PR.
• Dependency bumps: the uv-minor-patch group (22 packages), setuptools
  84.0.0, the docker-images group (uv 0.12.9, a new temurin 11 digest) and
  arrow 59.3.0 in rust/gs_clustering. Notable inside the group: fastapi
  0.136.3 -> 0.141.1, sqlmodel 0.0.38 -> 0.0.42, rapidyaml 0.13.0 -> 0.15.2,
  redis 8.0.0 -> 8.1.0, cassandra-driver 3.30.0 -> 3.30.1, ty 0.0.73 -> 0.0.79.
  ruff is deliberately held at 0.15.17: 0.16 widens the default rule set
  and reports 3442 findings (2741 autofixable) on a tree that is clean today,
  which is what failed every job on the group's PR -- the tests passed there.
  Adopting it is a separate change, either the autofixes or an explicit
  lint.select.
• A second round of bumps taken from the open Dependabot PRs: anyio 4.13.0
  -> 4.14.2 (#170), the uv build image 0.12.9 -> 0.12.16 (#165) and arrow
  59.3.0 -> 60.0.0 in rust/gs_clustering (#164). All seven open Dependabot
  PRs are based on master, which is 32 commits behind this release window, so
  none could be merged as it stands: develop already carries most of the
  uv-minor-patch group, and the "from" versions in those PR tables are
  master's, not this branch's. The three above were applied by hand instead, as
  in 2.16.3, and verified here rather than on the PR — the full Python suite
  (2397 passed, 1 skipped, containers and slow included), ruff, ty and
  cargo test (22 passed) against the updated locks. arrow 60 is a major
  bump, but confined to the clustering extension: it adds arrow-cmp, moves
  the transitive atoi to 3.1.0, and needed no source change in the crate.
• The github-actions group (#167) and the clients/python group (#166) were
  applied as well. #167 pins docker/setup-buildx-action to v4.4.0 and
  docker/build-push-action to v7.4.0 across the three image workflows and
  moves actions/setup-java v5 -> v6 in the three Scala ones; both SHAs were
  re-verified against their tags before pinning. Its red spark-test job was
  not the bump: the failure is a JVM OutOfMemoryError: Java heap space in
  org.graphsense.utxo.TransformationTest, spark_tests.yml is green on every
  recent develop run, the sbt heap options are identical on both branches,
  and re-running that exact job on that exact commit passed. setup-java v6
  changes distribution handling, caching and the Zulu/Azul API — nothing that
  touches heap.
• #166 was not a lockfile-only bump, and hid a real bug. click 8.4.2 ->
  8.5.0 moved get_text_stream behind a module-level __getattr__ that raises
  a DeprecationWarning (removal in Click 9.0), and the CLI resolved its
  stderr through it on every write. The deprecation hook resolves that stream
  inside a try/except Exception, so under the client's -W error suite the
  warning became the exception the hook swallows and every "endpoint is
  deprecated" message vanished —
  test_deprecation_header_triggers_stderr_warning is what caught it, and no
  amount of bumping ty fixed the matching call-non-callable diagnostic,
  because the annotation really is -> object. graphsense/cli/context.py now
  resolves sys.stderr directly, which is what the helper returned anyway and
  what the hook already does for stream=None. The client is not
  re-released in this window: on click 8.x the bug is invisible under Python's
  default warning filters, so the fix rides the next webapi-v* release.
  Verified with the client's own gate — lint, type-check, test-ci (122
  passed, -W error) and test-compat.
• Two Dependabot PRs are deliberately left for their own change. fastmcp
  3.4.2 -> 4.0.3 (#169) is a major carrying mcp 1.28.1 -> 2.2.0, and it fails
  tests/mcp/test_server_integration.py::test_tool_surface_shape, where
  Tool.inputSchema is now the deprecated spelling of input_schema; it also
  crosses the DNS-rebinding host check that made GS_MCP_ALLOWED_HOSTS
  necessary in the first place, so it wants a deployment check rather than a
  same-day bugfix slot. ruff 0.16.7 stays held for the reason above, and that
  keeps the rest of #168 with it: filelock 3.32.6, grpcio and
  grpcio-tools 1.84.0, psycopg2-binary 2.9.13, build 1.6.1 and ty
  0.0.81.

Fixed

• Tagstore: 0x hex addresses are lowercased on every network, not just
  ETH. A token tag without an explicit network (e.g. currency: USDT)
  gets the currency as its network, which skipped the ETH-only lowercasing, so
  checksummed addresses were stored verbatim and never matched the exact,
  lowercase lookup the API does. The insert path and the validation
  duplicate-key check now share one normalizer (tagpack.utils. normalize_tag_address), and the tagstore lookups (single and batch
  subject-id queries, user-reported tags) normalize incoming 0x ids the same
  way; batch results stay keyed by the ids the caller passed. Validation now
  warns when a tag's network is a token rather than a chain. Existing rows are
  not rewritten; fix them with
  UPDATE tag SET identifier = lower(identifier) WHERE identifier ~ '^0x[0-9a-fA-F]+$' AND identifier <> lower(identifier);
  after removing rows whose lowercase twin already exists, or by reimporting
  the affected tagpacks with --force (which drops their old rows).
• UTXO ingest no longer aborts on transaction versions >= 2^31 from
  Bitcoin Core >= 28.0. Core 28.0 changed the tx version to uint32
  (bitcoin/bitcoin#29325), so its RPC reports the two negative-version txs in
  BTC block 256818 as 2591798512 and 2187681472. The raw transaction.version
  column is int, and the bind failed ('i' format requires -2147483648 <= number <= 2147483647), stopping any BTC ingest that reaches that block. The
  parser now wraps such versions to the signed int32 older nodes returned, so
  the stored value matches keyspaces ingested from pre-28 nodes and no schema
  change is needed.
• transformation raw-to-transformed --local keeps a configured
  local[N] master. The flag used to force spark.master=local[*]
  unconditionally, so a spark.master: local[1] in the spark_config
  profile was silently widened to all cores. A local master from the config
  is now kept; a non-local master (e.g. spark://…) is still replaced by
  local[*], now with a warning.
• Every ArrowInvalid from the delta sink took the shrink-the-row-group
  retry, not just the two it was meant for. The guard read
  if "large_binary" in ste or "named input expected length": -- the second
  operand is a bare string literal, so the condition is always true and the
  else: raise e beneath it was dead code. An unrelated Arrow error therefore
  spent up to 20 attempts halving max_row_group_size before surfacing as
  "Could not write delta-file after 20 attempts", which names neither the real
  error nor the file. Found by the ty bump below, which is the entire reason
  that bump is worth taking.
• Running the suite without a Docker daemon now skips the container-backed
  tests instead of erroring ~1000 times. #160 made a daemon-less machine
  collect, but every test requesting the Cassandra/Postgres/Redis fixtures
  still failed at setup with a full urllib3 traceback each. The session
  fixtures now ping the daemon once (5s timeout, so a dead DOCKER_HOST is
  cheap) and skip with one reason plus a summary banner when it does not answer.
  On CI (CI set) an unreachable daemon stays a hard error -- skipping there
  would turn a broken runner green while covering nothing. (#161)
• ZEC Orchard and Ironwood value balances are accounted as shielded inputs and outputs. The parser recorded shielded flows from Sprout vjoinsplit and Sapling valueBalance only; the NU5 orchard and NU6.3 ironwood bundles were blacklisted wholesale, so every transparent↔Orchard and transparent↔Ironwood flow was invisible and input_value, output_value and fee were computed from the transparent, Sprout and Sapling sides alone. One code path now loops over both pools and reads the bundle's valueBalanceZat, which is already in zatoshi and must not go through _btc_to_satoshi; the float valueBalance beside it is deliberately not read, because Zebra renders it with a helper named lossy_zec that its own documentation says must not be used for consensus-critical calculations (zebra-rpc/src/methods/types/zec.rs). The sign convention is Sapling's — positive means value left the pool, so it becomes a shielded input; negative becomes a shielded output of the negated amount. A fully-shielded NU5 transaction makes the size of the old error concrete: bd84ece1a1f9930085768a880d99445ee1ee13686c2e5979b30d920e90866517 at height 1,687,194 moves +89,000 zat out of Sapling and −88,000 zat into Orchard for a 1,000 zat fee, where the parser previously saw the Sapling leg alone and reported a fee 89× too large. This needs a re-ingest from NU5 activation (height 1,687,104) and a re-transform to take effect, since shielded I/O is synthesized at ingest time and blocks already in the lake keep their old values.
• Two consequences of that re-transform are worth stating plainly, because neither is a simple value refresh. Existing relation values shift: totalInput is the denominator in inValue / totalInput * outValue, and it is summed over all inputs with no address filter (Transformator.plainAddressRelations, mirrored by get_total_input_sum in the delta updater), so adding a shielded leg lowers estimatedValue on the transparent relations of a mixed transaction rather than only adding rows. And cluster membership can change: is_coinjoin reads input_count, output_count and the output-value histogram, and shielded I/O is appended before those counts are taken, so a new leg can flip the coinjoin flag — which computeAddressCluster consumes. Treat the re-transform as a full cluster rebuild with possible attribution shifts. This is the behaviour Sprout and Sapling legs have always had, not a new hazard, but it now applies to far more transactions.
• A third point is about reading the result rather than producing it: a positive shielded balance is usually not an unshielding. Most often it is the fee of a fully-shielded transaction, paid out of the pool. Mapping it to a shielded input is right for value conservation and matches what Sapling has always done, but a shielded input must not be read as a z→t transfer.
• No spark/ change accompanies this. computeRegularInputs and computeRegularOutputs filter size(col("input.address")) === 1, so address-less shielded I/O never reaches the address graph, while plainAddressRelations explodes every input without that filter and therefore picks the new legs up on its own; the delta updater mirrors both. There is no Scala path that reads shielded I/O by type — the Scala tree contains no occurrence of shielded at all.
• Sapling now reads its integer valueBalanceZat too, so all three pools take the same field rather than leaving one on a float. On values this is a no-op — round(float * 1e8) is exact at ZEC supply scale, and 665,685 mainnet transactions sampled across the whole Sapling range agreed on every one — so it cannot disturb existing Sapling data. It was worth confirming rather than assuming, because both Sapling keys are Option fields in Zebra's RPC types and a missing one would drop the shielded row silently rather than raise: the two are written from a single construction site in Zebra (both Some, or both None) and from adjacent pushKV calls inside one guard in zcashd, and a scan of 112,958 blocks — eleven contiguous 10,000-block windows from Sapling activation to the tip, plus a stride sweep of the whole range — found the pair present together on 665,685 of 665,685 transactions, across transaction versions 4, 5 and 6, with no case of the float appearing alone. valueBalance and valueBalanceZat therefore exchange places between _TX_KNOWN_KEYS and _TX_BLACKLIST.
• The bundle is validated strictly rather than read permissively, so the next pool Zcash adds fails loudly instead of being silently dropped. validate_rpc_fields does not recurse, so the bundle, each of its actions and its flags object are enumerated at their own level. orchard and ironwood move into the transaction's known keys as optional entries — validation works by set difference, so an absent known key never raises — deliberately without a transaction-version gate: Zebra emits orchard on every transaction including pre-NU5 v1–v4 ones, where it is present but empty — confirmed on 328,134 NU6.3 transactions and on pre-NU5 transactions of all four versions down to block 1 — while zcashd gates it on nVersion >= ZIP225 (v5) and has no Ironwood field at all, so any version gate would be wrong against one node or the other. The transaction-level valueBalanceZat is read as Sapling's own balance and is separate from the identically-named key inside each bundle, which is read per pool; the two share a name but not a meaning, and the transaction-level float valueBalance beside them is blacklisted. flags also lists enableCrossAddress in advance: ZIP-229 defines that third bit and Zebra already carries it in its consensus types (zebra-chain/src/orchard/shielded_data.rs), but its RPC OrchardFlags struct exposes only enableSpends and enableOutputs, so no node emits it today — listing it now keeps the alert for a genuinely unexpected fourth flag while removing a break that would otherwise hit every Ironwood transaction at once the moment Zebra adds the serde field.

Removed

• graphsenselib.utils.pipeline and its tests. Nothing has imported the
  multiprocessing Pipeline since the account ingester was rewritten, and its
  tests failed on every spawn platform (macOS, Windows): the stages carried
  mp.Queue/mp.Event objects into child processes, which cannot be
  re-pickled. Linux forks, so only local macOS runs saw it. (#161)
• The exchange-rates coindesk provider (fetch/ingest) and
  graphsenselib.rates.coindesk. It called CoinDesk's old Bitcoin Price
  Index API (api.coindesk.com/v1/bpi), which CoinDesk has shut down: the
  hostname no longer resolves, so every run died with a bare
  NameResolutionError, and the API had stopped returning recent rates as
  far back as 2022. CoinDesk's surviving data API is the former CryptoCompare
  API (CoinDesk acquired CCData/CryptoCompare in 2024), so use
  exchange-rates cryptocompare with a cryptocompare_api_key from
  https://developers.coindesk.com/ instead. scripts/dev-ingest.sh no longer
  calls it.

What's Changed

• Detect 1inch swaps from transaction input by @frederik-raphael in #156
• test: build the testcontainers on first use by @mkzung in #160
• fix(mcp): default list-tool pagesize and advertise it in schemas by @frederik-raphael in #149
• fix(ingest): account ZEC Orchard and Ironwood value balances as shielded I/O by @steven3002 in #163

New Contributors

• @mkzung made their first contribution in #160
• @steven3002 made their first contribution in #163

Full Changelog: v2.16.3...v2.16.4