graphsense-lib 2.18.0

[2.18.0] - 2026-10-15

New read-only quality commands find where the tag summary hides an address's exchange attribution, actor clashes and garbled text in the tagstore; tied cluster tags are now picked deterministically.

Upgrade notes

Without any setting changed:

• Ties between equally confident cluster tags are decided deterministically (the tagpack whose id sorts first, then its first tag; stable across a re-insert of a pack) instead of by whichever row Postgres returns. Labels of tied clusters no longer change between requests, but can differ from what was shown before.
• Validation reports more, rejects nothing new: garbled or non-printable text and unknown same_as/related_actors ids are warnings; null characters fail as before. Actorpacks using the new context fields still validate and insert with older versions.
• Pack files are read as UTF-8 regardless of the locale; a rapidyaml parse failure falls back to PyYAML instead of aborting.
• POSTGRES_PORT is now honoured when tagpack-tool builds the URL from POSTGRES_*; environments that set it for something else connect to that port.
• Packs below a header.yaml are read by the fast YAML loader, which now gives PyYAML's result, so nothing changes for them except that an unquoted 0x… is text instead of a number (with either loader). Packs elsewhere, already read by the fast loader, can read differently than before where they relied on its old JSON behaviour (see Fixed).
• Address lookups answer 400 instead of 404 for an unindexed BTC, LTC, BCH (legacy) or ZEC address with a wrong checksum (see Fixed). Clients that treat 404 as "no transactions yet" now get an error for such typos.
• The actors of addresses and clusters list fewer actors: only those of actor-type tags (see Changed). An actor that only a mention or event tag carries no longer appears there, unless tag_summary_only_propagate_high_confidence_actors is turned off.
• An external backend that fails no longer turns shared answers into 500s. /stats, /capabilities, /search and cross-chain twin lists answer 200 without its networks and name it in X-Ikn-Backends-Declined; a request on one of its networks answers 503 with Retry-After: 30 instead of 500. A backend that does not answer a merge within the new external_backends.merge_timeout_s (default 5 s) is left out; proxied requests keep timeout_s.

Library

Added

• tagpack-tool tagpack validate --jobs N validates packs in parallel (0 = one process per CPU). With more than one job every pack is validated and every failure reported; with the default 1 the run stops at the first invalid pack, as before. Parsing is unchanged. On a synthetic repo of 300 packs of 5,000 tags each: 75 s with one job, 40 s with two, 22 s with four.
• tagpack-tool quality ranking-conflicts and quality actor-conflicts (read-only). ranking-conflicts runs every exchange-tagged address of a network through the same tag digest REST serves (direct tags plus the cluster's best definer, only_propagate_high_confidence_actors=True), lists the ones whose best_actor/best_label is not an exchange tag of theirs, and gives each a reason code (INHERITED_WINS, ACTOR_THRESHOLD_DROPPED, MENTION_WINS, DIRECT_HIGHER_CONF, NON_EXCHANGE_CLUSTER_DEFINER, DEFINER_TIE, …). It also checks each multi-address cluster's selected definer against the majority of its definers (each tagpack votes once per actor, weighted exp(confidence / 10); this vote is the check's own estimate, REST does not use it). actor-conflicts lists addresses with more than one actor and clusters whose definers or members disagree on the actor; actor pairs declared as same_as or related_actors in the actorpacks are not reported. Mention tags never count as exchange attributions: their category describes where the address was mentioned (e.g. a dark-web page), not its owner. Both write stable-sorted CSV or JSON (--out) and print per-reason counts to stderr. They read only the tagstore: cluster ids come from its address→cluster mapping (--clustering auto|legacy|v2|both). --limit N checks only N addresses and their clusters, for a quick trial run; add -v (graphsense-cli -v tagpack-tool …) to see progress.
• Actorpacks can declare same_as and related_actors in an actor's context. same_as marks another actor entry as the same organisation (rebrand, duplicate), related_actors a different organisation that legitimately tags the same addresses (e.g. a custodian and its wrapped token). Validation rejects self-references and pairs listed in both fields and warns about ids not in the same pack. Actors declared same_as no longer get the "share the same domain / twitter_handle / coingecko_id / defilama_id, consider merging" warnings. Older versions ignore the fields, so actorpacks can use them right away. See tagpack/docs/README.md.
• Actorpacks can declare sub_service_of and nested_in in an actor's context. sub_service_of names the one actor this one belongs to as a product or division of the same organisation (e.g. an exchange's mining pool). nested_in lists actors whose addresses or accounts this one runs on as a separate organisation (e.g. an exchange that keeps its funds with a custodian, a service operating through exchange accounts). Validation rejects self-references, an id in both fields and sub_service_of cycles, and warns about ids not in the same pack; sub_service_of pairs get no "consider merging" warning. quality actor-conflicts does not report either pair. quality ranking-conflicts accepts a summary that shows the same organisation (same_as, sub_service_of) or a nested service instead of the address's exchange tag, but still reports a summary that shows the host and hides the nested service. The tag summary itself is unchanged. Older versions ignore both fields.
• tagpack-tool quality list-bad-text (read-only). Lists every value already in the tagstore that tagpack validation now rejects (see below), per tagpack or actorpack and field, with the probably intended text for mis-encoded values, so it can be fixed in the pack.

Deprecated

• The watch module (graphsense-cli watch money-flows) is deprecated and will be removed in a release after 2027-04-09. Invoking it prints a deprecation notice; constructing FlowWatcherFactory raises a DeprecationWarning. It still works unchanged until then.

Fixed

• The fast YAML loader reads files as PyYAML does. It used to go through JSON, which turned some unquoted values into something else: digits with an exponent (12E34, address-like strings) into numbers or inf, inf/nan into .inf/.nan, aliases (*id001, written by yaml.dump for repeated lists) into the literal text, and kept yes/no, ~, 012 as text. Now rapidyaml only parses and unquoted values are interpreted with PyYAML's rules. With that, packs in a directory with a header.yaml use the fast loader too (before, all of them went through PyYAML); only packs with an !include still use PyYAML. In both loaders an unquoted 0x… stays text instead of a hexadecimal number. The differences of older versions are listed in tagpack/docs/README.md ("How TagPack files are parsed"). Validation of a 300-pack repository below a header.yaml: 74 s → 42 s (one job), 21 s (two jobs).
• tagpack-tool disappeared after a fresh install. SQLAlchemy 2.1 no longer installs greenlet by default, and the tagstore's asyncio engine needs it; the tagstore extra now declares sqlalchemy[asyncio]. When an optional CLI (tagpack-tool, tagstore) cannot be imported, graphsense-cli now reports the import error and the extra to install instead of "No such command".
• Claude Code sees the whole MCP server instructions and every tool description. Claude Code cuts both at 2048 characters without telling the server. The instructions were 7.4k characters, so only the first rules reached the model and the Pathfinder link and tag field guidance never did; build_pathfinder_file (5.4k) and lookup_tx_details (3.1k) lost their ends too. The instructions now hold only cross-tool rules (1.9k); per-tool guidance moved into the tool descriptions, argument detail into the input schema, and the tag field fallback order into list_tags_by_address. A test keeps everything under 2048.
• Investigations start with an investigator, not with trace-funds. With the Claude Code plugin, Claude picked the shared trace-funds skill directly for "trace the funds from …" and skipped the investigators, so no investigator set how far to go. The shared skills' descriptions now say an investigator loads them and to pick one first.
• A tagpack the fast YAML parser rejects no longer aborts with a bare ExceptionParse. rapidyaml's exception carries no message (it prints line and column to stderr); loading now retries with PyYAML, which either loads the file or fails with the line and column of the error.
• Tagpacks and actorpacks are read as UTF-8 whatever the machine's locale. Files were opened with the platform default encoding, so on a non-UTF-8 locale an insert stored UTF-8 text as mojibake.
• tagpack-tool reads POSTGRES_PORT. The URL built from POSTGRES_* environment variables used to hard-code port 5432; it now uses POSTGRES_PORT when set.
• tagpack add-actors keeps !included headers and the file's formatting (#33). It used to write back the merged in-memory pack, inlining the header's fields along with defaults filled in on load (confidence, network) and JSON-encoded context values. It now edits the file as written: it inserts the new actor lines and updates lastmod, and leaves everything else, including comments and the header: !include … reference, unchanged. When a change can't be made as a line edit (tags in {…} flow style), it warns and rewrites the whole file: the header stays an !include, but comments are lost.
• tagpack add-actors no longer moves a shared actor to pack level. When every tag it asked about got the same actor, it moved that actor to the pack and removed actor from all tags. That also hit tags it hadn't asked about: with --categories it raised KeyError on tags outside the filter, and it silently replaced an actor a tag already had. Each tag now keeps the actor assigned to it.
• UTXO ingest fails loudly instead of reading a missing amount as 0 (#182). Hardening only; no node emits these shapes and the parsed output is unchanged on all mainnet ZEC blocks. A ZEC transaction or Orchard/Ironwood bundle with spends, outputs or actions but no valueBalanceZat, a Sprout joinsplit with vpub_old/vpub_new but no *Zat integer, and a vout without value (any network) now raise. ZEC Sprout amounts and outputs are read from the integers (vpub_*Zat, valueZat). zcashd < 2.0.7 is no longer accepted for transactions with joinsplits.
• UTXO ingest with resolve_inputs_via_cassandra recomputes fee after resolving inputs (#183). It was left at the exporter's value, which leaves out the transparent inputs and is usually negative. Only the Delta lake stores the UTXO fee, and nothing reads it; the default configuration was not affected.
• UTXO ingest with resolve_inputs_via_cassandra can write to the Delta lake (#189). Inputs resolved from Cassandra carried the stored integer address type, which the parquet schema rejects (ArrowTypeError), so the batch reached neither sink. The resolver now returns the type name, as for outputs seen in the same run. The default configuration was not affected.
• A delta update whose worker is killed (e.g. by the OOM killer) now fails instead of hanging with its locks held. With --parallel-workers > 1, the pool waited for the surviving workers on its way out, and one still busy with its chunk ignored the executor's SIGTERM (workers ignore it on purpose), so the run hung without logging anything: its lock heartbeat kept going, so no stale-lock alert fired either. On an error the pool now kills its workers and the exception reaches the caller, which releases the locks; the next run replays the torn batch from the WAL. A run that ends normally or after Ctrl-C still lets every chunk finish. The pool also starts all its workers up front: started on demand, a worker that died right after starting could go unnoticed, with the same hang.
• ingest from-node with an exchange-rates provider no longer warns Token-rate pre-ingest … failed: table token_configuration does not exist on UTXO networks. The token-rate step ran for every network, but only account-based networks (ETH, TRX) have tokens; it now runs only there. Nothing else changes: the warning was the only effect, and block ingest always continued.
• The UTXO delta update needs far less memory and time for batches with large CoinJoins, which create one address relation per input/output pair. Writes are now sent 10,000 at a time without keeping each statement and result (about 5 KB → 0.15 KB per change; a timeout retries only the failing chunk), merging the per-transaction deltas is linear instead of quadratic, and the cluster delta no longer deep-copies the address delta. What gets written is unchanged.
• tagpack-tool no longer hangs forever after a network outage. A tagstore connection whose server side was gone (e.g. after a VPN drop) looked open, so a quality run waited for an answer that never came, without an error. Connections now use TCP keepalive and a TCP user timeout, so a dead one fails within about two minutes, and the quality commands' pool replaces a connection that died while idle (pool_pre_ping). Long queries on a live server are not affected.
• An outage of an external backend alerts once instead of once per request. Its first failure logs at ERROR (the Slack level), later ones at WARNING, and the first answer after it logs how long it lasted. MCP tools no longer alert per call on the backend's 503. Each worker process alerts once.
• MCP tool results that leave out the externally served networks say so. During an outage search, get_statistics and lookup_address with cross-chain addresses start with a note naming the missing networks, so a missing hit reads as not checked instead of not found.

Changed

• MCP lookup_address reads the cluster from /addresses/{address}/cluster instead of the deprecated /addresses/{address}/entity, which is sunset on 2026-10-31. The response is unchanged.
• MCP clients without the plugin now find the investigation skills. The server instructions name the investigators and say to pick one before any tracing, read from skill://<name>/SKILL.md when the skills aren't installed. The model does this only when the user asks for an investigation, so a single lookup costs no extra tokens.
• investigate-strict is the default investigator (was investigate-advisor). Without a hint on how far to go, an investigation answers what was asked and stops. Ask for next steps to get the advisor.
• Tagpack and actorpack validation reports mis-encoded and non-printable text (warning for now). A field is reported if it contains UTF-8 that was read as Latin-1/cp1252, also when only part of it is damaged (e.g. Ð\x9bÐ\x98Ф for ЛИФ, München for München; the message names the probably intended text), or if it contains control characters other than tab and line breaks. Each pack logs one summary warning; the values are still inserted, because some generated packs carry many of them and are being repaired at their generators. A later release will reject them (TEXT_PROBLEMS_ARE_ERRORS). Null characters are still rejected. The check now covers actorpacks and looks inside lists; quality list-bad-text lists what is already in the tagstore and scripts/one-off-fixes/fix_tagpack_mojibake.py repairs pack files.
• sqlmodel is no longer capped <0.0.45. The tagstore's datetime columns (lastmod on tagpack, actorpack, actor and tag; created on address) are now declared as DateTime(timezone=False) explicitly, so sqlmodel 0.0.45+ no longer maps them to timestamptz and rejects naive values. The generated column type is unchanged (TIMESTAMP WITHOUT TIME ZONE), so existing tagstores need no migration.

Web API + Python client

The API and client versions move to 2.18.0, in line with the library.

Changed

• The actors lists of addresses and clusters count only actor-type tags, the same tags the tag summary's best actor is chosen from. A mention or event tag keeps its actor as a link (it still appears in the tag list and under the actor), but no longer lists that actor as one of the address's or cluster's actors: a forum post about a gambling site or a ransom payment to an exchange deposit address names whom the label refers to, not who controls the address. REST ties this to the tag summary's setting tag_summary_only_propagate_high_confidence_actors (on by default): with it off, the lists show every tag's actor as before. quality actor-conflicts always applies the rule.
• POST /graph/summary and POST /graph/compare carry request body examples in the OpenAPI spec, with real BTC transactions and addresses, so the generated Python client docs show a working call instead of an empty Model().

Fixed

• A mistyped BCH CashAddr or TRX address in a REST request is rejected with 400 instead of resolving to a different address. Neither checksum was verified, and a truncated or extended CashAddr decoded to an unrelated legacy address. User input now goes through a checked conversion (checksum, hash length, padding); ingest keeps the unchecked fast path for node data. The same lookups now also accept a CashAddr with an uppercase or missing bitcoincash: prefix or an all-uppercase payload, all-uppercase bech32 on BTC and LTC, and the 41… hex form of TRX addresses. Tagpack import verifies CashAddr checksums too, and keeps a mistyped one as written (with a warning) instead of converting it.
• A BTC, LTC, BCH (legacy) or ZEC address with a wrong checksum that is not indexed now gets 400 instead of 404, naming the checksum mismatch as a probable typo. Before, a mistyped base58 or bech32 address looked like a valid address that has no transactions yet. The check runs only after the lookup found nothing, so an indexed address is never rejected whatever its checksum, and it only judges the checksum (either bech32 constant counts, any witness version), so unusual but real outputs are never called typos. nonstandard placeholders and unrecognised formats still get 404.
• An outage of the external backend (the lite networks' service) no longer takes the core networks down with it. /stats, /capabilities, /search and twin lists failed with 500 for every network, so the dashboard did not start and MCP search and get_statistics failed for everyone; a hung backend held each of them for 60 s first. They now answer the core networks' part: in under 0.1 s with the backend down, after merge_timeout_s (5 s) with it hung. A request on one of its networks gets a 503 the browser can read (the generic 500 carried no CORS headers).
• The headers the dashboard reads are named in Access-Control-Expose-Headers (X-Ikn-Backends-Declined, Retry-After, X-Served-By): on a request with credentials browsers read the * wildcard as a header named *.

What's Changed

• fix(ingest): raise on missing required ZEC values (non-empty bundles, Sprout, vouts) by @steven3002 in #192
• fix(ingest): recompute fee after resolving inputs through Cassandra by @steven3002 in #193

Full Changelog: v2.17.0...v2.18.0