Accuracy

Accuracy

How often Hopline's guesses are right, measured on transactions where the answer is known, and what that means for how far to lean on a badge. The method is docs/benchmark.md and every number, with its 95% interval, is in docs/benchmark-results.md.

• What was measured. Six scripted scenarios on a Bitcoin Core test network (regtest): ordinary payments between wallets with different habits, peel chains, exchange-style batches and consolidations, CoinJoins, PayJoins, and an exchange sweeping its deposit addresses. 3,647 transactions in all, made by a script that knows who owns every address and which output of every payment is the change. Each guess Hopline makes is scored against that.

• Read this first. The data is synthetic and every wallet in it is Bitcoin Core with different settings (address types, where the change goes, round or odd amounts, address reuse). It shows that the rules do what they claim and where they break. It does not say how often they are right on real wallets, which run other software with other habits, so expect lower numbers there. A sample of the real chain and one documented case were also measured (the last two bullets): they say how often a flag fires and what a rule does on one route, not whether a guess is right. A test with real wallet software on a public test network is built but has no data yet, because it needs people with wallets and faucet coins; see the ROADMAP. The first run was reported on the same seed the benchmark was built with, so it is a report and not a test on fresh data.

• The change guess: the label means something. Of the 3,068 transactions in these cases that have a change output, Hopline named one on 74%. A high guess was right 902 times in 903, a medium guess 93.5% of the time and a low guess 90.8%. A rule that always answers does much worse: the largest output is right 56% of the time, the last 54%, a random one 48%. So high, medium, low is an order of trust. Do not read the percentages as probabilities for your case.

• When the change guess is wrong.

  • The wallet pays its change to an address of another type than the one it received on: right 73% to 82% of the time then, against 99.8% when the types are the same. The "only output of the same address type" sign then points at the payment.
  • A transaction with no change at all: Hopline still guessed on 86 of 408 (one of them high). A guess that rests on one sign says so; read the reasons.
  • A PayJoin gave the one wrong high guess.
  • Exchange-style wallets get almost no guesses (2%): their transactions do not carry the signs.
  • One sign is a weaker lead. A medium guess on one sign was right 89.5% of the time against 99.5% on two signs, and "same address type" alone was the weak one (84%); on real traffic 66% of the guesses rest on one sign. Capping those at low is on the roadmap and waits for real-wallet data.
  • Which sign matters: "same address type" buys coverage and costs precision (guesses on 74% of the transactions and right 95.6% of the time, against 49% and 99.9% without it). Round amounts help both.

• Clusters. Where one sender spent every transaction, addresses spent together were put together correctly every time (about 12,700 pairs, none wrong). Two things break it. A PayJoin joins two owners and the rule cannot tell: in the PayJoin case 95% of the pairs were right and 150 of the 273 clusters held two owners. A CoinJoin is left out on purpose, which is worth it: without that exclusion only 49% of the pairs in the CoinJoin case would share an owner. The price is that an exchange paying several customers the same amount looks like a CoinJoin and its addresses stay apart (65% of the pairs that spending together could find were found in that case).

• The pattern badges name shapes, and ordinary behaviour makes the same shapes.

  • COINJOIN? found all 60 real CoinJoins and also flagged 75 of the other 375 transactions in that case (the exchange payouts of equal amounts), so a flag was right 44% of the time there.
  • CONSOL found every consolidation, but a payment out of five or more coins has the same shape: 59 of 1,200 ordinary payments (5%) were flagged, and in the case that mixes the two a flag was right 40% of the time.
  • BATCH found every batch and invented none: up to three inputs paying six or more outputs is a clear shape.
  • PEEL found chains step by step (71% of the steps of real chains were inside a detected chain) but only 2 of 14 whole chains came out whole, because one step that pays more than a third of the coin ends the run. Ordinary payments out of one coin made a chain-shaped run 4 to 15 times per 1,000 transactions.

• On real traffic (4,674 transactions from 8 blocks, 2013 to 2026). Nobody knows who owns what there, so this says how often, not how right.

  • Both COINJOIN? flags in the first sample were wrong: two batches from one wallet in 2023 (20 inputs, 31 outputs) that paid every equal amount to one address (381, 546 and 600 satoshis). The detector now counts the different addresses an equal amount pays, not the outputs, because a CoinJoin pays each participant a different address. On a second sample it was not made on (10,723 transactions, 8 other blocks) the change dropped 2 of 12 flags (again one address paid over and over) and kept 10, including all three Whirlpool-shaped transactions in it; nothing was added, and no number on the synthetic cases moved. The samples are too small to say how many real CoinJoins it misses.
  • The shape of a peel step (one coin in, a small and a large output) is 41% of all transactions, and 3.6% sat in a detected chain even with only a sample's neighbours to form one: with a whole investigation loaded, a PEEL badge will often be ordinary traffic. Sorted by shape, about a fifth of the steps in detected chains (20% and 25% on two samples) peeled exactly the dust limit of an address type: 546 satoshis next to an OP_RETURN output (2019) and 330 satoshis (2025), which looks like token and inscription transfers and not money. A peel step now has to be at least 1,000 satoshis. On a third sample the change was not made on (9,247 transactions) it dropped 187 of 502 steps in detected chains (181 of them dust peels, 6 collateral) and added none; the rest are chains inside one block or the same amount peeled again and again, which look like a service paying out in turn. A longer minimum chain length trades real chains found against chains that were not made as one; the benchmark shows both columns, and the default (4) is unchanged.
  • CONSOL fired on 3.5% and BATCH on 3.2% of the transactions.
  • A change guess was made on 71% of the transactions that can have change, about as often as on the synthetic cases, but on other grounds: address reuse is behind 46% of the real guesses (3% of the synthetic ones), and 66% rest on one sign only.

• One documented case: the Colonial Pipeline ransom. From the ransom payment of 8 May 2021 to the address the US seized 63.7 BTC from, a route of four transactions (the payment, a 63.75 / 11.25 split, 63.7 BTC moving on, a consolidation of 24 inputs). Following the largest output reached the end in 3 hops and 7 requests with no wrong branch; following every output got there in 10 requests with 2 wrong branches and then kept loading, because it cannot know it has arrived; the peel-chain rule stopped at the first transaction with two inputs; the likely-change rule follows what the sender kept, not the payment, and ended at an unspent coin. One case is a case study, not a rate. The court documents themselves could not be read (the Justice Department site answers with a bot check), so the end address rests on Elliptic's statement that it is the affidavit's; the case page says what was read.

• What to do with this. Use the label as an order, not as a probability. Read the reasons the page shows, and follow both outputs when a guess is medium or low. Treat a badge as a place to look, not a finding. A cluster that holds a PayJoin can hold two owners. See Analysis and findings for what each result is and is not.

• Run it yourself. node test/benchmark/run.js measures the committed cases (no network), node --test fails when a number falls below its floor in test/benchmark/floors.json, and python tools/bench/make_regtest.py --out <folder> makes the cases again from the seed (it needs Docker and takes about half an hour). See Development.