Repository navigation
src2sink 3.0.0
src2sink can now say which entry points reach which sinks, with evidence.
That is the capability the tool is named for, and until this release it did not
have it: it found sources, it found sinks, and nothing connected them.
⚠️ Upgrading
A full rescan is required. DETECTION_VERSION moves 10 → 11, so records
built by an earlier version are not reused. SCHEMA_VERSION stays at 2 — the
record structure is unchanged and an existing metabase still parses — but two
things about its contents changed, and one of them will break a consumer that
does not expect it.
Breaking: call-site observations now come in two shapes. Observation was
widened from sink-shaped names to every call, because the middle of a layered
path was recorded nowhere. Recording the full SQL-evidence set for every call
measured at 3.4x the nodes and 4.7x the record size on a real repository, so an
ordinary call now records only symbol, receiver, arguments and its
enclosing scope. raw, receiver_is_database, library_hint,
file_sql_evidence and parameterised are absent on ordinary calls and
present only on sink-shaped ones. Read them with .get(), not []; absent means
"nothing vouched for this call being SQL, because nothing looked". See
SCHEMA.md.
Records grow ~1.6x in nodes and ~1.7x in bytes. That is the measured cost of
the widening after two mitigations: dropping raw from ordinary calls, and
pruning calls that name nothing declared in the repository — 77% of observed
calls, being get, append, len and the rest of the standard library, which
an intra-repo path cannot pass through by definition.
Added
-
Source-to-sink paths inside a service (
OI-17). The headline. A new
tainted-pathfinding states that a value from an entry point reaches a sink,
and cites every hop with itsfile:line, its resolution tier, and the argument
that carried the value. Reachability alone would report every endpoint as
reaching every sink its service can touch; a hop carrying no tainted value is
pruned, so a decoy calling a static query does not appear at all rather
than appearing with a low score.Path confidence is the minimum hop, never a product, with length recorded
separately and the weakest link named — a reader can act on "8 hops, weakest
link is theB→Cbinding" and nobody can act on0.058. Depth is unbounded:
measured on a 2,000-service fleet, capping at three hops finds 25% of what
depth eight finds. There is no confidence floor, because for an indicator a
floor converts cheap false positives into expensive, invisible false negatives. -
Tiered call resolution (
OI-17, step 3).stockService.process(...)now
resolves to a specific declaration: T1 from a declared field type (high),
T2 an interface expanded to its implementations (medium, and explicitly
ambiguous when there is more than one), T3 a name unique in the repo
(low); not unique, dropped. The tier is recorded on every edge. T2 is why the
answer can never be "unreachable" — a constructor-injected interface field is
the standard Spring shape, so stopping at the declared type would report a
confident dead end for most of a JVM fleet. -
The first
intra-repoedges.FlowEdgehas advertised the kind since the
schema was written and nothing ever emitted one. -
Call arguments are recorded, which is what makes a path evidence rather
than reachability. -
A persisted fleet index (
OI-15).metabase/index.sqlite3, built during
aggregation, holds the four things a trace consults, keyed by target repo. A
trace now answers from indexed lookups and never loads the fleet — peak
memory is a function of what arrives at the target, not of fleet size.
Previously a trace read every record in the metabase to answer a question about
one repo, which at 34 GB on disk needs roughly 222 GB resident merely to be
held. Staleness is checked on every read; a stale, missing or corrupt index
falls back to loading rather than serving a wrong answer.
Fixed
-
The producer scan read the whole fleet once per binding (
OI-30). Reported
from the field at 70 minutes — the slowest part of a scan bar fleet-wide
traces. The binding loop was on the outside, so a 34 GB checkout was read from
disk once per binding and the only thing that differed between passes was which
regex ran over text already in memory. Now the fleet is walked once and matched
against every binding while resident: N× fewer file reads, plus a further 2×
from building the indices once per run instead of twice. -
Discovered bindings named the client library instead of the service
(OI-40). When a client library is published from its own repository, the
repo declaring the coordinate is the library — so 42 of 191 candidates
pointed at a library rather than the service it calls, while 0 of 99
hand-authored bindings do. The records looked entirely correct, which is why
nothing rejected them. A target with no inbound endpoints is now corrected to
the single sibling service that has them, the substitution is shown to the
reviewer, and a target that cannot be corrected is flagged rather than dropped
— because "no endpoints" is also what a detection gap looks like. -
A gate against silent detection failures (
OI-36). The principle that a
detection path must not fail to empty without saying so has been written down
since 1.1.0 — and was then violated in every release since, including three
Kotlin gaps that left a JVM fleet reporting clean while detecting nothing. It
is now enforced: a handler that discards an exception must emit a signal or be
listed with a reason. 59 were found, 15 justified, and 44 frozen as debt that
can only shrink. No behaviour change; what it prevents is the next one. -
A versioned
api/latest/directory contributed nothing to the metabase
(OI-39). The test-path predicate's camelCase branch ran under
IGNORECASE, so any path segment ending in "test" was treated as a test
directory —latest,protest,contest,attest,greatest. Extraction
returns nothing for a test path, so a repository laid out that way produced no
endpoints, no sinks and no dependencies, with no note and no count. The same
predicate was too narrow in the other direction: test files living beside
their code (routes.spec.ts,handler_test.go,test_views.py) were
extracted as though they shipped. Both fixed;DETECTION_VERSION12 → 13. -
Any
.get("…")in JavaScript was an inbound HTTP endpoint (OI-37). The
Express pattern began at the dot, so it matched a verb-named call on any
receiver — and JavaScript is the language where.get(key)is ubiquitous for
reasons unrelated to HTTP. Over a 746-repo estate that produced 10,245
inbound endpoints of which 20 were real, all atconfidence: "high":
reactive-form field access, Cypress selectors, cache writes, and 605 outbound
client calls recorded as doors into the service. The receiver is now required.
Confidence is derived from how anchored a pattern is rather than hardcoded, and
rawnow contains the receiver so a reviewer can audit a match without
re-reading the source.DETECTION_VERSION11 → 12. -
The trace index never described the batch that had just run (
OI-38). Only
the build wrote it, and the build runs before any trace exists — so its
catalogue-coverage figure was always one run stale, and on a first run no index
appeared at all. The batch now writes it too. -
Discovery's two passes could never agree, so
discovery_method: "both"was
unreachable (OI-33). The supply-side pass named a target by the directory
that declares the client coordinate — inside the repo for a multi-module
build — while the demand-side pass used the repo id. An exact-string lookup
could not bridge them, so the strongest signal the design produces occurred
zero times in 226 candidates. Targets are now normalised to the longest known
repo id, which also restores thepathsandservice_aliasesthe bad key was
suppressing. Reviewer decisions on already-triaged candidates are preserved
across the key change, and a coordinate that resolves to nothing now says so
instead of arriving as an ordinary weak candidate. -
Api-client discovery rescanned the whole fleet once per class (
OI-35).
The demand-side pass asked "which repos contain a file calledStockClient"
by walking every node of every record, once per target and per class — so
node visits grew about 15x for every doubling of the repository count. The
question is corpus-wide and target-independent, so it is now answered from an
index built in one pass: 576x fewer node visits at 48 repos, and the
saving keeps growing with the fleet. -
The checkout was walked 25 times per run to find a handful of files
(OI-31).Path.rglob(name)traverses the whole tree and filters by name,
so four filenames cost four traversals — and no phase shared a walk with any
other. Now one traversal serves everything: 25 → 3, and → 1 when the
CLI names every pattern up front. -
--discover-api-clientswas silently ignored outside a full scan
(OI-31). Combined with--graphs-onlyor--aggregate-onlyit printed
Done.and wrote no candidates file — the flag accepted, doing nothing, with
no clue why. Discovery reads records and the checkout, both of which those
modes have. -
Three Kotlin gaps, all silent since 2.1.0. Each made Kotlin produce a
clean-looking result rather than a wrong one, which is the failure mode
OI-13exists to prevent, and each passed its original parity test because
those tests compared the easy half of a record:- interfaces were never recognised as interfaces — Kotlin has no
interface_declarationnode, so a call on an interface-typed field bound to
the bodiless method and the chain stopped; - every method recorded an empty parameter list — Kotlin exposes no
parametersfield, so nothing could be tainted and no Kotlin path existed
anywhere in a fleet; - every call recorded an empty argument list — Kotlin names no argument
field, so no Kotlin hop could carry a value.
- interfaces were never recognised as interfaces — Kotlin has no
-
A caller's reported confidence was whichever edge came last (
OI-29).
collect_service_edgesemits several edges per caller, one per route it might
be addressing, and the merge kept the last rather than the strongest — so a
highedge was routinely overwritten by alowone for the same caller.
Callers now report their best-evidenced confidence, which for an indicator
matters: understating evidence suppresses the lead it should raise. -
Kotlin call sites were invisible to the AST pass (
OI-13). The dispatch
named Kotlin and then routed it to the Java walker, which requires a
method_invocationnode Kotlin never produces. Every Kotlin call site returned
nothing, so Kotlin SQL sinks were found only when a regex tier happened to
match — and nothing reported the gap. Kotlin repositories now yieldsql
sinks,script-execsinks andraw-code-payloadfindings from the AST tier,
so counts rise for any Kotlin service. A prerequisite for everything above:
reachability that silently covers one language is worse than none.
Also added in this release
- Type facts for call resolution (
OI-17, step 2). A newtype-decl
observation records each declared type's field types, supertypes and whether it
is an interface. A field's declared type is what makesstockService.process()
resolvable offline without a compiler; the supertypes are what let a call on an
interface reach the implementations that have a body. Kotlin constructor
properties count as fields, since that is how the standard Spring shape declares
its collaborators. - Nodes know which method they are in (
OI-17, step 1). A node recorded
fileandlineand nothing about its enclosing method, so there was no
"entrypoint 1 of B" to reason about. Callables are now recorded asmethod-decl
observations with class, parameters and span, and every node carries
enclosing_class/enclosing_method— or neither, if it sits inside no method.
Foundation for call resolution and reachability; neither is possible without it. - Entry points beyond HTTP (
OI-21). A way into a service counted only if it
was an HTTP annotation, so queue consumers, gRPC services, GraphQL resolvers,
scheduled jobs and CLI entry points were invisible as front doors. A new
derivedentry-pointfamily unifies them, withmechanismnaming which kind
andexternally_triggeredseparating a door a caller can open from one only
the clock opens.
A@KafkaListenerneeded no new extraction — it has produced aqueue-sub
node since 1.x and was simply never treated as a way in.
Both versions bump, so the fleet rescans.