Skip to content

src2sink 3.0.0

Choose a tag to compare

@github-actions github-actions released this 07 Aug 14:22
· 16 commits to main since this release
b11eec7

src2sink can now say which entry points reach which sinks, with evidence.
That is the capability the tool is named for, and until this release it did not
have it: it found sources, it found sinks, and nothing connected them.

⚠️ Upgrading

A full rescan is required. DETECTION_VERSION moves 10 → 11, so records
built by an earlier version are not reused. SCHEMA_VERSION stays at 2 — the
record structure is unchanged and an existing metabase still parses — but two
things about its contents changed, and one of them will break a consumer that
does not expect it.

Breaking: call-site observations now come in two shapes. Observation was
widened from sink-shaped names to every call, because the middle of a layered
path was recorded nowhere. Recording the full SQL-evidence set for every call
measured at 3.4x the nodes and 4.7x the record size on a real repository, so an
ordinary call now records only symbol, receiver, arguments and its
enclosing scope. raw, receiver_is_database, library_hint,
file_sql_evidence and parameterised are absent on ordinary calls
and
present only on sink-shaped ones. Read them with .get(), not []; absent means
"nothing vouched for this call being SQL, because nothing looked". See
SCHEMA.md.

Records grow ~1.6x in nodes and ~1.7x in bytes. That is the measured cost of
the widening after two mitigations: dropping raw from ordinary calls, and
pruning calls that name nothing declared in the repository — 77% of observed
calls, being get, append, len and the rest of the standard library, which
an intra-repo path cannot pass through by definition.

Added

  • Source-to-sink paths inside a service (OI-17). The headline. A new
    tainted-path finding states that a value from an entry point reaches a sink,
    and cites every hop with its file:line, its resolution tier, and the argument
    that carried the value. Reachability alone would report every endpoint as
    reaching every sink its service can touch; a hop carrying no tainted value is
    pruned, so a decoy calling a static query does not appear at all rather
    than appearing with a low score.

    Path confidence is the minimum hop, never a product, with length recorded
    separately and the weakest link named — a reader can act on "8 hops, weakest
    link is the B→C binding" and nobody can act on 0.058. Depth is unbounded:
    measured on a 2,000-service fleet, capping at three hops finds 25% of what
    depth eight finds. There is no confidence floor, because for an indicator a
    floor converts cheap false positives into expensive, invisible false negatives.

  • Tiered call resolution (OI-17, step 3). stockService.process(...) now
    resolves to a specific declaration: T1 from a declared field type (high),
    T2 an interface expanded to its implementations (medium, and explicitly
    ambiguous when there is more than one), T3 a name unique in the repo
    (low); not unique, dropped. The tier is recorded on every edge. T2 is why the
    answer can never be "unreachable" — a constructor-injected interface field is
    the standard Spring shape, so stopping at the declared type would report a
    confident dead end for most of a JVM fleet.

  • The first intra-repo edges. FlowEdge has advertised the kind since the
    schema was written and nothing ever emitted one.

  • Call arguments are recorded, which is what makes a path evidence rather
    than reachability.

  • A persisted fleet index (OI-15). metabase/index.sqlite3, built during
    aggregation, holds the four things a trace consults, keyed by target repo. A
    trace now answers from indexed lookups and never loads the fleet — peak
    memory is a function of what arrives at the target, not of fleet size.
    Previously a trace read every record in the metabase to answer a question about
    one repo, which at 34 GB on disk needs roughly 222 GB resident merely to be
    held. Staleness is checked on every read; a stale, missing or corrupt index
    falls back to loading rather than serving a wrong answer.

Fixed

  • The producer scan read the whole fleet once per binding (OI-30). Reported
    from the field at 70 minutes — the slowest part of a scan bar fleet-wide
    traces. The binding loop was on the outside, so a 34 GB checkout was read from
    disk once per binding and the only thing that differed between passes was which
    regex ran over text already in memory. Now the fleet is walked once and matched
    against every binding while resident: N× fewer file reads, plus a further 2×
    from building the indices once per run instead of twice.

  • Discovered bindings named the client library instead of the service
    (OI-40).
    When a client library is published from its own repository, the
    repo declaring the coordinate is the library — so 42 of 191 candidates
    pointed at a library rather than the service it calls, while 0 of 99
    hand-authored bindings do. The records looked entirely correct, which is why
    nothing rejected them. A target with no inbound endpoints is now corrected to
    the single sibling service that has them, the substitution is shown to the
    reviewer, and a target that cannot be corrected is flagged rather than dropped
    — because "no endpoints" is also what a detection gap looks like.

  • A gate against silent detection failures (OI-36). The principle that a
    detection path must not fail to empty without saying so has been written down
    since 1.1.0 — and was then violated in every release since, including three
    Kotlin gaps that left a JVM fleet reporting clean while detecting nothing. It
    is now enforced: a handler that discards an exception must emit a signal or be
    listed with a reason. 59 were found, 15 justified, and 44 frozen as debt that
    can only shrink. No behaviour change; what it prevents is the next one.

  • A versioned api/latest/ directory contributed nothing to the metabase
    (OI-39).
    The test-path predicate's camelCase branch ran under
    IGNORECASE, so any path segment ending in "test" was treated as a test
    directory — latest, protest, contest, attest, greatest. Extraction
    returns nothing for a test path, so a repository laid out that way produced no
    endpoints, no sinks and no dependencies, with no note and no count. The same
    predicate was too narrow in the other direction: test files living beside
    their code (routes.spec.ts, handler_test.go, test_views.py) were
    extracted as though they shipped. Both fixed; DETECTION_VERSION 12 → 13.

  • Any .get("…") in JavaScript was an inbound HTTP endpoint (OI-37). The
    Express pattern began at the dot, so it matched a verb-named call on any
    receiver — and JavaScript is the language where .get(key) is ubiquitous for
    reasons unrelated to HTTP. Over a 746-repo estate that produced 10,245
    inbound endpoints of which 20 were real
    , all at confidence: "high":
    reactive-form field access, Cypress selectors, cache writes, and 605 outbound
    client calls recorded as doors into the service. The receiver is now required.
    Confidence is derived from how anchored a pattern is rather than hardcoded, and
    raw now contains the receiver so a reviewer can audit a match without
    re-reading the source. DETECTION_VERSION 11 → 12.

  • The trace index never described the batch that had just run (OI-38). Only
    the build wrote it, and the build runs before any trace exists — so its
    catalogue-coverage figure was always one run stale, and on a first run no index
    appeared at all. The batch now writes it too.

  • Discovery's two passes could never agree, so discovery_method: "both" was
    unreachable (OI-33).
    The supply-side pass named a target by the directory
    that declares the client coordinate — inside the repo for a multi-module
    build — while the demand-side pass used the repo id. An exact-string lookup
    could not bridge them, so the strongest signal the design produces occurred
    zero times in 226 candidates. Targets are now normalised to the longest known
    repo id, which also restores the paths and service_aliases the bad key was
    suppressing. Reviewer decisions on already-triaged candidates are preserved
    across the key change, and a coordinate that resolves to nothing now says so
    instead of arriving as an ordinary weak candidate.

  • Api-client discovery rescanned the whole fleet once per class (OI-35).
    The demand-side pass asked "which repos contain a file called StockClient"
    by walking every node of every record, once per target and per class — so
    node visits grew about 15x for every doubling of the repository count. The
    question is corpus-wide and target-independent, so it is now answered from an
    index built in one pass: 576x fewer node visits at 48 repos, and the
    saving keeps growing with the fleet.

  • The checkout was walked 25 times per run to find a handful of files
    (OI-31).
    Path.rglob(name) traverses the whole tree and filters by name,
    so four filenames cost four traversals — and no phase shared a walk with any
    other. Now one traversal serves everything: 25 → 3, and → 1 when the
    CLI names every pattern up front.

  • --discover-api-clients was silently ignored outside a full scan
    (OI-31).
    Combined with --graphs-only or --aggregate-only it printed
    Done. and wrote no candidates file — the flag accepted, doing nothing, with
    no clue why. Discovery reads records and the checkout, both of which those
    modes have.

  • Three Kotlin gaps, all silent since 2.1.0. Each made Kotlin produce a
    clean-looking result rather than a wrong one, which is the failure mode
    OI-13 exists to prevent, and each passed its original parity test because
    those tests compared the easy half of a record:

    • interfaces were never recognised as interfaces — Kotlin has no
      interface_declaration node, so a call on an interface-typed field bound to
      the bodiless method and the chain stopped;
    • every method recorded an empty parameter list — Kotlin exposes no
      parameters field, so nothing could be tainted and no Kotlin path existed
      anywhere in a fleet;
    • every call recorded an empty argument list — Kotlin names no argument
      field, so no Kotlin hop could carry a value.
  • A caller's reported confidence was whichever edge came last (OI-29).
    collect_service_edges emits several edges per caller, one per route it might
    be addressing, and the merge kept the last rather than the strongest — so a
    high edge was routinely overwritten by a low one for the same caller.
    Callers now report their best-evidenced confidence, which for an indicator
    matters: understating evidence suppresses the lead it should raise.

  • Kotlin call sites were invisible to the AST pass (OI-13). The dispatch
    named Kotlin and then routed it to the Java walker, which requires a
    method_invocation node Kotlin never produces. Every Kotlin call site returned
    nothing, so Kotlin SQL sinks were found only when a regex tier happened to
    match — and nothing reported the gap. Kotlin repositories now yield sql
    sinks, script-exec sinks and raw-code-payload findings from the AST tier,
    so counts rise for any Kotlin service. A prerequisite for everything above:
    reachability that silently covers one language is worse than none.

Also added in this release

  • Type facts for call resolution (OI-17, step 2). A new type-decl
    observation records each declared type's field types, supertypes and whether it
    is an interface. A field's declared type is what makes stockService.process()
    resolvable offline without a compiler; the supertypes are what let a call on an
    interface reach the implementations that have a body. Kotlin constructor
    properties count as fields, since that is how the standard Spring shape declares
    its collaborators.
  • Nodes know which method they are in (OI-17, step 1). A node recorded
    file and line and nothing about its enclosing method, so there was no
    "entrypoint 1 of B" to reason about. Callables are now recorded as method-decl
    observations with class, parameters and span, and every node carries
    enclosing_class/enclosing_method — or neither, if it sits inside no method.
    Foundation for call resolution and reachability; neither is possible without it.
  • Entry points beyond HTTP (OI-21). A way into a service counted only if it
    was an HTTP annotation, so queue consumers, gRPC services, GraphQL resolvers,
    scheduled jobs and CLI entry points were invisible as front doors. A new
    derived entry-point family unifies them, with mechanism naming which kind
    and externally_triggered separating a door a caller can open from one only
    the clock opens.
    A @KafkaListener needed no new extraction — it has produced a queue-sub
    node since 1.x and was simply never treated as a way in.
    Both versions bump, so the fleet rescans.