Skip to content

feat: analyze JavaScript — discovery, dynamic method idioms, and a loud jelly failure - #86

Merged
rahlk merged 7 commits into
release/0.xfrom
feat/issue-84
Aug 6, 2026
Merged

feat: analyze JavaScript — discovery, dynamic method idioms, and a loud jelly failure#86
rahlk merged 7 commits into
release/0.xfrom
feat/issue-84

Conversation

@rahlk

@rahlk rahlk commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Closes #84. Closes #85. Closes #88.

Two related gaps that together meant JavaScript projects were unanalyzable, plus one fix to stop a silent degradation.

Targets the 0.x line (release/0.x, off v0.5.0), not mainpython-sdk pins codeanalyzer-typescript==0.4.3 and its model layer is the v1 flat TSApplication, so nothing released on the v2 line reaches a consumer. Planning context: roadmap.

1 — Discovery never offered a JavaScript file (#84)

src/syntactic_analysis/discovery.ts:5 restricted discovery to .ts/.tsx/.mts/.cts, so a JS-only project emitted an empty symbol table and exited 0 with no warning. On OWASP NodeGoat: 0 modules, an 84-byte analysis.json.

Nothing downstream needed changing — defaultCompilerOptions() already set allowJs and Jelly already accepts .js/.mjs/.cjs; both were simply never handed a file.

  • SOURCE_EXTS += .js/.jsx/.mjs/.cjs
  • isTestFile regex covers the same four, so .test.js is skipped like .test.ts
  • buildSymbolTable warns when discovery finds nothing, instead of succeeding silently

2 — Two method idioms were never callables (#85)

Edges are gated to allSignatures, built from the symbol table, so a method that isn't a callable can never be a call target:

  • this.<name> = fn inside a constructor function — landed in local_variables
  • object-literal members ({ foo(){} }, { foo: function(){} }) — dropped entirely

Language-neutral, not a JavaScript gap: NodeGoat renamed .js.ts yielded the same 24 callables before this change.

Five sites: contributorName names both forms (and lets a variable bound to an object literal contribute its name); namedBoundary treats them as callable boundaries; walkBody's dispatch becomes a callableOf helper; buildStatemented walks module-level object literals, which no function body covers; and resolveCalleeSignature gains a matching branch — the checker returns these as BinaryExpression/PropertyAssignment declarations, which isCallableDecl does not cover, so edges were still dropped after the symbol table was already correct.

3 — A jelly failure was silent on JS

The union provider degrades to tsc-only when the jelly leg throws, and reported it at info, which is not printed at default verbosity. On JS that is a ~81% edge loss with no signal. Now reported at error level when most analyzed modules are JavaScript; the reason is truncated, since execFileSync inlines the whole command line (27 file paths on NodeGoat).

The default stays union deliberately: measured with and without node_modules, union − jelly is the same 5 edges and jelly − union is empty. Two are library phantoms including needle.get, NodeGoat's SSRF sink, which stays tsc-only even once jelly can see node_modules.

Results on unmodified NodeGoat (deps installed, -a 2)

before after #84 after #85
modules 0 27 27
callables 0 24 59
tsc resolved call sites 0 28 51
union edges 0 161 184
named graph nodes 0 32 62
call-site resolution 11% 20%

59 callables matches parser-derived ground truth exactly — 115 function-like nodes in source, of which 59 are nameable.

The module key set exactly equals the set of .js files outside node_modules, vendor and test trees (27/27, diff empty).

Security-scoped reachability

The DAO method layer now appears in the graph, which it did not before:

app/routes/allocations.AllocationsHandler.displayAllocations
  → app/data/allocations-dao.AllocationsDAO.getByUserIdAndThreshold
       call_site: find(...) receiver_expr='allocationsCol' line=86
       local_variables: allocationsCol initializer='db.collection("allocations")'
       parameters: [db]

So "does user input from POST /allocations reach a Mongo query?" is answerable from the output. A machine-checkable edge into mongodb still does not exist — that needs interprocedural points-to and is tracked in #87, deliberately not faked here.

4 — Neo4j labels and relationship types are namespaced per source language (#88)

BREAKING. Node labels carried a TS twin; relationship types carried nothing at all. A database holding output from more than one analyzer therefore mingled edges — codeanalyzer-python already namespaces all 18 of its relationship types (PY_CALLS, PY_DECLARES, …) while this analyzer emitted bare CALLS/DECLARES.

Now namespaced by source language, not by analyzer:

labels : n:JSModule  n:JSCallSite  n:JSVariable      ← language-bearing
         n:TSApplication  n:TSPackage               ← language-less, analyzer namespace
rels   : JS_CALLS  JS_DECLARES  JS_DECLARES_VAR  JS_HAS_CALLSITE
         JS_HAS_MODULE  JS_RESOLVES_TO  TS_MEMBER_OF

Rules: a node carrying _module takes that module's language; nodes with none of their own (application root, packages, external library symbols) take the analyzer's TS namespace, since a sibling analyzer emits its own; an edge takes its source module's language, falling back to its target's — so application→module on a JavaScript project is JS_HAS_MODULE.

Implemented at the two hooks RowBuilder already exposed rather than at the 21 edge call sites: expand now receives the node's props, and a new retype runs in finish() where both endpoints' props are known. REL_TYPES stays the single source of truth and REL_TYPES_NS derives both namespaces, so catalog and projection cannot drift.

Also fixes a latent bug this change would otherwise have introduced: wipe() and bolt's DESCENDANTS are hand-written traversals over bare relationship types. Once edges were namespaced they matched nothing — the wipe would have silently deleted zero rows and left stale data. Both now derive from nsAlt().

Migration

Neo4j schema version 1.1.0 → 2.0.0. Every stored query must move:

MATCH ()-[:CALLS]->()          →  MATCH ()-[:TS_CALLS|JS_CALLS]->()

The version change forces a full re-upsert on the next incremental push.

Verification

  • 15 new tests across 3 files, each watched fail before implementing; the end-to-end ones re-proven RED by stashing src/
  • sample-app analysis.json byte-identical to v0.5.0 — no signature churn for code that already resolved
  • Error path verified against a real forced failure (JELLY_BIN=/nonexistent) at default verbosity: loud on NodeGoat, silent on sample-app

Known gaps, not addressed

rahlk added 3 commits August 5, 2026 17:47
`SOURCE_EXTS` held only the four TypeScript extensions, so a JavaScript-only
project produced an empty symbol table and exited 0 with no warning. Measured on
OWASP NodeGoat: 0 modules, 0 edges, an 84-byte analysis.json.

Nothing downstream needed changing — `defaultCompilerOptions()` already sets
`allowJs`, and Jelly already accepts .js/.mjs/.cjs; both were simply never handed
a file. Discovery was the whole gate.

- SOURCE_EXTS gains .js/.jsx/.mjs/.cjs
- isTestFile's regex covers the same four, so .test.js is skipped like .test.ts
- buildSymbolTable warns when discovery finds nothing, instead of succeeding silently

On unmodified NodeGoat this now yields 27 modules, and a call graph of 161 union
edges with dependencies installed (136 with --no-build). The module key set
exactly equals the set of .js files outside node_modules, vendor and test trees;
discovery is unaffected by dependency state. sample-app output is byte-identical
before and after.

Does not model CommonJS `require`/`module.exports` at the module level — imports
and exports stay empty on CJS input. Relative `require()` call targets do resolve
through the tsc resolver.

Closes #84
The union provider degrades to tsc-only when the jelly leg throws, and reported
that at `info` level — which is not printed at default verbosity, so the failure
was entirely silent.

That is tolerable on TypeScript, where the resolver carries the graph. On
JavaScript it is a cliff: measured on OWASP NodeGoat with dependencies installed,
jelly supplies 156 of the 161 union edges, so a silent degradation drops the call
graph by ~81% with no signal. The failure is now reported at error level when
most analyzed modules are JavaScript, and stays at info otherwise.

Also truncate the reason. execFileSync puts the whole command line in
Error.message, which on NodeGoat meant 27 file paths inlined into an error the
user is meant to act on.

The default stays `union`, deliberately: union is a strict superset of jelly on
JS. Measured both with and without dependencies materialized, union - jelly is
the same 5 edges and jelly - union is empty. Three of those five target
`const x = () => {}` callables declared inside a constructor function
(app/routes/session.js:14,138 and app/data/allocations-dao.js:60) that jelly
misses; two are library phantoms, including `needle.get` — NodeGoat's SSRF sink —
which stays tsc-only even once jelly can see node_modules. Jelly is the better
source of external symbols overall (21 to tsc's 2 with deps present), which is
why the two legs are kept complementary rather than one being preferred.

Success path is unchanged: sample-app analysis.json byte-identical to v0.5.0.
Two ways of declaring a method were never materialized as callables, so calls to
them could not resolve: edges are gated to `allSignatures`, which is built from
the symbol table.

  • `this.<name> = fn` inside a constructor function — landed in local_variables
  • object-literal members (`{ foo(){} }`, `{ foo: function(){} }`) — dropped

Language-neutral, not a JavaScript gap: NodeGoat renamed .js -> .ts yielded the
same 24 callables before this change.

Four sites: `contributorName` names the two new forms (and lets a variable bound
to an object literal contribute its name, so members are homed under it);
`namedBoundary` treats them as callable boundaries; `walkBody`'s dispatch is
replaced by a `callableOf` helper; and `buildStatemented` walks module-level
object literals, which no function body covers.

`resolveCalleeSignature` needed a matching branch — the checker hands these back
as BinaryExpression / PropertyAssignment declarations, which `isCallableDecl`
does not cover, so edges were still dropped after the symbol table was correct.
`buildCallable` falls back to `contributorName` for the display name, which was
otherwise "(anonymous)".

Measured on OWASP NodeGoat (deps installed, -a 2):

  callables            24 -> 59   (parser-derived ground truth: 59 nameable)
  tsc resolved         28 -> 51
  tsc edges            30 -> 53
  union edges         161 -> 184
  named graph nodes    32 -> 62   (positional share 75% -> 61%)
  call-site resolution 11% -> 20%

The DAO method layer now appears in the call graph, which it did not before:
  app/routes/allocations.AllocationsHandler.displayAllocations
    -> app/data/allocations-dao.AllocationsDAO.getByUserIdAndThreshold

sample-app analysis.json stays byte-identical to v0.5.0 — no signature churn for
code that already resolved.

Closes #85
@rahlk rahlk changed the title feat(discovery): analyze .js/.jsx/.mjs/.cjs sources feat: analyze JavaScript — discovery, dynamic method idioms, and a loud jelly failure Aug 5, 2026
rahlk added 2 commits August 5, 2026 19:47
briefly(): drop the never-varied limit param, one line instead of four.
isJavaScriptMajority(): regex instead of a JS_EXTS array duplicating SOURCE_EXTS.
Comments cut where they ran longer than the code they explained.

129 -> 113 added lines. No behavior change: 42 tests green, typecheck clean.
- version 0.5.0 -> 0.6.0 in package.json and src/utils/version.ts, in lockstep.
  ANALYZER_VERSION is the only thing that invalidates a cache (utils/cache.ts:24) —
  the per-file source hash cannot see that extraction logic moved, and this release
  extracts more callables from unchanged sources. Verified: a 0.5.0-stamped cache is
  rejected (27 built, 0 cached) where a matching one is reused (0 built, 27 cached).
- CHANGELOG.md, following the codeanalyzer-python house format. The repo had none;
  the release-announcement task in CLAUDE.md already assumed one existed.
- CI on release/0.x, ported from main's ci.yml with the branch filter changed. No
  automated check had ever run on this line — the tag pipeline, which publishes to
  PyPI, GitHub Releases and the Homebrew tap, would have been the first.
rahlk added 2 commits August 5, 2026 21:06
…uage

Node labels carried a TS twin; relationship types carried nothing. A database
holding output from more than one analyzer therefore mingled edges —
codeanalyzer-python already namespaces all 18 of its relationship types
(PY_CALLS, PY_DECLARES, …) while this analyzer emitted bare CALLS/DECLARES.

Now per source language, not per analyzer: a .js module is :Module:JSModule and
a .ts module is :Module:TSModule, and every relationship type is prefixed.

Rules:
- a node with `_module` takes that module's language;
- nodes with no language of their own — application root, packages, external
  library symbols — take the analyzer's own TS namespace, since a sibling
  analyzer emits its own;
- an edge takes its source module's language, falling back to its target's, so
  application->module on a JavaScript project is JS_HAS_MODULE.

Implemented at the two hooks RowBuilder already exposed rather than at the 21
edge call sites: `expand` now sees the node's props, and a new `retype` runs in
finish(), where both endpoints' props are known. REL_TYPES stays the single
source of truth; REL_TYPES_NS derives both namespaces so the catalog and the
projection cannot drift.

Also updates the hand-written traversals in wipe() and DESCENDANTS, which
matched bare types and would have silently deleted nothing.

BREAKING: Neo4j schema version 1.1.0 -> 2.0.0. Stored queries must move from
`[:CALLS]` to `[:TS_CALLS|JS_CALLS]`; the version change forces a full
re-upsert on the next incremental push.
…hip types

The container suite is skipped without Docker, so a local run stayed green while
CI failed: `MATCH (:Callable)-[:CALLS]->` matches nothing now that edges are
namespaced. This is the same migration the CHANGELOG asks consumers to make.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant