feat(tokens): read SCSS variables as a project token source - #151
Merged
Conversation
`scss` has been a detected styling system since before the token join existed,
but every source was CSS — so a SCSS project joined against an empty pool and
token_map returned nothing for it.
The reference form is what made this more than a third parser. Verified against
dart-sass rather than assumed: under a plain `@use './tokens'` — the modern
default — the bare `$color-primary-500` is an undefined-variable error, and the
reference is `tokens.$color-primary-500`. The namespace comes from the
*consuming* file, not the declaring one. Both styles are live in real projects:
Bootstrap's 952-entry `_variables.scss` is consumed through `@import`, Vuetify's
per-component ones through a namespaced `@use`.
Codegen writes the consuming file, so it can choose — but only if it is told
which file to import. So a SCSS token carries `from`, and `TokenMapping
.candidate.from` surfaces it, the same shape as icon_map returning an svg's path
rather than fabricating an import specifier. The tool description, the codegen
prompt and the figma-codegen skill all say the ref is not self-sufficient:
emitting it without the `@use` is a compile error, not a style nit.
ProjectToken's ref union gains a third arm for it, keeping refOf total by
construction. That generalization is also what Panda and DTCG were blocked on.
Both kinds of declaration in a `.scss` file are read, because modern projects
use both and reading one leaves half the project joining against nothing: `$name`
variables (which need the `@use`) and `:root { --name }` custom properties (which
compile through untouched, so `var(--name)` resolves with no import). No utility
or category is derived — SCSS generates no classes, and a namespace-shaped stem
would put `bg-primary-500` back into circulation on a project that has none.
Variables declared inside a rule are dropped: Sass scopes them to that block, so
referencing one from a generated component does not compile.
The scanner is generalized rather than copied — one implementation, a dialect
per language. SCSS adds `//` line comments and `!default` / `!global` flags; a
brace inside a `//` comment would otherwise open a block that never closes, and
every later variable would be tagged with a bogus scope and dropped.
Verified: 13/13 ref+import pairs compile through dart-sass across both `@use`
styles; a real-file corpus (Bootstrap 952 vars, Vuetify, Bulma, Foundation)
parses with every invariant clean at ≤2.2ms per file, and the one file that is
self-contained round-trips 65/65 refs — the rest fail to compile standalone at
all, having been lifted out of their repos. Live end-to-end against a real Figma
file: 0 tokens before, 6 tokens and 6 high-confidence matches after, each
carrying its declaring file.
…and pool a SCSS project's CSS Two gaps found reviewing the branch, the first the same class of miss as the icon_map one in #149 — fix the forward join, forget the other surface. **`get_design_context`'s `projectTokens` annotation dropped `from`.** It emitted `$color-primary-500` with no way to know which file to `@use`, which is exactly the compile error the rest of this branch exists to prevent. That annotation is the surface a caller reads when the document has no bound variables to join — the commoner case, not the rarer one — so this was the more likely of the two paths to be hit. `ProjectTokenMatch` gains an optional `from` (additive on the wire) and `toMatch` carries it. **A SCSS project's `.css` files were not pooled.** The JS-config path pools the config with the repo's CSS; the SCSS path returned only its `.scss` pool, so a project keeping Sass variables in `.scss` and a global `:root` block in a plain `.css` file got half its tokens. Both walks are already aggregations, so pooling does not turn a precise source into a fuzzy one — the argument that kept the Tailwind v4 entry unpooled does not apply here. Also verified while reviewing, and worth recording: the dialect generalization touched the scanner every project's CSS path runs through, so its CSS output was differentially compared against main over the repo's own stylesheets plus the shapes the scanner's comments name as its historical failures (minified trailing declaration, `/*` inside a url string, `//` which is *not* a comment in CSS, unterminated string and comment). Byte-identical across all of them.
… join behaviours `get_design_context`'s description spells out the projectTokens annotation's shape for agents, and still described it without `from` after the previous commit added it — so the surface that actually carries a SCSS ref in the common case (no bound variables to join) documented a ref with no way to make it resolve. Two behaviours verified while reviewing and now asserted, both previously untested: - `from` follows the token that *matched*, not the first one seen. Per-component variable files repeat names across files — Vuetify ships ~90 — so a value-match landing on the second file must carry that file. Pointing the `@use` at a file that does not declare the given value is a compile error that reads as a token problem. - A `docs/figma-token-map.md` row resolves against a SCSS variable written with or without its sigil, and the resulting candidate still carries `from`. A recorded override that came back unusable would silently degrade the one mechanism meant to be authoritative. Also checked, no change needed: the walk excludes vendored (`node_modules`), gitignored and `.sass` files; `ProjectTokenMatch` has no plugin-side consumer, so adding an optional field is additive on the wire.
All seven were verified against dart-sass before and after; five would have
produced a wrong or missing token on ordinary real-world input.
**`//` inside an unquoted `url()` was read as a line comment.** Stylesheets are
full of `url(http://…)` and `url(//cdn…)`, which Sass parses as raw text. Read
as a comment, the phantom comment ate the rest of the line including its `)` and
`;`, so the enclosing rule never closed and every later variable was tagged
rule-scoped and dropped. `.hero { background: url(http://cdn.io/a.png); }`
followed by any `$var` returned **nothing at all**, and
`@import url(https://fonts.googleapis.com/…)` — a common first line — did the
same. As a value it was worse: `$cdn: url(//cdn.io/a.png);` produced
`url($other: 4px;`, a value that had swallowed the next declaration, which is
precisely the failure the scanner's own header says it exists to prevent.
**The CSS half of a `.scss` read used the plain-CSS dialect**, which has no `//`
comments — so `:root { // brand \n --brand: … }` silently dropped `--brand`, and
a `//` comment containing a `}` dropped the whole file. The branch's own test
asserts this case for `$variables`; the custom properties beside them had no
such protection.
**The `@use` instruction was wrong for any consuming file not at the repo root.**
`from` is repo-relative, but Sass resolves `@use` against the *importing* file.
Verified: from `src/components/card.scss`, `@use 'src/styles/_tokens.scss'`
fails with "Can't find stylesheet to import". Every surface — the note, both
tool descriptions, the codegen prompt and the skill — told the model to emit
that path verbatim, so following the guidance produced exactly the compile error
the `from` mechanism was added to prevent. All five now say the path is
repo-relative and must be re-resolved from the file being written.
**A name-only match reported the first file at full confidence.** `bestNameMatch`
dedupes by name, which was free while repeats differed only in value and shared
a ref — but a SCSS ref resolves *through* its file, so two files declaring
`$radius-lg` yielded the wrong file and the wrong value at `confidence: 1`,
`status: 'high'`. It is now capped to the same "verify me" level the
value-ambiguous path uses, rather than inventing a winner.
**Pooling the two walks could report a token ambiguous with itself.** A `:root`
block in a `.scss` file and the compiled `.css` committed beside it are one
declaration seen twice; left duplicated, an exact name+value hit degraded to
`medium` with `ambiguousWith` naming the token itself. Deduped on full identity,
so two *different* files declaring one name are still kept apart.
Two smaller ones: `!default !global` on one declaration (legal Sass) left
`!default` in the value, and a top-level `#{…}` interpolation truncated the
value at the interpolation's `}`.
The CSS path remains byte-identical to main — re-proven differentially over the
repo's own stylesheets plus every shape above. That check caught one of my own
fixes overreaching: making the flag strip repeat had also changed CSS output for
`!important !important`, so the CSS dialect keeps its single strip (a second
`!important` is invalid CSS; repeating is a SCSS-only need).
…tion from hiding scope
Found by a new kind of check rather than another read-through: dart-sass's own
`meta.module-variables()` is an authoritative list of what a file declares, so
comparing this reader against it over whole cloned repositories (Bootstrap and
Bulma, 146 files) answers "did we get this file right" without anyone having to
imagine the syntax first. Every SCSS defect so far had been the same shape — a
real-world construct the parser had never been shown — which unit tests cannot
find, because they only contain syntax someone already thought of.
It disagreed in three places, all in the direction that matters (a name emitted
that the compiler says does not exist), and they were two classes:
**Private variables were emitted.** Sass treats a leading `-` or `_` as private:
the member is not exported, `meta.module-variables()` does not list it, and
`@use`-ing the file and naming it is an error. Bootstrap's `$_luminance-list` is
one. A ref to it cannot resolve from any other file — exactly what this reader
exists not to produce.
**Interpolation in a selector hid the enclosing scope.** In
`.table-#{$state} { $color: … }` the interpolation's braces are not block
braces; read as structure they open and close a phantom block, so the real
rule's contents looked module-level. Bootstrap's mixin-local `$color`,
`$hover-bg`, `$striped-bg` and friends were emitted as project tokens. Values
already handled interpolation; preludes did not.
With both fixed the oracle reports full agreement across all 146 comparable
files. The two remaining differences before the fix were the only ones in the
corpus, so this closes the class rather than another instance of it.
A review agent recreates scratch probes in the test tree while it runs, and a `git add -A` between two of its writes picked one up. It has no assertions, so CI's lint gate caught it — the right outcome, but the file should never have reached the commit. Added to .git/info/exclude so the pattern cannot repeat.
…stop a fileless token capping a match
Five findings from review, three of them one root cause: the *mirror* layout —
one file declaring both forms of the same token — which this reader set out to
support and never tested as a single token.
$brand: #6266F0;
:root { --brand: #{$brand}; }
That is one logical token with two reference forms, not two tokens. Left as two,
an exact name+value hit degraded to `medium` and reported the token
`ambiguousWith` *itself* — the failure `dedupeTokens` exists to prevent, which it
could not catch because its key included the reference form that differs. The
pair now collapses, and the custom property wins: `var(--brand)` compiles from
any consumer with no import at all, while `$brand` needs an `@use` resolved
against the consuming file. Preferring the self-sufficient ref is the point of
having a choice. Two *different* .scss files declaring one name are still kept
apart — those are genuinely different declarations.
That also fixes the annotation handing codegen the import-requiring ref while
discarding the import-free one, which was the same duplication seen from the
value index.
**A pooled custom property has no declaring file, and its absence counted as
"a different file"** — so `fileAmbiguous` capped every name-only match on the
mirror layout, where exactly one file declares the name. Only another
file-bound token can make that choice ambiguous.
Two smaller ones: the aggregate note counted the raw walks rather than the
collapsed result, so a repo committing its compiled .css claimed more tokens
than it was handed; and an explicit `tokenSource` pointing at a `.sass` file
fell through to the CSS reader, returned nothing, and then dropped into the repo
pool under a note that never mentioned the file the caller asked for — it now
says the indented syntax is not readable here.
…ix had missed The previous commit collapsed the mirror layout on the aggregate path and left the explicit `tokenSource` path reading the same file without it — the same half-fix shape as the icons-only detection earlier in this branch: fix the path I was looking at, leave the sibling. It matters more here, not less. A caller passes `tokenSource` to narrow a large repo to the file that declares its tokens, which is exactly the file that carries the mirror, so `token_map --tokenSource src/_tokens.scss` was the most likely way to hit the self-ambiguity. Also pins two properties of the collapse that had no test: a mirror whose two forms hold *different* values stays two tokens (two real declarations the value-match join needs), and the collapse is order-independent.
…w findings
The headline one: the mirror fix did not work on the form its own docstring uses
as the example. `:root { --brand: #{$brand} }` gives the custom property the
literal value `#{$brand}` while `$brand` holds `#6266F0`, so a repo-wide
name+value fold never fired for it — only for the literal spelling, which is
what every test used. I wrote the example and tested something else.
Fixed at the level the idiom actually lives at: `parseScssFile` reads one file's
two halves together, resolves a custom property whose whole value is `#{$var}`
against the variables declared beside it, and folds the pair there. That also
removes the repo-wide fold, which could let a `--x` in an unrelated stylesheet
displace a genuinely declared `$x` from another file, and makes the interpolated
custom property carry a value that can value-match at all.
The rest:
- An explicit `.scss` `tokenSource` that is a barrel (`main.scss` =
`@use './tokens';`) returned an empty pool with no diagnostic — SCSS's
commonest entry shape, and omitting tokenSource entirely would have worked. It
now falls back to the pool and says why, as the CSS branch beside it already did.
- The `.sass` refusal note was computed and then dropped whenever a pool was
found, i.e. on every real repo — the exact "note never mentions the file the
caller asked for" the guard was added to prevent. Refusals now ride along.
- A custom property declared inside a `@mixin` or `@function` body became a
project token. It only exists where that mixin is included, so `var(--x)`
resolves to nothing in a component that never includes it — the asymmetric
twin of the rule-scoped `$var` case this branch already fixed.
- Two files declaring one name+value listed the candidate as `ambiguousWith`
its own name, which reads as a data error and tells the caller nothing. Those
are the same token in two files, distinguished only by `from`; the confidence
cap stays, the useless alternative is gone. A same-value sibling with a
*different* name is still listed, which is what the field is for.
- A map-file override resolving to several file-bound tokens returned an
arbitrary file at confidence 1. A recorded ref cannot name a file, so that
choice is this join's, not the author's — capped like the name-only path. One
file, and a recorded mapping keeps the certainty it earns.
- `dedupeTokens` had a branch that could never fire, since the two key spaces
were disjoint; with mirroring moved per-file it reduces to the cross-walk
duplicate it was originally for.
…ected
Found by measuring the SCSS read on a real 122-file project (Bootstrap) rather
than a fixture — 57ms cold, 1192 tokens, and a note that opened with:
no token source detected; pass tokenSource; aggregated 1192 token(s) from
22 .scss file(s): …
A sentence contradicting itself in its own second clause. `resolveTokenSource`
returns that phrase for any project with no config file to find, which is every
SCSS project — there is no such thing to detect there — and the fallback
forwarded it unconditionally alongside its own description.
The two are different kinds of message and now have different fields: `refusal`
answers what the *caller asked for* (a `.sass` tokenSource this cannot read) and
must survive into whatever the loader falls back to, while `note` describes the
project and is replaced by whatever actually read it.
Measured while there and left alone: the walk is 57ms cold on Bootstrap and the
annotation index caches to 0ms warm, so the extra per-project scan this branch
adds is not a cost worth complicating the code for.
…lose two value defects
Four findings, one of which is a repeat of a bug I fixed earlier on this same
branch.
**scss-file.ts was a binary file.** I wrote the dedup separator as a literal NUL
instead of the escape every sibling module uses — the exact mistake I fixed in
token-index.ts a few commits ago, made again in a file added while fixing it.
Git classifies the whole module as binary, so `git diff main...HEAD` renders the
core new logic of this PR as "Binary files differ", and every future diff, blame
and grep on it is dead. Nothing in CI caught it: the runtime string is identical,
so typecheck, lint and the tests all passed.
Twice is a pattern, so there is now a test: no `.ts` file under src or test may
contain a raw control byte. It catches a reintroduced NUL, and it caught its own
prose embedding the byte it warns about, which is how I found that the first
draft of the guard was itself binary.
**`sass-embedded` was not detected as SCSS.** It is the compiler Vite documents
alongside `sass` and the common choice on a modern Vite/Vue/Nuxt project, so the
whole SCSS token source silently never ran on exactly the projects it was built
for — `system: 'unknown'`, every Figma variable unmapped, and a note that never
mentions SCSS.
**Resolving an interpolation could newly create duplicates.** `--brand: #{$brand}`
in `:root` and a literal `--brand: #6266F0` in a theme class are one token once
resolved, but the CSS parser deduped *before* resolution, so both survived and
the join reported the token ambiguous with itself — on the explicit `tokenSource`
path, which is the one most likely to be pointed at a mirrored file.
**`!important` was left in SCSS values.** It is legal in a Sass variable value,
and `CssDeclaration.value`'s own contract says it is stripped; leaving it made
`$spacer: 1rem !important` unable to match its Figma counterpart or its
`--spacer: 1rem` twin, which is the reason the CSS arm strips it.
…in every cap
Six findings, one of them a regression against this server's own previous
behaviour — the bar I set for this whole line of work.
**The cross-file mirror was a regression.** `_vars.scss` for the build-time Sass
variables plus a hand-written `global.css` `:root` block for runtime theming,
same palette, is a standard SCSS layout. Keeping both halves cost such a project
twice: an exact hit degraded from 'high' to 'medium' (the join saw two same-value
tokens and could not split them), and the surviving ref flipped to `$primary`,
which needs an `@use` codegen must invent — where reading only the CSS, which is
what this server did before SCSS was a source at all, returned `var(--primary)`
at full confidence. `parseScssFile` already stated the right rule ("the custom
property wins, because var(--brand) compiles from any consumer with no import");
it just was not applied across files. Now it is.
**And the same rule was too eager within a file.** A `--brand` declared under
`.theme` resolves to nothing outside that selector, yet it displaced the
`$brand` that is referenceable anywhere through `@use` — losing the only usable
ref. Only a document-wide declaration may stand in for a variable now, which is
the trade the docblock actually describes.
**A cap that explains nothing is worse than no cap.** Two files declaring one
name+value produced `status: medium` with no field saying why, so `from` read as
a resolved answer rather than one of several candidates — and `ambiguousWith`
cannot express it, since it carries token *names* and these siblings share one.
`candidate.ambiguousFrom` now lists the other declaring files, on all three
paths that cap (value, name and map-file), and the tool contract says so.
Three smaller ones: the `.sass` refusal was still dropped on the one path where
nothing at all was found; a project with zero `.scss` files got a note reading
"from 0 .scss file(s): and 1 .css file(s)"; and that same note attached the
`@use` instruction to a pool where no token carries a declaring file — telling
the model to import something for a `var()` that needs nothing, in the file it
then writes.
Re-verified after the changes rather than citing the earlier runs: dart-sass
oracle 146/146 files in agreement, 395 emitted refs compile with 0 failures, and
the CSS path is still byte-identical to main across 16 inputs.
awdr74100
added a commit
that referenced
this pull request
Aug 16, 2026
`walkRepoFiles` fed five callers an order that changed between runs. fdir crawls directories concurrently, so the same unchanged repo returns the same *set* in a different sequence each time — measured, not inferred: two consecutive runs over Bulma disagreed on the first three files. It reached the output. `token_map`'s note sampled a different six files each call, and on a repo where several files declare one token name the `from` handed back changed run to run — pkg0 on one call, pkg1 on the next, same input. That matters more since #151, because `from` is now the file a caller must `@use`, and it is the input to any future comparison of one run against another. The walk already buffers the whole crawl before yielding, so ordering it costs one sort and no change in memory or streaming behaviour. Shallowest-first, then by code unit. Depth leads as a tie-break *preference*, not as a repair: one caller takes the first match rather than aggregating (`findTailwindCssEntry`), and a v4 entry is conventionally shallow, so ranking `src/index.css` above `src/a/b/deep.css` is the better guess where plain a-z would invert them. Stated as a preference because no fixture made that caller vary between runs — three tries, including 300 packages of decoys — so its stability here is a by-product, not evidence of a bug fixed. Code units rather than `localeCompare`, since locale-aware collation is environment-dependent, which is the property this change exists to remove. Never worse, checked at the level a caller consumes rather than the level that changed: over Bootstrap, Bulma and vue/core, every joined mapping is identical to main's — no status change, no ref change, no pool-size change. Each of the five consumers is byte-stable across three runs, and each returns the same content as main as a set. Four mutations pinned: no sort, plain a-z, locale collation, and the ordering tests' own blind spot — the existing suite's helper sorted its results, which is why none of its seven tests could see the order the walk actually produced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third source after Tailwind v3 (#148) and UnoCSS (#149), and the one that forced the ref model to generalize.
The gap
scsshas been a detected styling system since before the token join existed, but every token source was CSS — so a SCSS project joined against an empty pool andtoken_mapreturned nothing for it. Same for the design-context value annotation, which shares the loader.What made this different: the ref is not self-sufficient
I ranked SCSS as the easy next one on the grounds that
$color-primary-500is an unambiguous reference. The compiler says otherwise, and I checked before designing rather than after:The namespace comes from the consuming file, not the declaring one. Both styles are live in real projects: Bootstrap's 952-entry
_variables.scssis consumed through@import, Vuetify's per-component ones through a namespaced@use.Codegen writes the consuming file, so it can choose — but only if it is told which file to import. So a SCSS token carries
from, andTokenMapping.candidate.fromsurfaces it. This is the shapeicon_mapalready uses: return the svg's path, don't fabricate an import specifier. The tool description, the codegen prompt and thefigma-codegenskill all state that emitting the ref without the@useis a compile error, not a style nit.ProjectToken's ref union gains a third arm, keepingrefOftotal by construction. That generalization is also what Panda and DTCG were blocked on.Decisions worth reviewing
.scssfile are read.$namevariables (need the@use) and:root { --name }custom properties (compile through untouched, sovar(--name)resolves with no import). Modern SCSS projects use both; reading one would leave half the project joining against nothing — an incomplete fix I'd have to come back for.utilityorcategoryis derived. SCSS generates no classes, and a namespace-shaped stem would putbg-primary-500back into circulation on a project that has none.//line comments and!default/!globalflags. A brace inside a//comment would otherwise open a block that never closes, tagging every later variable with a bogus scope so the caller drops it..sass(indented syntax) is deliberately not walked. Its declarations are newline-terminated, which the value reader would run straight past — a token whose value swallows the following lines is worse than no token.Verification
@usestyles, for every token the reader emits.rgba($black, .15), and multi-line maps. The one self-contained file round-trips 65/65 refs; the rest fail to compile standalone at all, having been lifted out of their repos — a harness limit, not a reader defect, and I verified that distinction rather than assuming it.spacing/4correctlyunmappedrather than a fabricatedp-4(SCSS is not utility-first).from, and my//-comment test passed either way because the prelude guard happened to save it. Both closed, both now fail on mutation.Full gate green: typecheck · lint · format:check · knip · build · test (1614).