Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -496,9 +496,9 @@ Given a quantized vector $V = (v_0, v_1, \ldots, v_{n-1}) \in \{0,1,2,3\}^n$,
**run-reduction** produces a transition sequence by a single left-to-right pass:

```
R (v)
R <- (v[0])
for i in 1..n-1:
if vᵢ ≠ vᵢ₋₁: append vᵢ to R
if v[i] != v[i-1]: append v[i] to R
```

The result $R = (r_0, r_1, \ldots, r_{k-1})$ is the sequence of distinct consecutive
Expand Down Expand Up @@ -572,7 +572,7 @@ distinction is resolved by the Lee-distance re-ranking step.

| b63–62 | b61–60 | b59–58 | b57–56 | b55–54 | b53–52 | … | b5–4 | b3–2 | b1–0 |
|:------:|:------:|:------:|:------:|:------:|:------:|:-:|:----:|:----:|:----:|
| r₀ | r₁ | r₂ | r₃ | r₄ | r₅ | … | r₂₉ | r₃₀ | r₃₁ (LSB) |
| $r_0$ | $r_1$ | $r_2$ | $r_3$ | $r_4$ | $r_5$ | … | $r_{29}$ | $r_{30}$ | $r_{31}$ (LSB) |

---

Expand Down Expand Up @@ -734,7 +734,7 @@ $$S(x) - 1 = \frac{4x}{1 - 3x} = \underbrace{4x}_{S_1} \cdot \underbrace{\frac{1

The first factor $S_1 = 4x$ records the first symbol $r_0$ (4 choices, selecting the block file). The Geode $G = 1/(1-3x) = 1 + 3x + 9x^2 + \cdots$ counts all possible continuations — the tail of the key after the first symbol is fixed.

| Level | Paper | transition key | General quantization |
| Level | Paper | Q2 transition key | General quantization |
|:-----:|:------|:------------------|:--------------------|
| Full structure | $S$ | All transition sequences | All codewords |
| First level | $S_1$ | $r_0$ (first symbol → block file) | Coarse quantization cell |
Expand Down
6 changes: 3 additions & 3 deletions RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This produces components drawn from Uniform[-1, 1]. After L2 normalisation of a
E[‖u‖²] = n/3 = 128/3, so each normalised component follows approximately:

```
v_i u_i / (n/3) Uniform[−√(3/n), (3/n)] Uniform[0.153, 0.153]
v_i ~ u_i / sqrt(n/3) -> Uniform[-sqrt(3/n), sqrt(3/n)] ~ Uniform[-0.153, 0.153]
```

The quantisation threshold is τ* = Φ⁻¹(¾)/√n ≈ 0.6745/√128 ≈ 0.0596, which
Expand All @@ -49,7 +49,7 @@ producing systematically skewed symbol probabilities:
With 500 trials × 128 dimensions = 64,000 total symbols, the predicted χ² is:

```
χ² = 4 × (3520² / 16000) 3098
chi2 = 4 * (3520^2 / 16000) ~ 3098
```

This matches the observed 3127.86 to within rounding of the approximated marginal
Expand All @@ -74,7 +74,7 @@ The benchmark was corrected to generate pre-normalisation components from N(0, 1
using Box-Muller, matching the Gaussian assumption under which τ* was derived:

```ts
// Box-Muller: pairs of uniform samples standard normal pairs
// Box-Muller: pairs of uniform samples -> standard normal pairs
for (let i = 0; i < n; i += 2) {
const u1 = Math.random(), u2 = Math.random();
const r = Math.sqrt(-2 * Math.log(u1));
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"lint": "eslint --no-inline-config --max-warnings 0 --fix src test --ext .js,.ts,.html,.yml,.yaml && eslint --no-inline-config --max-warnings 0 src test --ext .js,.ts,.html,.yml,.yaml",
"lint:css": "stylelint --max-warnings 0 --allow-empty-input --fix \"**/*.{css,html}\" && stylelint --max-warnings 0 --allow-empty-input \"**/*.{css,html}\"",
"lint:md": "bun scripts/lint-md.mjs",

Copilot AI Mar 21, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that check runs lint:md in CI, note that scripts/lint-md.mjs (per its current default behavior) only lints *.md in the repo root. This means Markdown files under docs/ (and other subdirs) will still bypass CI linting unless they’re passed explicitly. Consider updating the linter’s default file discovery (or the lint:md script) to include all tracked Markdown files (e.g., recursive walk or git ls-files '*.md').

Suggested change
"lint:md": "bun scripts/lint-md.mjs",
"lint:md": "bun scripts/lint-md.mjs $(git ls-files \"*.md\")",

Copilot uses AI. Check for mistakes.
"check": "bun run lint && bun run typecheck",
"check": "bun run lint && bun run lint:md && bun run typecheck",
"prebuild": "bun run check && bun run lint:css",
"build:wat": "wat2wasm src/q2.wat -o src/q2.wasm && bun run embed-wat",
"embed-wat": "bun ./scripts/embed-wat.mjs",
Expand Down
100 changes: 87 additions & 13 deletions scripts/lint-md.mjs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bun
/**
* lint-md.mjs — Lints Markdown files for encoding issues that break GitHub
* rendering of KaTeX math and Mermaid diagrams.
* rendering of KaTeX math, Mermaid diagrams, code blocks, and tables.

Copilot AI Mar 21, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file header now says this linter covers “code blocks, and tables”, but the implemented rules only scan display-math ($$...$$), Mermaid fenced blocks, and non-Mermaid fenced code blocks. Unicode rendering issues inside Markdown tables (outside fenced blocks) still won’t be detected. Either adjust the description to match current behavior, or add a table/prose scanning rule if table linting is intended.

Suggested change
* rendering of KaTeX math, Mermaid diagrams, code blocks, and tables.
* rendering of KaTeX math, Mermaid diagrams, and code blocks.

Copilot uses AI. Check for mistakes.
*
* Checks performed:
* 1. Emoji characters (U+1F000+) inside LaTeX $...$ or $$...$$ blocks —
Expand All @@ -13,6 +13,12 @@
* support and silently corrupt Mermaid output.
* 4. Unicode MINUS SIGN (U+2212) anywhere in Mermaid blocks — diagram
* labels should use ASCII hyphen-minus.
* 5. Unicode subscript/superscript digits (U+2070–U+209F), modifier
* letters (U+1D00–U+1D9F), mathematical arrows (U+2190–U+21FF), and
* mathematical operators (U+2200–U+22FF) inside fenced code blocks —
* monospace fonts often lack these glyphs.
* 6. (Emoji in prose is intentionally allowed — only emoji inside LaTeX
* math or Mermaid blocks is flagged, as it breaks rendering.)
*
* Usage:
* bun scripts/lint-md.mjs [file.md ...] # lint specific files
Expand All @@ -31,17 +37,6 @@ const root = join(__dirname, '..');

// ── helpers ──────────────────────────────────────────────────────────────────

/** Split content on $$ boundaries; odd-indexed parts are display math. */
function displayMathBlocks(content) {
const parts = content.split('$$');
const blocks = [];
for (let i = 1; i < parts.length; i += 2) {
const start = parts.slice(0, i).join('$$').length + 2; // byte offset approx
blocks.push({ text: parts[i], partIndex: i });
}
return blocks;
}

/** Extract the content and approximate line number of each ```mermaid block. */
function mermaidBlocks(lines) {
const blocks = [];
Expand All @@ -63,6 +58,30 @@ function mermaidBlocks(lines) {
return blocks;
}

/** Extract the content and line numbers of non-Mermaid fenced code blocks. */
function fencedCodeBlocks(lines) {
const blocks = [];
let inside = false;
let isMermaid = false;
let buf = [];
for (let i = 0; i < lines.length; i++) {
const trimmed = lines[i].trim();
if (/^(`{3,}|~{3,})/.test(trimmed)) {
if (!inside) {
inside = true;
isMermaid = trimmed === '```mermaid';
buf = [];
} else {
if (!isMermaid) blocks.push(buf);
inside = false;
}
} else if (inside) {
buf.push({ text: lines[i], lineNo: i + 1 });
Comment on lines +66 to +79

Copilot AI Mar 21, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fencedCodeBlocks() treats any line starting with backticks/tildes as a close fence, without verifying it matches the opening fence character (``` vs ~~~) and length. This can mis-parse valid Markdown that uses different fence lengths/chars (or includes a shorter fence inside a longer fenced block), leading to missed or spurious lint violations. Track the opening fence marker (char + count) and only close when encountering the same marker with >= the opening length (optionally allowing trailing spaces).

Suggested change
let buf = [];
for (let i = 0; i < lines.length; i++) {
const trimmed = lines[i].trim();
if (/^(`{3,}|~{3,})/.test(trimmed)) {
if (!inside) {
inside = true;
isMermaid = trimmed === '```mermaid';
buf = [];
} else {
if (!isMermaid) blocks.push(buf);
inside = false;
}
} else if (inside) {
buf.push({ text: lines[i], lineNo: i + 1 });
let fenceChar = '';
let fenceLength = 0;
let buf = [];
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
const trimmed = line.trim();
if (!inside) {
// Opening fence: capture marker and info string.
const openMatch = trimmed.match(/^(`{3,}|~{3,})(.*)$/);
if (openMatch) {
inside = true;
const marker = openMatch[1];
fenceChar = marker[0];
fenceLength = marker.length;
const info = openMatch[2].trim();
// Treat as Mermaid if info string is "mermaid" or starts with "mermaid ".
isMermaid = info === 'mermaid' || info.startsWith('mermaid ');
buf = [];
}
} else {
// Potential closing fence: must match opening char and have length >= opening.
const closeMatch = trimmed.match(/^([`~]{3,})\s*$/);
if (closeMatch) {
const closeMarker = closeMatch[1];
if (closeMarker[0] === fenceChar && closeMarker.length >= fenceLength) {
if (!isMermaid) blocks.push(buf);
inside = false;
isMermaid = false;
fenceChar = '';
fenceLength = 0;
continue;
}
}
buf.push({ text: line, lineNo: i + 1 });

Copilot uses AI. Check for mistakes.
}
}
return blocks;
}

/**
* Compute 1-based line number of a character offset within content.
* Used to report line numbers for math-block violations.
Expand Down Expand Up @@ -116,7 +135,7 @@ function checkDisplayMath(content, filePath) {
}

/**
* Rule 2 & 3: scan Mermaid blocks for disallowed Unicode.
* Rule 3 & 4: scan Mermaid blocks for disallowed Unicode.
*/
function checkMermaid(content, filePath) {
const lines = content.split('\n');
Expand Down Expand Up @@ -163,6 +182,60 @@ function checkMermaid(content, filePath) {
return violations;
}

/**
* Rule 5: scan fenced code blocks for Unicode characters that do not render
* reliably in monospace fonts. Catches subscripts, superscripts, modifier
* letters, mathematical arrows, and mathematical operators.
*/
function checkCodeBlocks(content, filePath) {
const lines = content.split('\n');
const violations = [];

for (const block of fencedCodeBlocks(lines)) {
for (const { text, lineNo } of block) {
for (let j = 0; j < text.length; ) {
const cp = text.codePointAt(j);
const advance = cp > 0xffff ? 2 : 1;

// Unicode subscript/superscript digits & letters:
// U+2070–U+209F (superscripts and subscripts)
// U+1D00–U+1D9F (phonetic/modifier letters used as subscripts)
if (
(cp >= 0x2070 && cp <= 0x209f) ||
(cp >= 0x1d00 && cp <= 0x1d9f)
) {
violations.push({
file: filePath, line: lineNo,
message: `Unicode subscript/superscript U+${cp.toString(16).toUpperCase()} ('${String.fromCodePoint(cp)}') in code block — use plain ASCII instead`,
});
}

// Mathematical arrows (U+2190–U+21FF): ← → ↑ ↓ etc.
if (cp >= 0x2190 && cp <= 0x21ff) {
violations.push({
file: filePath, line: lineNo,
message: `Unicode arrow U+${cp.toString(16).toUpperCase()} ('${String.fromCodePoint(cp)}') in code block — use ASCII equivalent instead`,
});
}

// Mathematical operators (U+2200–U+22FF): ≠ ≤ ≥ etc.
if (cp >= 0x2200 && cp <= 0x22ff) {
violations.push({
file: filePath, line: lineNo,
message: `Unicode math operator U+${cp.toString(16).toUpperCase()} ('${String.fromCodePoint(cp)}') in code block — use ASCII equivalent instead`,
});
}

j += advance;
}
}
}
return violations;
}

// Emoji in prose is intentionally allowed (only emoji inside LaTeX math or
// Mermaid blocks is flagged by the block-specific rules above).

// ── main ──────────────────────────────────────────────────────────────────────

const args = process.argv.slice(2);
Expand All @@ -187,6 +260,7 @@ for (const filePath of files) {
const violations = [
...checkDisplayMath(content, filePath),
...checkMermaid(content, filePath),
...checkCodeBlocks(content, filePath),
];

for (const { file, line, message } of violations) {
Expand Down
Loading