feat(vba): encoding robustness — UTF-8 BOM strip and Windows-1252 fallback for VBA files

[ardelperal]

Priority

P3 — defense in depth; dysflow normally guarantees UTF-8.

Context

Access SaveAsText / VBIDE Export write Windows-1252 (ANSI). dysflow converts everything to UTF-8-no-BOM on export (Convert-AnsiToUtf8NoBom in scripts/dysflow-vba-manager.ps1) and offers a Fix-Encoding action. But codegraph-vba can be pointed at exports produced by other tools, older dysflow runs, or hand-copied modules that are still CP1252 on disk.

Current behavior

Files are read as UTF-8 unconditionally. A CP1252 file with accented identifiers — ubiquitous in Spanish-language Access code (Sub ActualizarSituación, enum member Sí, module headers ' MÓDULO:) — decodes to replacement characters, \p{L} regexes stop matching, and the symbol silently disappears from the graph. A BOM-carrying UTF-8 file additionally corrupts the first line (breaks Attribute VB_Name detection → wrong module name).

Expected behavior

• UTF-8 BOM stripped before extraction.
• When a .bas/.cls/.form.txt/.report.txt file's bytes are invalid UTF-8, fall back to decoding as Windows-1252 (cheap sniff: TextDecoder('utf-8', { fatal: true }) throws → re-decode as windows-1252, which Node's TextDecoder supports).
• Optionally surface a low-severity ExtractionError/warning noting the fallback so users learn to normalize.

Implementation sketch

• Locate the single point where VBA-routed file content is read (extraction orchestration / file-read helper) and wrap the decode for VBA-family extensions only — do NOT change decoding for tree-sitter languages.
• BOM strip: if (text.charCodeAt(0) === 0xFEFF) text = text.slice(1).
• Unit tests with fixture buffers: CP1252 bytes for Sub Función(), UTF-8-BOM file, plain UTF-8 (unchanged path).

Acceptance criteria

• CP1252 file yields the same nodes as its UTF-8 twin.
• UTF-8-BOM file resolves Attribute VB_Name correctly.
• Valid UTF-8 path byte-identical to today (no behavior change, no perf cliff — sniff only on decode failure).

Validation

Re-index C:\Proyectos\dysflow\E2E_testing\src (all UTF-8): zero delta. Corrupt one copy to CP1252 in a temp fixture and verify parity.

[ardelperal]

Re-cierre con evidencia (2026-07-04): commit 8b6b093 "feat(vba-extractor): UTF-8 BOM strip and Windows-1252 fallback for VBA files (closes #53)" (PR #71) cubre el hardening del read-path para archivos VBA-family. Helper nuevo src/extraction/vba-source.ts con isVbaFamilyFile / readVbaSource (Buffer + BOM strip + UTF-8 fatal sniff + fallback CP1252 via TextDecoder windows-1252) y stripUtf8Bom defensivo. Wire-up en 7 read sites de src/extraction/index.ts (batch 1412, retry 1524, comment-strip 1568, single-file 1688, sync reconcile 1983, git fast-path 2067, full-scan 2112) via readForExtraction / readForExtractionSync que ramifica por isVbaFamilyFile. 8 tests unit en tests/vba-source.test.ts (BOM strip, CP1252 fallback, onFallback callback, empty buffer, UTF-8 happy path, isVbaFamilyFile truth table) usando DI readFile para buffers in-memory. TDD: tests/vba-source.test.ts + verificacion de no-regresion en suite VBA completa. Verificacion: pnpm exec vitest run tests/vba-source.test.ts (8/8 verde en 403ms), suite VBA 6 archivos (236/236 verde en 5.03s, cero regresiones), pnpm run build OK, CI Node 22 ubuntu/windows verde tras re-run (1m52s/1m15s ubuntu, 4m39s/3m7s windows). Scope disciplinado: solo archivos VBA-family reciben BOM strip + CP1252 fallback; tree-sitter-routed languages (TS/JS/Python/Go/Rust/etc.) mantienen readFile(path, 'utf-8') byte-identical. PR #71 squash a release-origin/main (8b6b093). Tambien cerrado flake de windows CI en main: PR #72 (fix(tests): raise vitest testTimeout to 30s on the actively-loaded .mts config) merged como f354e34, previo PR #70 fue no-op efectivo porque vitest.config.ts estaba shadowed por el .mts existente.