Backlog: open findings after v0.3.1

[Warhorze]

Consolidating all known open findings into one place post v0.3.1. Full detail for each is in ARCHITECTURE_REVIEW.md.

Next sprint (HIGH)

Finding 6 — Star projection column lineage (no edges for SELECT *)
sqlglot.optimizer.qualify can expand SELECT * into explicit columns when schema is available. Requires a two-pass index (DDL first → populate SchemaResolver → ETL pass uses qualify before sg_lineage). This is the biggest remaining gap for column lineage coverage.
Closes #5 when done.

Finding 11.6 — Indexer too slow for real corpora
index_repo is single-threaded. Parsing is CPU-bound (can't use threads — GIL). Needs ProcessPoolExecutor for parallel file parsing + bulk DB commits (N=50) to replace per-file fsync round-trips. Expected ~6-7x throughput gain on 8-core hardware. Implement after finding 6 (SchemaResolver must be pickle-safe first).

Medium priority

Finding 11.4 — Parse warnings go to stdout (#13)
Redirect WARNING log output to ~/.sqlcg/index.log by default; print only a count summary to stdout. Add --verbose to restore current behaviour.

Finding 11.7 — E8 dynamic identifier marker (observability only)
col_lineage_skip:dynamic_source: marker is already emitted (v0.3.1). No further action unless corpus gains statically-resolvable variable patterns.

Low priority / deferred

Finding 12 — Case-sensitive table lookup (#12)
Uppercase unquoted identifiers at index time and normalise lookup input. sqlglot dialect normalization likely handles most of this.

Finding 3.3 — SchemaResolver not thread-safe under sqlcg watch
Add threading.Lock around cache mutation in SchemaResolver. Blocked on finding 6 (resolver redesign anyway).

[Warhorze]

Sprint-05 update

Finding 6 (star projection) — DONE, landing in PR #18.

What shipped:

• DDL CREATE TABLE now persists HAS_COLUMN edges + SqlColumn nodes (source='ddl')
• Parser emits StarSource markers for SELECT * statements
• Indexer upserts STAR_SOURCE edges and runs post-indexing expansion Cypher to create COLUMN_LINEAGE {transform: 'STAR_EXPANSION', confidence: 0.8} edges
• New sqlcg load-schema <csv> command loads production information schema; HAS_COLUMN edges from CSV are tagged source='information_schema' and take precedence over DDL-inferred columns (gold guard)
• .sqlcg/schema.csv convention: sqlcg index auto-loads this file if present — zero extra steps for large-repo users

SqlColumn: 0 from issue #5 is now resolved once PR #18 merges.

---

Sprint-06 plan is drafted (plan/sprint_06_lineage_coverage.md). Top 5 tickets based on a full-corpus experiment (1,445 Snowflake files):

1. FIX-E5 — 1,280 Cannot find column errors on CTE-derived columns in CREATE DYNAMIC TABLE files. Root cause: SchemaResolver-loaded information schema is not passed as sources= to sg_lineage(). This is now the single biggest coverage gap.
2. FIX-E2 — 60.4% of SELECT columns silently skipped (unaliased non-exp.Column expressions). Best-effort name extraction needed.
3. FIX-DDL-SKIP — Pure-DDL files (no DML body) go through sg_lineage() unnecessarily, causing 2,507 E3 noise events and 18–21s p99 outliers on WTDH_ARTIKEL* files.
4. FIX-E4 — 6 DML files lose all column lineage due to sqlglot Snowflake dialect gaps (IF NOT EXISTS, unexpected-token, UNPIVOT).
5. OPT-SCOPE — Build Scope once per statement; estimated 40–60% speedup on multi-column SELECTs.

[Warhorze]

Actual corpus numbers from the sprint-05 experiment (1,445 Snowflake files)

Pre-sprint-05 baseline (from results_full.json):

Metric	Value
Files with any successful lineage	18 / 1,445 (1.2%)
Total column expressions processed	730
Successful lineage edges emitted	37 (5.1% of cols processed)
E5 Cannot find column errors	1,280 (175% of cols — multiple per file)
E2 unaliased expression skips	441 (60.4% of cols)
E3 Command fallback noise	2,507 events
E4 full-skip parse failures	6 files

The 1.2% file resolution rate is the number that matters for sprint-06 planning. Even fixing E5 alone (the dominant error) would not immediately flip many files to "resolved" because most DML files hit multiple error types — but it unblocks the columns that can be resolved once the schema is available.

Sprint-06 priority order remains as documented in plan/sprint_06_lineage_coverage.md.