perf(engine): cols.vtype is an Enum, not a word repeated per column

[FBumann]

Carries #174 onto polars-engine. That PR measured the right thing against the wrong engine — it is CREATE TYPE ... AS ENUM and DESCRIBE cols against duckdb, which #189 deletes, so it cannot merge as written. The reasoning survives the engine; the SQL does not.

What it costs today

vtype holds one word per column, and the same three words for the whole model. As a string, polars stores that word once per row:

at 9.8M columns
vtype as String	0.098 GB
vtype as Enum	0.010 GB
cols frame	0.333 → 0.245 GB
four model frames	0.77 → 0.68 GB

Measured on transport/l, which is the case where the cols frame is largest.

The members are the contract

The storage is the reason to do it; the vocabulary is the reason to test it. An Enum rejects a value outside its categories, so a fourth entry added to plan.VariableType and not reaching this column now fails where the column is built — rather than downstream in whichever sink first compares against a name it does not know (lp_file filters on it, highs tests it for integrality).

#174's test comes with it, rewritten against the frame instead of DESCRIBE cols. I checked it bites: reverting the dtype to String fails both tests in the file.

One wrinkle

_DTYPES maps column names to dtype names rather than dtypes, because the executor imports polars lazily and a module-level pl.Enum(...) would force the import at module load. An Enum has to be constructed, so this adds _dtype() to resolve them rather than a fourth string entry.

399 passed / 1 xfailed, ruff and pyrefly clean.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9187d5f6-13d6-4c32-aa08-cf34f40b0bd8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/carry-over-vtype-enum

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[FBumann]

Why can't we change the import to eager? polars is a hard dep there...

We can — you're right that availability isn't the constraint, and I overstated the case in the PR body. polars>=1.30 is in [project] dependencies, not an extra, so it is always there. Measured, so the trade is explicit rather than a convention nobody re-examined:

import farkas today	84 ms
polars on its own	49 ms
polars in sys.modules after import farkas	False
polars in sys.modules after fk.check(...)	False

farkas/__init__ already imports relational.executor — the module costs 0 ms to import precisely because its polars imports are in functions. Hoist them and every import farkas pays the 49 ms, taking it to ~133 ms.

So the only thing the convention buys is fk.check. Validation parses YAML and reads no data, and today it never touches polars. Everything else — build, solve, write — pays the import anyway, just later.

Worth being clear that the hard rule doesn't apply here either: ARCHITECTURE.md rule 0 and test_lazy_intra_package_imports_are_all_declared only govern farkas.* imports (node.module.startswith('farkas')). The 48 in-function import polars as pl sites across 6 modules are outside it — so they are a convention, not a constraint, and nothing fails if they move.

If you want it eager I'll do it as its own change rather than inside this one: it deletes _dtype() here (the Enum goes straight into _DTYPES as a dtype rather than a name), and touches compiler.py, frames.py and the three sinks too. Say the word and I'll take it, and this PR gets simpler as a result.

[FBumann]

Rebased onto polars-engine after #212 landed — that is what the conflict was.

One thing to flag: I clobbered your merge of #213 and have restored it. You merged #213 into this branch at 07:40 (cb7f9e0). My rebase started from a stale local ref, so --force-with-lease compared against what I had last fetched rather than the true remote, passed, and dropped that merge commit.

This branch now carries both commits — the vtype Enum and the eager polars import — replayed on top of #212 rather than merged, so it is linear and conflict-free. Same content your merge produced; aa99be6 has 0 lazy polars imports, the Enum, and #212's factored labels from the base.

400 passed / 1 xfailed, ruff and pyrefly clean.