Plan the next two work streams

[jdatcmd]

Plans written before any code, for the two streams remaining after Phase G.

PARQUET_FOLLOWONS_PLAN.md sequences the three remaining Parquet roadmap items smallest first: INT32/INT64-backed DECIMAL reads, a recursive directory walk, then Hive-style partition pruning which depends on the walk. Two decisions are made explicitly rather than by default:

• The walk will not descend into symlinked directories. A visited-inode set is more machinery than the rule is worth, and "we do not follow directory symlinks" is something a user can reason about.
• Partition columns will be declared as a table option rather than inferred from the tree, for the same reason the column definition list is required rather than inferred: a wrong guess silently changes which rows a query returns.

CASCADE_ENCODING_PLAN.md disagrees with the roadmap's "low to medium effort" and says why. The item bundles two independent ideas:

• Cascading (encode one scheme's output with another) is where the size win is, and it changes the on-disk format. A chunk header must express a chain, which means a chain descriptor, a corrupt-file guard on the chain length, and an explicit decision about old builds reading new files. Recommend a format version bump so that failure is loud.
• Sampling is a write-throughput optimization whose value depends entirely on what selection costs today, which has never been measured. ColumnarEncodeChunk currently runs every applicable encoder in full and keeps the smallest.

So step 1 is to measure selection cost on the 6M-row bench and let the number decide whether the sampling half is built at all. Finding out it is not worth it is a legitimate outcome, and much cheaper to reach now than after building it.

Design docs only, no code, so no gate.

🤖 Generated with Claude Code