feat - implement RFC 002 Substrait boundary

[dannymeijer]

Summary

This PR implements the current RFC 002 Substrait boundary in InQL and reorganizes the package around that boundary. The package now exposes proto-backed Substrait plan/schema/conformance modules, explicit join/set operation helpers, registered extension URI metadata, and real conformance/testing coverage, while also tightening the RFC/docs language so it stops overstating what is already implemented. It also adds RFC 006 and RFC 007 draft/spec work needed to position unnest and Prism relative to the new Substrait boundary.

Type of change

• Bug fix
• New feature
• Refactor / maintenance
• Documentation
• CI / tooling
• RFC (adds/updates docs/rfcs/*)

Area(s)

Select the primary areas touched (labels sync from checked lines when the triage workflow runs):

• Package & tests
• Specification (RFCs)
• Documentation
• Automation & repo config
• Other

Key details

• User-facing behavior: InQL now exposes a real proto-backed Substrait package surface under substrait.*, including schema helpers, plan builders, conformance scenarios, explicit join/set operation enums, reference ordinal preservation, and registered extension URI helpers. Dataset carriers and tests now lower through that boundary successfully.
• Internals: The package was reorganized from flat modules into nested dataset.* and substrait.* modules. RFC 002 was tightened to treat Substrait as a thin emitted boundary, while Prism is documented separately as the future internal planning substrate. Conformance coverage now runs against real proto emission instead of doc-only claims.
• Risks: This is a broad branch. It changes package layout, public exports, docs structure, tests, lockfile state, and CI config in one PR. The Substrait boundary is intentionally still thin, so richer ProjectRel / AggregateRel semantics remain deferred. There is also a temporary workaround in the package for incan empty-list lowering, tracked separately in incan#229.

Testing / verification

• make ci (or make fmt-check, make build, make test)
• Manual verification described below

Manual verification notes:

• Ran incan build --lib in the InQL repo: passed.
• Ran incan test in the InQL repo: passed (20 passed).
• Ran incan --check src/substrait/plan.incn: passed.
• Filed the related compiler follow-up for empty list literal lowering as incan#229.

Docs impact

• No docs changes needed
• Docs updated
• Docs follow Divio intent (tutorial/how-to/reference/explanation) where applicable

If docs updated:

• Link(s):
  • docs/rfcs/002_apache_substrait_integration.md
  • docs/rfcs/006_unnest_core_substrait.md
  • docs/rfcs/007_prism_planning_engine.md
  • docs/language/reference/substrait/conformance.md
  • docs/language/reference/substrait/operator_catalog.md
  • docs/language/reference/substrait/read_root_binding_contract.md
  • docs/language/reference/substrait/revision_and_extension_policy.md
  • docs/architecture.md
  • docs/release_notes/v0_1.md

Checklist

• I kept public docs user-focused and moved internals to contributing docs when appropriate
• I avoided duplicating canonical install/run instructions in multiple places
• I added/updated tests where it materially reduces regressions

Closes #3