Add Apache Spark dialect, observe-fork options, and BigQuery COALESCE fix

[richardwooding]

Summary

Ports recent feature work from cel2sql Go (PRs #117, #113) and cel2sql4j (PRs #10, #9) so pycel2sql is back at feature parity with both siblings.

What's in this PR

• Apache Spark SQL dialect (sixth dialect): ? positional params, RLIKE, get_json_object, concat(), array_contains (arg-order swap), 0-indexed arrays, EXPLODE / (SELECT collect_list(...)), format_string(), (dayofweek(t) - 1) for day-of-week. JSON array membership raises (the dialect contract doesn't expose the candidate element needed to build a boolean predicate). Regex passes through to Python's re engine after the same ReDoS validators DuckDB / Postgres use. Reserved-keyword + 128-char identifier validation.
• json_variables option: declare CEL variable names as flat JSONB columns. context.host and context["host"] both emit dialect-specific JSON extraction. Nested chains supported (tags.corpus.section → tags->'corpus'->>'section'). Comprehension iter vars shadow json_variables. has(jsonvar.key) emits write_json_existence.
• column_aliases option: map CEL identifier name → SQL column name. The alias is validated against the dialect's identifier rules.
• param_start_index option (convert_parameterized only): shift the placeholder counter so the first parameter is $5 / @p5 instead of $1. Values < 1 clamp to 1. Positional-? dialects ignore the index in placeholder text but preserve parameter ordering.
• Dialect.write_format() per-dialect dispatch (fixes latent bug exposed by adding Spark): Postgres + BigQuery emit FORMAT(...), SQLite + DuckDB emit printf(...), Spark emits format_string(...), MySQL raises UnsupportedDialectFeatureError. Previously every dialect emitted FORMAT(...), which is invalid SQL on SQLite / DuckDB / MySQL / Spark.
• BigQuery COALESCE NULL-safety fix: write_array_length and write_json_array_length now wrap output in COALESCE(..., 0), matching the four other dialects. Ports cel2sql4j PR ci: restrict GITHUB_TOKEN to contents:read #9 / cel2sql4j upstream commit 1689adc.

Test results

• 704 unit tests pass (607 baseline + 97 new across tests/test_spark.py (47 tests), tests/test_options.py (26 tests), and per-dialect format() cases in tests/test_string_functions.py (5 tests) plus a BigQuery COALESCE assertion).
• Ruff clean.
• Mypy non-lark errors unchanged from baseline (6); one added lark Tree[Any] generic-arg note matches the existing pattern documented in CLAUDE.md.

Docs

• README: Spark badge, Conversion Options section (json_variables / column_aliases / param_start_index / format() per-dialect table), updated placeholder table, sixth-dialect imports.
• CLAUDE.md: dialect count bumped to six, Spark added to the dialect-files list and Dialect Differences bullets, new options documented under Important Conventions.

Follow-up scheduled

A one-time remote agent fires on Mon 11 May 2026 to verify the BigQuery COALESCE wrap against the BigQuery emulator (PYCEL2SQL_TEST_BIGQUERY=1). It will post a confirmation comment on success, or open an issue on failure. Routine: trig_01XUGxNA7onedEDdRnuK4ixL.

Test plan

• uv run pytest tests/ --ignore=tests/integration -v — 704 passing
• uv run ruff check src/ tests/ — clean
• uv run mypy src/pycel2sql/ — no NEW non-lark errors
• Optional: PYCEL2SQL_TEST_BIGQUERY=1 uv run pytest tests/integration/ -v (deferred to scheduled follow-up agent)
• Manual smoke for each new option in a Python REPL (json_variables, column_aliases, param_start_index, Spark format() → format_string)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Ports feature parity updates from the Go/Java sibling projects into pycel2sql by adding an Apache Spark dialect, new conversion options (JSON-root variables, identifier aliases, param start offset), and fixing BigQuery array-length NULL behavior.

Changes:

• Added SparkDialect with Spark-specific SQL generation (positional ? params, RLIKE, get_json_object, array_contains, format_string, EXPLODE/collect_list, 0-based arrays, Spark identifier validation).
• Introduced conversion options: json_variables, column_aliases, and param_start_index (for convert_parameterized).
• Implemented per-dialect format() dispatch via Dialect.write_format() and fixed BigQuery COALESCE(..., 0) wrapping for array length functions.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/pycel2sql/dialect/spark.py	New Spark SQL dialect implementation including regex validation, JSON extraction, arrays, and comprehensions.
src/pycel2sql/dialect/_base.py	Adds new DialectName.SPARK and makes write_format() an abstract requirement for dialects.
src/pycel2sql/dialect/__init__.py	Exports SparkDialect and wires it into get_dialect().
src/pycel2sql/dialect/postgres.py	Implements write_format() using FORMAT(...).
src/pycel2sql/dialect/bigquery.py	Adds COALESCE(..., 0) wrapping for array lengths and implements write_format() using FORMAT(...).
src/pycel2sql/dialect/sqlite.py	Implements write_format() using printf(...).
src/pycel2sql/dialect/duckdb.py	Implements write_format() using printf(...).
src/pycel2sql/dialect/mysql.py	Implements write_format() by raising UnsupportedDialectFeatureError.
src/pycel2sql/_converter.py	Adds support for json_variables/column_aliases/param_start_index, JSON-root routing, aliasing, and delegates format() generation to the dialect.
src/pycel2sql/__init__.py	Public API wiring for new options and refactors shared kwargs building.
tests/test_spark.py	New Spark-specific unit test suite.
tests/test_options.py	New tests covering json_variables, column_aliases, and param_start_index.
tests/test_string_functions.py	Adds per-dialect format() dispatch assertions.
tests/test_bigquery.py	Updates BigQuery array length expectation to include COALESCE.
tests/test_dialect_parametrized.py	Adds Spark to the parametrized dialect matrix.
tests/conftest.py	Adds spark_dialect fixture and includes Spark in ALL_DIALECTS.
README.md	Documents Spark support and the new options / per-dialect format() behavior.
CLAUDE.md	Updates project guidance to include Spark and the new options / format dispatch behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The inline-flag validation misses combined flag groups like (?im) / (?mi): the current substring checks only catch (?m/(?s) when those flags appear first. This allows non-i flags through despite the intent to reject them. Consider parsing (?[a-zA-Z]+) groups and rejecting any set that includes flags other than i (including combined forms).

[richardwooding]

Fixed in 6eb6436. The check now uses _INLINE_FLAG_GROUP_RE = re.compile(r"\(\?([a-zA-Z]+)[:)]") to extract the flag-letter set from each (?<flags>) or (?<flags>: group and rejects any group containing flags other than i. The bare (?- clear form is also rejected. Added parametrized tests for (?im), (?mi), (?ix:foo), (?-i), plus a sanity check that (?i)foo alone still passes.

[Copilot]

The docstring for TestFormatPerDialect is now incomplete: it documents FORMAT/printf/MySQL-raises, but this PR also adds Spark's format_string(...) mapping. Please update the docstring so it matches the actual per-dialect behavior being tested elsewhere in the suite.

Suggested change

	printf for SQLite/DuckDB, raises for MySQL."""
	format_string for Spark, printf for SQLite/DuckDB, raises for MySQL."""

[richardwooding]

Fixed in 6eb6436 — applied your suggested wording verbatim.

[Copilot]

Comment says size() returns -1 for NULL, but in Spark SQL size(NULL) evaluates to NULL. The COALESCE wrapper is still correct, but the comment should be updated to avoid misleading future maintainers.

Suggested change

	# Spark size() returns -1 for null; COALESCE collapses to 0.
	# In Spark SQL, size(NULL) evaluates to NULL; COALESCE converts that to 0.

[richardwooding]

Fixed in 6eb6436 — applied your suggested wording verbatim. Good catch: the -1 claim was carried over from a different language port and is wrong for Spark SQL.

[Copilot]

This comment says numeric specifiers are converted to %s "for printf-style dialects", but the conversion is applied unconditionally before dispatching to Dialect.write_format() (including Postgres/BigQuery/Spark). Update the comment to reflect the actual behavior, or make the conversion dialect-specific if that was the intent.

Suggested change

	# Convert %d, %f etc. to %s for printf-style dialects
	# Normalize supported numeric specifiers (%d, %f, etc.) to %s before
	# dispatching to Dialect.write_format().

[richardwooding]

Fixed in 6eb6436. Updated comment to "Normalize supported numeric specifiers (%d, %f, etc.) to %s before dispatching to Dialect.write_format()". Kept the conversion unconditional (rather than making it dialect-specific) because %s is universally accepted by FORMAT/printf/format_string and avoids type-coercion mismatches when CEL arguments do not match the original specifier exactly.