Add SF1 ground-truth answers for query correctness testing - #127
Merged
Conversation
Commit reference answers for all 12 SpatialBench queries at SF1 in two formats: q<n>.parquet (type-faithful canonical, what the harness compares against) and q<n>.csv (review companion; GitHub renders it and diffs are readable). The answers are the output of the canonical SedonaDB dialect (the reference oracle), independently cross-checked against DuckDB on every query it can run, so they are validated by two engines (11/12 agree within rtol=1e-6). Answers are normalized to an engine-neutral form (durations -> seconds with a _seconds suffix, decimals -> float, timestamps as datetime) and bounded to <=100 rows per query (the queries were bounded in #124), keeping the fixtures tiny. DuckDB's lateral-join Q12 is infeasible at SF1 (no KNN operator), so Q12 is not cross-checked here; it is validated by the KNN-capable engines in the correctness harness (follow-up). Part of #126.
jiayuasu
force-pushed
the
feature/query-ground-truth-answers
branch
from
July 22, 2026 19:07
c30d571 to
37be567
Compare
jiayuasu
marked this pull request as ready for review
July 22, 2026 19:31
Contributor
There was a problem hiding this comment.
Pull request overview
Adds committed SF1 “ground-truth” query results (Parquet canonical + CSV review copies) to support upcoming cross-engine correctness testing, and updates user-facing docs to explain correctness verification and dataset download options.
Changes:
- Add SF1 reference answer fixtures for q1–q12 in both Parquet (canonical) and CSV (diff-friendly) formats.
- Document answer provenance/normalization/comparison semantics under
benchmark/answers/. - Update README/docs/notebook with correctness-verification guidance and a Hugging Face download alternative to local data generation.
Reviewed changes
Copilot reviewed 19 out of 31 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds a “Verifying correctness” section and Hugging Face dataset download instructions. |
| notebooks/queries.ipynb | Adds Hugging Face download instructions as an alternative to local generation. |
| docs/queries.md | Adds Hugging Face download instructions. |
| docs/queries.zh.md | Adds Hugging Face download instructions (Chinese). |
| docs/index.md | Adds Hugging Face download instructions and mentions committed reference answers. |
| docs/index.zh.md | Adds Hugging Face download instructions and mentions committed reference answers (Chinese). |
| benchmark/answers/README.md | Documents answer layout, generation, normalization, and harness comparison semantics. |
| benchmark/answers/sf1/q1.csv | Adds SF1 CSV reference output for query 1. |
| benchmark/answers/sf1/q2.csv | Adds SF1 CSV reference output for query 2. |
| benchmark/answers/sf1/q3.csv | Adds SF1 CSV reference output for query 3. |
| benchmark/answers/sf1/q4.csv | Adds SF1 CSV reference output for query 4. |
| benchmark/answers/sf1/q5.csv | Adds SF1 CSV reference output for query 5. |
| benchmark/answers/sf1/q6.csv | Adds SF1 CSV reference output for query 6. |
| benchmark/answers/sf1/q7.csv | Adds SF1 CSV reference output for query 7. |
| benchmark/answers/sf1/q8.csv | Adds SF1 CSV reference output for query 8. |
| benchmark/answers/sf1/q9.csv | Adds SF1 CSV reference output for query 9. |
| benchmark/answers/sf1/q10.csv | Adds SF1 CSV reference output for query 10. |
| benchmark/answers/sf1/q11.csv | Adds SF1 CSV reference output for query 11. |
| benchmark/answers/sf1/q12.csv | Adds SF1 CSV reference output for query 12. |
| benchmark/answers/sf1/q1.parquet | Adds SF1 canonical Parquet reference output for query 1. |
| benchmark/answers/sf1/q2.parquet | Adds SF1 canonical Parquet reference output for query 2. |
| benchmark/answers/sf1/q3.parquet | Adds SF1 canonical Parquet reference output for query 3. |
| benchmark/answers/sf1/q4.parquet | Adds SF1 canonical Parquet reference output for query 4. |
| benchmark/answers/sf1/q5.parquet | Adds SF1 canonical Parquet reference output for query 5. |
| benchmark/answers/sf1/q6.parquet | Adds SF1 canonical Parquet reference output for query 6. |
| benchmark/answers/sf1/q7.parquet | Adds SF1 canonical Parquet reference output for query 7. |
| benchmark/answers/sf1/q8.parquet | Adds SF1 canonical Parquet reference output for query 8. |
| benchmark/answers/sf1/q9.parquet | Adds SF1 canonical Parquet reference output for query 9. |
| benchmark/answers/sf1/q10.parquet | Adds SF1 canonical Parquet reference output for query 10. |
| benchmark/answers/sf1/q11.parquet | Adds SF1 canonical Parquet reference output for query 11. |
| benchmark/answers/sf1/q12.parquet | Adds SF1 canonical Parquet reference output for query 12. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+8
to
+11
| ``` | ||
| answers/ | ||
| sf1/q1.parquet q1.csv ... q12.parquet q12.csv # scale factor 1 (SF10 to follow) | ||
| ``` |
Member
Author
There was a problem hiding this comment.
Fixed in e1c12b6 — the layout now shows the real benchmark/answers/sf1/q{1..12}.{parquet,csv} paths instead of the ambiguous answers/ tree.
|
|
||
| ### Verifying correctness | ||
|
|
||
| SpatialBench ships reference answers for every query under [`benchmark/answers/`](benchmark/answers/) (currently scale factor 1). They are the output of the canonical SedonaDB dialect, independently cross-checked against DuckDB, and let you confirm that each engine returns the same result for the same query — not just that it runs. See the [answers README](benchmark/answers/README.md) for the format and comparison rules. |
Member
Author
There was a problem hiding this comment.
Fixed in e1c12b6 — qualified the claim to "cross-checked against DuckDB for every query DuckDB can compute at this scale (all except Q12, whose lateral-join plan is infeasible at SF1)", matching the caveat in the answers README.
Reference the committed ground-truth answers (benchmark/answers/) in the README and docs site as the basis for verifying engine correctness, and add the Hugging Face dataset (apache-sedona/spatialbench, scale factors 0.1/1/10/100) as a way to obtain the benchmark data without building the generator. Covers the README, the docs landing page, and the queries page (EN + zh). Exclude benchmark/answers/ from codespell -- the answers are data with real place and entity names, not prose. Part of #126.
jiayuasu
force-pushed
the
feature/query-ground-truth-answers
branch
from
July 22, 2026 19:38
d578a03 to
bd5daf8
Compare
- Q6: average t_distance, aligning the SQL, GeoPandas, and PyCanopy implementations with the docs and Spatial Polars (which already used t_distance); regenerate the Q6 fixtures. The `avg_distance` alias had been computing t_totalamount in three implementations, so valid results would have been rejected. Also drop the internal "corrected from t_totalamount" note from the Q6 doc example. - RAT: add the ASF license header to benchmark/answers/README.md and exclude the generated CSV/Parquet fixtures in rat_exclude_files.txt. - Docs: replace `huggingface-cli download` with `hf download` (huggingface-hub v1 removed the huggingface-cli entrypoint). - Docs and notebook: load data via a DATA_DIR that handles both the flat CLI output and the partitioned Hugging Face layout, so the advertised download works end to end. - Answers README: document Q4 as the one query not capped at 100 rows (one row per zone over the top-1000 tipped trips).
jiayuasu
force-pushed
the
feature/query-ground-truth-answers
branch
from
July 22, 2026 21:14
9d4ac24 to
d68b394
Compare
…-checked - benchmark/answers/README.md: show the real benchmark/answers/sf1/ paths in the layout tree instead of an ambiguous answers/ example. - README.md: qualify the DuckDB cross-check claim -- every query except Q12 (whose lateral-join plan is infeasible at SF1) is cross-checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First of two PRs for #126 (a correctness harness that verifies participating engines return the same answer for the same query). This PR commits the ground-truth answers the harness will check against; the harness itself follows in PR B.
Committing answers is practical now because #124 bounded every query to ≤100 rows, so each fixture is tiny.
What's here
benchmark/answers/sf1/q{1..12}.{parquet,csv}— the expected result of each query at SF1, in two formats written from the same normalized frame:q<n>.parquet— type-faithful canonical answer (timestamps stay timestamps, ints stay ints); the harness compares against this.q<n>.csv— review companion: GitHub renders it as a table and diffs are readable when an answer changes.Plus
benchmark/answers/README.mddocumenting provenance, the canonical form, and comparison semantics.How the answers were produced
rtol=1e-6. 11/12 queries agree between the two engines.Canonical, engine-neutral form
Engines represent some types differently, so answers are normalized:
_secondscolumn suffixRow order is significant and preserved (deterministic
ORDER BY+ key tiebreakers, thenLIMIT).Caveat: Q12 at SF1
DuckDB has no KNN operator, so its Q12 uses a lateral cross-join that is infeasible at SF1 (does not finish in reasonable time). Q12 is therefore not cross-checked by DuckDB here — it will be validated by the KNN-capable engines (SedonaDB, Spatial Polars, PyCanopy) in the PR B harness.
Follow-ups