Add SF10 ground-truth answers - #129
Merged
Merged
Conversation
Dumping reuses each query's timed run (no extra execution), so gating it on committed answers already existing served no purpose and prevented bootstrapping answers for a new scale factor. Always dump when --result-dir is given; the verify job independently skips scale factors that have no committed answers. Drops the now unused --answers-dir option. Part of #126.
SF10 queries can spill several GB to disk; on the standard runner SedonaDB's Q5 (convex-hull aggregation) failed with "No space left on device". Reclaim space from preinstalled toolchains (dotnet, android, CodeQL, ...) before each query so disk-heavy SF10 queries can run. Part of #126.
Commits the SedonaDB reference results for scale factor 10
(benchmark/answers/sf10/q{1..12}.{parquet,csv}), produced by the benchmark run in
CI (the --result-dir dumps), not on a local machine. The type-faithful parquet is
reconstructed from the CI csv dumps using the SF1 schema (same queries, same
column types across scale factors).
SF10 was generated with SedonaDB 0.3.0. SedonaDB 0.4.0 cannot compute Q5 at SF10 —
its grouped convex-hull aggregation spills >100 GB and aborts
(apache/sedona-db#1077) — while 0.3.0 computes it in modest space. The two versions
produce identical results for the other 11 queries at SF10, so the values are
version-independent; only Q5's feasibility differs.
The correctness verify job cross-checks all engines against these answers.
Part of #126.
… purge - Duration precision (verify_results.py): SedonaDB truncates interval averages to milliseconds, so a duration column (`*_seconds`) can differ from a full-precision engine by up to ~1 ms — e.g. SF10 Q3 answer 7.666 vs 7.666666667. Apply a ~1 ms absolute tolerance to duration columns; other float metrics keep the tight default. No fixtures regenerated (the values are the oracle's). - Dump robustness (run_benchmark.py): now that dumping is unconditional, remove any stale dump for the queries being run before writing (dump filenames aren't scale-scoped, so a reused --result-dir could otherwise mix scales), and fail the run if result capture was requested (--result-dir) but a successful query produced no dump — so a bootstrapping run can't pass silently on a failed capture. - Conditional disk purge (benchmark.yml): the reclaim step now runs only for scale factors >= 10 (it added ~1 min to every SF1 job for no benefit) and runs before cache restore and dependency install so the space is available to them too. Part of #126.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the SF10 ground-truth answers (
benchmark/answers/sf10/q{1..12}.{parquet,csv}), completing the correctness fixtures for #126. The answers are the SedonaDB oracle's output, produced in CI — no queries were run on a local machine. The verify job now gates SF10 as well as SF1.How the answers were produced
--result-dir); the SedonaDB SF10 dumps are the answers. The type-faithful.parquetis reconstructed from the CI.csvdumps using the SF1 schema (same queries, same column types across scale factors).ST_Collect_Agg→ST_ConvexHull) spills >100 GB and aborts (filed as apache/sedona-db#1077). 0.3.0 computes it in modest space.rtol=1e-6), so the values are version-independent — only Q5's feasibility differs.Supporting changes
run_benchmark.py— dump normalized results whenever--result-diris set (previously gated on committed answers already existing). Dumping reuses the timed run, so it's free; this is what let CI bootstrap the SF10 answers. The verify job independently skips scale factors with no answers. (Drops the now-unused--answers-dir.)benchmark.yml— free ~20 GB of preinstalled toolchains before each query. SF10 queries spill several GB; without headroom SedonaDB's Q5 hitNo space left on deviceon the standard runner.After merge
The correctness verify job runs every engine against the SF10 answers, so each is independently cross-checked wherever the other engines can compute the query at scale.