Merge upstream Vortex 0.79.0 into spiceai-54#74
Merged
Conversation
See the `README.md` added as part of this PR how to use pyVortex CUDA. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…rtex-data#8607) ## Summary <!-- Why are you proposing this change, and what is its impact? Is it part of a long term effort, or a bigger change? If this PR is related to a tracked effort or an open issue, please link to the relevant issue. --> Wires SpatialBench into the `vx-bench` / `bench-orchestrator` pipeline so it can be run end-to-end like the other benchmarks (datagen → Parquet → Vortex conversion → query). It builds on the WKB datagen landed in vortex-data#8598. Running command: ``` uv run --project bench-orchestrator vx-bench run spatialbench --engine duckdb --format parquet,vortex --opt scale-factor=N --queries 1,2,3,4,5,6,7,8,9 --iterations 3 ``` ## Limitation <!-- No need to duplicate information from the previous section, but if you're touching many parts of the code base, its worth explicitly noting the important changes or how they are tested. --> - DuckDB-only. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in `BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }`. - No dictionary encoding / compaction on the WKB column. WKB geometry blobs are large and effectively unique, so running the dictionary builder over them balloons memory (tens of GB) for zero compression gain. The normal compaction path is preserved for every other column on every other benchmark. - Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index. ## Performance SF=1.0 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 39.1ms │ 14.6ms (0.37x) │ │ 2 │ 72.1ms │ 24.9ms (0.35x) │ │ 3 │ 57.7ms │ 20.1ms (0.35x) │ │ 4 │ 113.1ms │ 70.6ms (0.62x) │ │ 5 │ 354.2ms │ 288.4ms (0.81x) │ │ 6 │ 169.5ms │ 91.6ms (0.54x) │ │ 7 │ 156.7ms │ 71.5ms (0.46x) │ │ 8 │ 196.5ms │ 80.3ms (0.41x) │ │ 9 │ 20.3ms │ 18.7ms (0.92x) │ └───────┴───────────────────────┴─────────────────┘ ``` SF=3 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 50.7ms │ 31.5ms (0.62x) │ │ 2 │ 126.2ms │ 60.3ms (0.48x) │ │ 3 │ 71.9ms │ 42.9ms (0.60x) │ │ 4 │ 539.9ms │ 64.9ms (0.12x) │ │ 5 │ 948.7ms │ 874.5ms (0.92x) │ │ 6 │ 656.2ms │ 121.7ms (0.19x) │ │ 7 │ 256.6ms │ 232.1ms (0.90x) │ │ 8 │ 273.8ms │ 244.6ms (0.89x) │ │ 9 │ 35.7ms │ 27.9ms (0.78x) │ └───────┴───────────────────────┴─────────────────┘ ``` SF=10 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ 1 │ 158.6ms │ 114.4ms (0.72x) │ │ 2 │ 255.1ms │ 219.3ms (0.86x) │ │ 3 │ 229.2ms │ 181.5ms (0.79x) │ │ 4 │ 184.5ms │ 134.3ms (0.73x) │ │ 5 │ 3.30s │ 3.08s (0.93x) │ │ 6 │ 476.4ms │ 348.9ms (0.73x) │ │ 7 │ 918.2ms │ 961.2ms (1.05x) │ │ 8 │ 980.6ms │ 926.7ms (0.94x) │ │ 9 │ 33.7ms │ 33.6ms (1.00x) │ └───────┴───────────────────────┴─────────────────┘ ``` <!-- Are there any user-facing changes that might require documentation updates Is any public API changed? --> --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…tive geometry (vortex-data#8608) ## Summary <!-- Why are you proposing this change, and what is its impact? Is it part of a long term effort, or a bigger change? If this PR is related to a tracked effort or an open issue, please link to the relevant issue. --> Adds a native `MultiPolygon` geometry extension type to `vortex-geo`, and a WKB-export helper (`to_wkb`) for the native geometry types (`Point`, `Polygon`, `MultiPolygon`). `MultiPolygon` (`vortex.geo.multipolygon`) mirrors the existing `Polygon` type, mainly for SpatialBench `z_boundary` column, which is mixture of `Polygon` and `MultiPolygon`. `to_wkb` serializes a native geometry array to WKB (via `geoarrow-cast`), the form DuckDB's `GEOMETRY` vectors carry, for communicate with DuckDB. <!-- No need to duplicate information from the previous section, but if you're touching many parts of the code base, its worth explicitly noting the important changes or how they are tested. --> ## What APIs are changed? Are there any user-facing changes? <!-- Are there any user-facing changes that might require documentation updates Is any public API changed? --> Purely **additive**, new public API in `vortex-geo`: - `extension::MultiPolygon`, the new extension type (with its `ExtVTable` + Arrow import/export). - `extension::{PointData, PolygonData}` + `to_wkb` on the existing `Point`/`Polygon` types. --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…, and wire into benchmark orchestrator (vortex-data#8623) ## Summary <!-- Why are you proposing this change, and what is its impact? Is it part of a long term effort, or a bigger change? If this PR is related to a tracked effort or an open issue, please link to the relevant issue. --> Wires vortex geo native format into SpatialBench, and wire into the `vx-bench` / `bench-orchestrator` pipeline so it can be run end-to-end like the other benchmarks. Running command: ``` vx-bench run spatialbench -e duckdb -f parquet,vortex,vortex-native --opt scale-factor=1.0 --queries 1,2,3,4,5,6,7,8,9 ``` ## Limitation <!-- No need to duplicate information from the previous section, but if you're touching many parts of the code base, its worth explicitly noting the important changes or how they are tested. --> - DuckDB-only as before. For now SpatialBench queries use DuckDB-specific ST_* spatial SQL that DataFusion has no functions for yet. There is a a single ad-hoc entry in `BENCHMARK_ENGINES = { SPATIALBENCH: {DUCKDB} }`. - Queries 10, 11, 12 is timeout simply because DuckDB poorly support on Spatial index. ## Performance SF=1.0 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ duckdb:vortex-native ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ 1 │ 40.8ms │ 13.8ms (0.34x) │ 23.2ms (0.57x) │ │ 2 │ 181.2ms │ 26.3ms (0.14x) │ 35.3ms (0.20x) │ │ 3 │ 57.8ms │ 19.8ms (0.34x) │ 30.9ms (0.53x) │ │ 4 │ 331.9ms │ 61.8ms (0.19x) │ 111.0ms (0.33x) │ │ 5 │ 356.1ms │ 294.3ms (0.83x) │ 299.3ms (0.84x) │ │ 6 │ 441.9ms │ 86.3ms (0.20x) │ 135.1ms (0.31x) │ │ 7 │ 157.2ms │ 73.7ms (0.47x) │ 93.1ms (0.59x) │ │ 8 │ 197.8ms │ 78.1ms (0.39x) │ 91.9ms (0.46x) │ │ 9 │ 20.3ms │ 18.6ms (0.91x) │ 20.7ms (1.02x) │ └───────┴───────────────────────┴─────────────────┴──────────────────────┘ ``` SF=3 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ duckdb:vortex-native ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ 1 │ 51.5ms │ 26.6ms (0.52x) │ 61.6ms (1.20x) │ │ 2 │ 127.6ms │ 56.2ms (0.44x) │ 93.1ms (0.73x) │ │ 3 │ 69.1ms │ 46.3ms (0.67x) │ 80.0ms (1.16x) │ │ 4 │ 543.4ms │ 64.5ms (0.12x) │ 119.6ms (0.22x) │ │ 5 │ 980.6ms │ 881.6ms (0.90x) │ 894.4ms (0.91x) │ │ 6 │ 660.9ms │ 133.8ms (0.20x) │ 233.2ms (0.35x) │ │ 7 │ 255.7ms │ 240.6ms (0.94x) │ 264.9ms (1.04x) │ │ 8 │ 267.6ms │ 247.4ms (0.92x) │ 299.7ms (1.12x) │ │ 9 │ 30.1ms │ 28.8ms (0.96x) │ 30.6ms (1.02x) │ └───────┴───────────────────────┴─────────────────┴──────────────────────┘ ``` SF=10 ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ duckdb:vortex-native ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ 1 │ 160.4ms │ 128.5ms (0.80x) │ 214.8ms (1.34x) │ │ 2 │ 254.3ms │ 239.8ms (0.94x) │ 325.9ms (1.28x) │ │ 3 │ 231.0ms │ 198.7ms (0.86x) │ 284.4ms (1.23x) │ │ 4 │ 188.0ms │ 124.2ms (0.66x) │ 147.9ms (0.79x) │ │ 5 │ 3.14s │ 3.05s (0.97x) │ 2.92s (0.93x) │ │ 6 │ 480.5ms │ 361.7ms (0.75x) │ 467.7ms (0.97x) │ │ 7 │ 992.8ms │ 1.02s (1.02x) │ 915.2ms (0.92x) │ │ 8 │ 1.07s │ 961.8ms (0.90x) │ 1.02s (0.95x) │ │ 9 │ 34.2ms │ 34.1ms (1.00x) │ 43.5ms (1.27x) │ └───────┴───────────────────────┴─────────────────┴──────────────────────┘ ``` <!-- Are there any user-facing changes that might require documentation updates Is any public API changed? --> Takeaways: for now we do not have only pushdown to execute predicate in vortex kernel, so geo native types pay the tax of converting to geo-binary (talk to duckdb) but no gains, so in SF=10 vortex-native is less efficient compared with vortex-wkb. --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…p crate (vortex-data#8642) ## Rationale for this change `vortex-files` pull a lot of encodings which makes it a very big dependency. Some systems only need the core APIs, but enabling "tokio", "zstd" or "object-store" currently pulls in the full `vortex-file`. ## What changes are included in this PR? Make the features of "vortex" use a weak link to "vortex-file", so it doesn't pull it in by default. ## What APIs are changed? Are there any user-facing changes? Some feature configurations will require adjustment. --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>
It's unclear why we left it, we implement the boolean logic on vortex boolean arrays always
…ortex-data#8637) This changes LEGACY_SESSION top level lazy static into a method. We also disallow this method from being used in the codebase and enshrine existing call sites with exceptions
Replaces the per-element, data-dependent branch in the listview zip kernel's offset/size selection with a branchless, chunk-at-a-time mask select that the compiler can auto-vectorize. For each 64-bit mask chunk, each bit is expanded to a full-width lane mask and both sides are blended with `(t & m) | (f & !m)` via a shared `select_column` helper, so the inner loop is branch-free regardless of mask shape. `if_false` offsets are shifted into the second half of the concatenated `elements` as before; sizes are taken verbatim from the chosen side.
This module was never fully fleshed out and we have different ways to accomplish same things. This also migrates the PValue search sorted to be purely primitive based
## Summary <!-- Why are you proposing this change, and what is its impact? Is it part of a long term effort, or a bigger change? If this PR is related to a tracked effort or an open issue, please link to the relevant issue. --> Insert non-throwing geo predicate `vortex_dwithin` in DuckDB, which later pushdown into vortex, call `distance` scalar function on scanning, significantly improve Q1/Q3 performance in SpatialBench. ## What changes are included in this PR? <!-- No need to duplicate information from the previous section, but if you're touching many parts of the code base, its worth explicitly noting the important changes or how they are tested. --> 1. adding new geo predicate `vortex_dwithin` in DuckDB, which is non-throwing and can be pushdown. 2. adding SQL rewrite so for geo native type, `ST_dwithin` is text rewritten into `vortex_dwithin`. <!-- Are there any user-facing changes that might require documentation updates Is any public API changed? --> ## Performance ``` ┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Query ┃ duckdb:parquet (base) ┃ duckdb:vortex ┃ duckdb:vortex-native ┃ ┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ 1 │ 179.6ms │ 132.1ms (0.74x) │ 29.0ms (0.16x) │ │ 2 │ 268.1ms │ 255.3ms (0.95x) │ 312.5ms (1.17x) │ │ 3 │ 247.8ms │ 216.4ms (0.87x) │ 140.2ms (0.57x) │ │ 4 │ 199.6ms │ 146.9ms (0.74x) │ 144.5ms (0.72x) │ │ 5 │ 3.40s │ 3.43s (1.01x) │ 3.03s (0.89x) │ │ 6 │ 528.1ms │ 359.2ms (0.68x) │ 455.4ms (0.86x) │ │ 7 │ 984.0ms │ 983.2ms (1.00x) │ 887.7ms (0.90x) │ │ 8 │ 1.08s │ 945.4ms (0.87x) │ 997.6ms (0.92x) │ │ 9 │ 34.2ms │ 33.5ms (0.98x) │ 43.6ms (1.27x) │ └───────┴───────────────────────┴─────────────────┴──────────────────────┘ ``` Takeaways: Q1 and Q3 is significantly improved due to the single table geo predicate is pushdown into vortex scan. Q1 is more benefit from the manually fast path without going through `geo` crate for calculation. Q3 is also potentially can be benefit from manually calculation. --------- Signed-off-by: Nemo Yu <zyu379@wisc.edu>
…8479) Change benchmarks to force initialisation of vortex session before start of benchmarking loop
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [cxx](https://cxx.rs) ([source](https://redirect.github.com/dtolnay/cxx)) | dependencies | patch | `1.0.194` → `1.0.195` | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. --- ### `let_cxx_string!` uses uninitialized value due to exception safety violations [RUSTSEC-2026-0202](https://rustsec.org/advisories/RUSTSEC-2026-0202.html) <details> <summary>More information</summary> #### Details In affected versions of this crate, `let_cxx_string!` is not exception safe. After creating the `StackString`, if `match $value` panics, the content of `StackString` is not yet initialized, while the drop implementation of `StackString` unconditionally deinitializes the content, leading to use of uninitialized value. The soundness issue was fixed in version `1.0.195` by moving drop logics to separate drop guard after initializing the `StackString`. #### Severity Unknown #### References - [https://crates.io/crates/cxx](https://crates.io/crates/cxx) - [https://rustsec.org/advisories/RUSTSEC-2026-0202.html](https://rustsec.org/advisories/RUSTSEC-2026-0202.html) - [https://github.com/dtolnay/cxx/issues/1729](https://redirect.github.com/dtolnay/cxx/issues/1729) This data is provided by [OSV](https://osv.dev/vulnerability/RUSTSEC-2026-0202) and the [Rust Advisory Database](https://redirect.github.com/RustSec/advisory-db) ([CC0 1.0](https://redirect.github.com/rustsec/advisory-db/blob/main/LICENSE.txt)). </details> --- ### Release Notes <details> <summary>dtolnay/cxx (cxx)</summary> ### [`v1.0.195`](https://redirect.github.com/dtolnay/cxx/releases/tag/1.0.195) [Compare Source](https://redirect.github.com/dtolnay/cxx/compare/1.0.194...1.0.195) - Fix use of uninitialized value in `let_cxx_string!` on panic inside initialization expression ([#&vortex-data#8203;1729](https://redirect.github.com/dtolnay/cxx/issues/1729), [#&vortex-data#8203;1731](https://redirect.github.com/dtolnay/cxx/issues/1731)) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Add benchmarks for bitbuffer iteration methods
…ata#8657) Since we already skip cuda build I think this should be symmetrical and also be skipped Signed-off-by: Robert Kruszewski <github@robertk.io>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> --------- Signed-off-by: Robert Kruszewski <github@robertk.io> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Robert Kruszewski <github@robertk.io>
…ex-data#8636) recreation of vortex-data#8261 which had real performance benefits for iterating valid indices and had guidelines for future agents performance work
…32 (vortex-data#8658) FFI currently has two types of returned objects: ones are owned (inside of which there's an Arc), others are borrowed (because there's no Arc inside). One example of the latter is a struct field name which returns a borrowed vx_string. We initially had a comment caller shouldn't call the corresponding _free() function on these. This is not enough as caller can call a _clone() function on a borrowed vx_string which is UB because we try to increment a strong reference count on a pointer which isn't an Arc. From a callee's perspective, we can't distinguish owned and borrowed types, and this creates weird APIs like struct field dtype being owned but name being borrowed. This PR removes all borrowed/owned distinction from FFI layer. Each object now returned from an FFI function is owned. It can be freely cloned, and each must be freed explicitly with _free() functions. On another change, fixed size list constructor previously accepted a size_t (64-bit unsigned integer on most systems) but validated it against u32 because that's our file format limitation for fixed size lists. This PR changes the signature so that now it accepts uint32_t as length parameter. As the changes are significant enough to trigger bugs in our own tests, this PR also re-enables FFI sanitizer tests in CI. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Github approval logic is not flexible enough to not be super noisy or allow writes without allowing approvals. Policy-bot will let us have arbitrary approval configuration Signed-off-by: Robert Kruszewski <github@robertk.io>
…ortex-data#8647) While integrating vortex into another project I realised that these are more sensible apis
…ortex-data#8650) Existing logic was duplicated between jni and duckdb with slight differences. JNI logic didn't always handle local paths.
These are unnecessary now that we have policy-bot
…a#8667) ## Summary Closes: vortex-data#8666 The five per-type constant schemes (bool/int/float/string/binary) were "half builtin": they lived in `vortex-compressor`'s `builtins` module but still had to be registered. Constant handling was also scattered across three places: the compressor's own all-null short-circuit, the per-type constant schemes, and an inline `is_constant` check in `TemporalScheme`. Constant detection is now built directly into `CascadingCompressor`. A new crate-private `constant` module detects constant leaves using the cheapest available evidence per type. Changes in behavior: - The big change is that constant detection is unconditional:, as it applies even for a compressor constructed with an empty scheme list, and can no longer be disabled via `exclude_schemes`. I think this is fine because it is essentially never a good idea to not check for a constant array (except maybe for tiny arrays, but I don't think we need to worry about that), and we have no code that tries to exclude constant array. - Constant detection now covers ALL leaf types uniformly, adding decimal and extension arrays. - Constant string/binary arrays no longer pay for dict/FSST sampling before the (previously deferred, last-registered) constant check happens. - Also a small perf change: The old scheme-based path found the first valid element with a per-index `is_valid` loop that re-derived validity on every call; the new `compress_constant` function materializes the validity mask once and uses `Mask::first`. Signed-off-by: Connor Tsui <connor@spiraldb.com> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
- Add canonicalize function for vx_array. - Add data_ptr functions for canonicalized primitive and bool arrays. - Add sink abort method which doesn't write file footer and leaves file invalid. Semantics here is we want to abort the sink in C++ destructor - if user hasn't called Finalize(), file will not be written correctly. - Expose Rust's error code to FFI. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
Support mean() for decimals. Support casting decimals to f64. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
## Rationale for this change Make better use of sccache across our GH actions workflows. ## What changes are included in this PR? 1. Makes sccache opt in 2. When sccache is used, make sure its properly configured with runs-on. ## What APIs are changed? Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adam@spiraldb.com>
This also removes ArrayBuilder::extend_from_array and moves that logic in to ArrayRef::append_to_builder Plumb ExecutionCtx through more methods instead of creating fresh one. Fixes: vortex-data#3235
Grow List/ListView builders when we append to them. Since they're variable length we can't preallocate them Signed-off-by: Robert Kruszewski <github@robertk.io>
…ks. (vortex-data#8672) ## Rationale for this change `TableStrategy` currently hides panics that happen in the individual column streams. ## What changes are included in this PR? Instead of detaching the fanout work, we await it and make sure everything finishes successfully. ## What APIs are changed? Are there any user-facing changes? No API change, added test to verify panic propagation. Signed-off-by: Adam Gutglick <adam@spiraldb.com>
vx_string/vx_binary is not a zero-copy abstraction. For every utf/binary view you allocate memory. This PR replaces both with a vx_view non-owning view akin to C++'s string_view. Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…vortex-data#8781) Reset offsets operations if they start from a valid array they should stay valid
- Add tests for new C++ API. - Upload test results to codecov. - Make vx_array_new_primitive return an error on validity dtype/length mismatch Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…materialization (vortex-data#8783) ## Rationale for this change `PartitionPathUtils` (`java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java`) discovers Hive-style partition columns from file paths and materializes them as constant column vectors on the read path. It currently has no test coverage, despite encoding several subtle behaviors: URL decoding of keys and values, the `__HIVE_DEFAULT_PARTITION__` null sentinel, first-`=` splitting, type inference precedence (int → long → double → boolean → string), and per-type parsing into `ConstantColumnVector`. This continues the characterization-test series from vortex-data#8770 (read-path `ArrowUtils`) and vortex-data#8782 (write-path `SparkToArrowSchema`). ## What changes are included in this PR? A new `PartitionPathUtilsTest` covering all three public methods: - **`parsePartitionValues`**: extracts ordered `key=value` segments from paths; ignores segments without the `key=value` shape (including empty keys or values); URL-decodes both keys and values (`New%20York` → `New York`); splits on the first `=` so values may themselves contain `=`. - **`inferPartitionColumnType`**: infers `Integer` for int-width values, `Long` for wider integrals, `Double` for decimal-point values, `Boolean` for case-insensitive true/false, and falls back to `String` (including for null and the `__HIVE_DEFAULT_PARTITION__` sentinel). - **`createConstantVector`**: null and the Hive default-partition sentinel produce null vectors; string/int/long/short/byte/boolean/float/double values are parsed into their target types; `DateType` parses as days-since-epoch (int) and both timestamp types as micros-since-epoch (long); unrecognized target types fall back to UTF8 strings. This is a **test-only** change; no production code is modified. 19 tests, all passing locally (`:vortex-spark_2.12:test --tests 'dev.vortex.spark.read.PartitionPathUtilsTest'`). ## What APIs are changed? Are there any user-facing changes? None. No production code or public API is touched. Signed-off-by: jackylee <qcsd2011@gmail.com>
Testing SQL queries' correctness is done in vortex-sqllogictest, but we also want to test relative performance of queries not covered in existing benchmarks. One example is aggregation with filter which should be much more performant once we use zone maps statistics, but this type of query isn't present is existing benchmarks. This PR introduces a new query-per-file dataset and support for running it in CI Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
vortex-data#8782) ## Rationale for this change `SparkToArrowSchema` (`java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java`) converts Spark SQL schemas to Arrow schemas on the **write path**, and currently has no test coverage. It is the mirror image of `ArrowUtils` on the read path, which gained characterization tests in vortex-data#8770. Like that mapping table, this conversion is exactly the kind of code that regresses quietly: a wrong bit width, a dropped timezone, or lost nullability would silently corrupt written files rather than fail loudly. ## What changes are included in this PR? A new `SparkToArrowSchemaTest` that characterizes the conversion: - **Primitive mappings**: `Boolean` → `Bool`; `Byte/Short/Integer/Long` → signed `Int` 8/16/32/64; `Float/Double` → `SINGLE/DOUBLE` floating point; `String` → `Utf8`; `Binary` → `Binary`; `Decimal` → 128-bit Arrow `Decimal` preserving precision and scale; `Date` → `Date(DAY)`; `Timestamp` → `Timestamp(MICROSECOND, "UTC")` and `TimestampNTZ` → `Timestamp(MICROSECOND, null)`. - **Schema structure**: field names and ordering are preserved; field-level nullability is carried over to the Arrow field. - **Nested types**: `StructType` converts to a `Struct` field with recursively converted children (including per-child nullability); `ArrayType` converts to a `List` whose `element` child carries `containsNull`. - **Rejected types**: unsupported Spark types (e.g. `CalendarIntervalType`) raise `UnsupportedOperationException` naming the offending type. This is a **test-only** change; no production code is modified. 12 tests, all passing locally (`:vortex-spark_2.12:test --tests 'dev.vortex.spark.write.SparkToArrowSchemaTest'`). ## What APIs are changed? Are there any user-facing changes? None. No production code or public API is touched. Signed-off-by: jackylee <qcsd2011@gmail.com>
## Rationale for this change cuda compatible vortex files currently can't be written over ffi ## What changes are included in this PR? ffi to allow writing with the cuda compatible strategy, which only includes encodings that we have gpu kernels for, and also uses the cuda flat layout that uses the array tree from the footer. Also exclude alp-rd and pco from the cuda compatible encoding allowlist for completeness Signed-off-by: Onur Satici <onur@spiraldb.com>
## Rationale for this change We don't expose a way in ffi to use the pinned buffer pool on reading vortex files into the GPU ## What changes are included in this PR? cuda session now owns the buffer pool, and that is shared on scans done with this context --------- Signed-off-by: Onur Satici <onur@spiraldb.com>
…ite (vortex-data#8748) Implement VortexReadAt and VortexWrite using jvm native interfaces to be able to use java readers and writers natively in vortex
## Summary Decode CUDA FSST arrays directly into Arrow’s standard offset-based `Utf8`/`Binary` layout: `i32` offsets plus one contiguous device values buffer. Previously, FSST decoded to `Utf8View`/`BinaryView`. Since cuDF does not consume view arrays, it then had to materialize the same offset-based representation itself. Direct varbin export avoids that additional conversion and enables BenchPress GPU reads for Vortex string columns. ## Behavior changes - CUDA sessions now default variable-length exports to standard Arrow `Utf8`/`Binary`. - Consumers that support views can opt in with: ```rust CudaSession::with_varbin_export_layout(VarBinExportLayout::VarBinView) --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Currently LLVM cannot create a `bitcast <i1 x W> -> iW` instruction (https://llvm.org/docs/LangRef.html#bitcast-to-instruction), this may be added going forwards to support byte->bit packing case `collect_bool`. This means we must add aarch specific collect_bool impls as done here. Gives between 10 and 25x improvements Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
Currently we have a few places where we have arch/feature specific kernels, they current use slower feature detection for each call. This PR add a cache for the kernel to use and uses this pattern in the codebase. I think we would later want to register different kernels in the vortex session instead of this, but that can wait. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Tracking issue: vortex-data#8769 and vortex-data#7882 Splits the type-ID foundation for vortex-data#8769 into its own change. - changes Union variant type IDs from `i8` to `u8` across the dtype APIs, protobuf, FlatBuffers, and generated bindings - supports the full `u8` tag range - supports up to 256 Union variants with consecutive tags `0..=255` Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Tracking issue: vortex-data#8769 Adds the breaking Union dtype foundation before scalar and array support. ### Changes - changes `DType::Union` to carry independent outer nullability - keeps outer nullability separate from the nullability of each variant - removes runtime-derived Union nullability - updates dtype coercion, serialization, display, and downstream matches This changes the serialized Serde representation of Union dtypes and will require the compatibility override to be reviewed explicitly. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
# List Layout
Decomposes a `list<T>` column into three independently-written
sub-columns — `elements`,
`offsets`, and (when nullable) `validity` — each backed by its own
layout, under a single
`vortex.list` node.
```
vortex.list (ListLayout)
├── elements : <T's layout> element space — one entry per list *element*
├── offsets : primitive u64 row space + 1 — n+1 entries for n lists, monotonic
└── validity : bool row space — present iff the list is nullable
```
There are three pieces: the **dispatcher** enables using `ListLayout`
for arrays with `DType::List`, the **writer** shreds a list stream into
the three child streams, and the **reader** reassembles only the
children an expression actually needs.
Note that this a prototype and is not a performance improvement over
flattening lists... yet. As such, this PR does not yet enable writing
`ListLayout` for benchmarks.
---
## 1. `Recursive Writer Dispatch`
`TableStrategy` inspects the dtype of the stream it's handed and routes:
| dtype | written by |
| -------- | ------------------------------------------------------ |
| struct | `StructStrategy` |
| list | `ListLayoutStrategy` (when list decomposition is on) |
| anything | the configured leaf strategy |
To enable recursive handling of nested lists (elements are lists) or
structs it hands **a descended copy of itself** as
the child strategy. So a `list<struct<{ a, list<i32> }>>` recurses with
no manual wiring — each
level dispatches on its own dtype.
List decomposition is gated (`use_experimental_list_layout()` /
`VORTEX_EXPERIMENTAL_LIST_LAYOUT`,
or an explicit builder call). When off, lists fall through to the leaf
strategy unchanged.
---
## 2. `ListLayoutStrategy`
A *structural* writer: it does not compress or inspect element dtypes;
it shreds a list stream into
its three sub-streams and hands each to its own downstream strategy,
producing one `ListLayout`.
To do so it must:
1. **Canonicalize** each incoming chunk to `List` parts. A
gapped/reordered `ListView` is rebuilt
into a gapless `List` first (`list_from_list_view`).
2. **Rebase offsets to global `u64`.** Each chunk's local offsets are
shifted by `element_base`
(the running count of elements emitted so far) and widened to `u64`, and
the duplicated boundary
offset is dropped on every chunk after the first — so the concatenation
of all chunks is one
monotonic `[0 … total_elements]` array of length `row_count + 1` that
indexes into the single
concatenated `elements` child.
3. **Fan out** `elements`, `offsets`, and `validity` onto three bounded
(capacity-1) channels; each
child is written concurrently by its own strategy via `spawn_nested`.
The producer future is
joined with the child writers (`try_join`) so a producer error surfaces
instead of being hidden
as an early channel close.
Non-list input is forwarded to a `fallback` strategy unchanged.
Because each child is written through an *independent* strategy, the
children's physical shape is
decoupled: routing `elements` through the recursive dispatcher (→
repartition → chunked) makes it a
`ChunkedLayout` chunked in element space, while `offsets`/`validity` can
be written with a different
(e.g. row-space) strategy.
---
## 3. `ListReader`
To read as little as possible, `projection_evaluation` first
**classifies the
expression** (`get_necessary_list_children`) into the minimal set of
children it needs, then routes
to a matching read path:
| class (`ListChildrenNeeded`) | example expr | reads | path |
| ---------------------------- | ------------------------------- |
--------------------- | ------------------------- |
| `Validity` | `is_null(x)` / `is_not_null(x)` | validity only |
`project_validity` |
| `OffsetsAndValidity` | `list_length(x)` | offsets + validity |
`project_offsets_validity`|
| `All` | `x`, or anything over elements | elements + the rest |
`project_all` |
`project_all` materializes the list, and itself picks between two
sub-paths:
- **Whole-chunk / concurrent** (`project_all_concurrent`) — used for a
full-column range, or when
`elements` is a single flat segment (nothing to skip). Fires the
`elements`, `offsets`, and
`validity` reads concurrently, assembles the list, slices to the
requested range, and filters.
No offsets→elements ordering dependency.
- **Bounded** (`project_all_elements_bounded`) — used for a strict
sub-range over **chunked**
elements. Reads `offsets[range]`, decodes the first/last offset to bound
the elements read to
`[first … last)`, fetches only the element chunks that range overlaps,
and rebases the offsets
to index into the sliced buffer. Costs one offsets→elements round-trip
but avoids reading the
whole elements column for a partial scan split.
## TODOs:
* Add list-related stats for better pruning.
* Pruning for element zones.
* Correctly handle validity. Right now, validity is written as a flat
layout. We also probably do not need a reader.
* Better handling of arbitrarily nested lists.
* Perf improvement.
---------
Signed-off-by: "Matthew Katz" <katz@spiraldb.com>
Signed-off-by: "Matt Katz" <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Co-authored-by: Matthew Katz <katz@spiraldb.com>
## Rationale for this change The live benchmarks website is now located at https://github.com/vortex-data/benchmarks-website and it has been going strong for the past few weeks, so I think it is time to remove this. ## What changes are included in this PR? Deletes the old website code from here. ## What APIs are changed? Are there any user-facing changes? N/A Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
We used to walk through the patches after decoding ALP then apply them one by one. Now we scatter them among workers and apply these in parallel --------- Signed-off-by: Onur Satici <onur@spiraldb.com>
…ks. (vortex-data#8672) ## Rationale for this change `TableStrategy` currently hides panics that happen in the individual column streams. ## What changes are included in this PR? Instead of detaching the fanout work, we await it and make sure everything finishes successfully. ## What APIs are changed? Are there any user-facing changes? No API change, added test to verify panic propagation. Signed-off-by: Adam Gutglick <adam@spiraldb.com> (cherry picked from commit 4abe3d0)
Upgrades the spiceai-54 line from yanked 0.76.0 to upstream 0.79.0 (carries the TableStrategy panic-propagation fix natively, plus vortex-data#8474 BitBufferMut overflow fix, vortex-data#8667 constant-detection hardening, vortex-data#8406 dict-layout probe with configured compressor). Conflict resolutions: - vortex-layout/src/layouts/table.rs: upstream (0.79 contains our cherry-picked vortex-data#8672 plus the 0.78 TableStrategy/StructStrategy split) - vortex-arrow/* (moved from vortex-array/src/arrow/): upstream — the fork's d41e094 was 0.76 deprecation catch-up, superseded upstream - vortex-datafusion/src/persistent/sink.rs: upstream — the fork's #33 target-file-size sink patch is superseded by spice2's in-repo crates/vortex (vortex-datafusion path crate), which is what spiced actually consumes - .github/workflows/ci.yml: union — keep fork-only runner adaptations (guarded by repository != vortex-data/vortex) plus upstream's enable-sccache additions - benchmarks-website/: upstream removal accepted
This was referenced Jul 20, 2026
sgrebnov
approved these changes
Jul 20, 2026
pull Bot
pushed a commit
to TheRakeshPurohit/spiceai
that referenced
this pull request
Jul 21, 2026
* fix(vortex): bump pins past yanked 0.76.0 TableStrategy silent-corruption bug (spiceai#11942) Upstream yanked vortex 0.76.0 for a latent correctness bug: TableStrategy's detached column-chunk fan-out hides panics from parallel column-writer subtasks, letting surviving writers finalize columns at different lengths silently (the file still reports full row count). Every Cayenne Vortex write (staged deltas, mem-tier checkpoints, compaction outputs) goes through this writer. Likely root cause of the SF1000 chbench_q10 c_city divergence that forced spiceai#11910 to revert spiceai#11826 (the SF1000 replication-convergence lever). Pins move to fork branch lukim/spiceai-54-tablestrategy-fix (68badd6899c82156a3a6d4c4cf9906051a674ece) = the prior pin + a clean cherry-pick of upstream vortex-data/vortex#8672; the upstream regression test (table_fanout_panic_propagates) passes on the fork branch. Behavior changes only when a writer subtask dies mid-file: the write now fails loudly instead of persisting a desynced file. Adds light_encode_roundtrip: a cell-exact round-trip regression test over the light delta-encoding scheme sets (dict-cardinality boundaries incl. 255/256/257 and the u16 edge, near-unique + long out-of-line strings, null-dominated sparse, constants, sliced and jumbo multi-block batches), which also refutes the deterministic scheme-corruption alternative for the q10 divergence (0 mismatches at levels 1/2/3 + FULL). * style: cargo fmt on light_encode_roundtrip test * feat(vortex): upgrade fork pins to Vortex 0.79.0 (spiceai#11942) Moves the vortex pins from the TableStrategy-hotfix branch to lukim/spiceai-54-vortex-0.79.0 (spiceai/vortex#74): upstream 0.79.0 merged into the spiceai-54 line. 0.79.0 natively carries the spiceai#8672 panic-propagation fix plus spiceai#8474 (BitBufferMut overflow), spiceai#8667 (constant detection hardcoded into the cascading compressor), and spiceai#8406 (dict-layout probes with the configured compressor — benefits the restricted light delta scheme sets). Arrow requirement is unchanged (58.3). Port surface (all mechanical): - vortex::array::arrow / vortex::dtype::arrow -> vortex::arrow (the arrow integration moved to the new vortex-arrow crate) - DType::to_arrow_schema() -> session.arrow().to_arrow_schema(dtype) - Arc<dyn Datum>::try_from(scalar) -> scalar.to_arrow_datum() (ToArrowDatum) - Int/Float/StringConstantScheme removed from the light delta scheme sets: constant detection is built into the 0.79 cascade (spiceai#8667), so level 1 is now Sparse-only and levels 2-3 are Sparse+Dict(+numeric); docs updated. Tests: cayenne delta_encoding 8/8; light_encode_roundtrip 0 mismatches at 0.79; vortex-datafusion 256/256; clippy clean (cayenne + vortex-datafusion). * chore(vortex): repoint pins at merged spiceai-54 (vortex#73 + vortex#74 landed) * review: drop Box::leak for pool column names; fix stale level-1 scheme comment * feat(cayenne): swap the light delta-encoding path to string Zstd only The auto light level (2) becomes a single string::ZstdScheme pass instead of Sparse+Dict. Measured on the 417k-row round-trip harness (release, 16 CH-benCH-shaped columns): zstd-only 103.1 MB vs Sparse+Dict 172.5 MB (-40%) vs the FULL BtrBlocks+FSST cascade 115.8 MB (-11%), at statistically equal light-path encode wall time (~100 ms; FULL 126 ms); dict+zstd measured only ~4% smaller than zstd-only, so the light path keeps the single-scheme set. Numeric columns stay canonical on the light path (compaction's FULL re-encode optimizes the long-lived artifact); level 3 keeps the rich dict+numeric(+Zstd) set as the explicit A/B rung. Level semantics stay monotonic: 1 = sparse-only detection, 2 = zstd pass, 3 = rich light. Byte-size relief matters most on I/O-bound venues (EBS-class CI runners) where transient-file inflation is hot-path write I/O; encode-CPU relief vs the FULL cascade is preserved. Round-trip harness mirrors the new sets and now prints per-level encode timing. * review: cap harness pools in CI mode; avoid per-row String clones; sync trunk * review: correct CI-default dataset row counts in harness docs * test(cayenne): de-tune size-sensitive compaction file-count bound The exact post-compaction count tracks the light delta-encoding's file sizes; the Zstd-only light set graduates one more file at the 1 MiB target than the prior Sparse+Dict set (7 vs 6 for these 20 appends). Assert the behavior under test (20 appends fold to a small handful, rows preserved) with headroom for encoding-size drift.
github-actions Bot
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jul 22, 2026
* fix(vortex): bump pins past yanked 0.76.0 TableStrategy silent-corruption bug (#11942) Upstream yanked vortex 0.76.0 for a latent correctness bug: TableStrategy's detached column-chunk fan-out hides panics from parallel column-writer subtasks, letting surviving writers finalize columns at different lengths silently (the file still reports full row count). Every Cayenne Vortex write (staged deltas, mem-tier checkpoints, compaction outputs) goes through this writer. Likely root cause of the SF1000 chbench_q10 c_city divergence that forced #11910 to revert #11826 (the SF1000 replication-convergence lever). Pins move to fork branch lukim/spiceai-54-tablestrategy-fix (68badd6899c82156a3a6d4c4cf9906051a674ece) = the prior pin + a clean cherry-pick of upstream vortex-data/vortex#8672; the upstream regression test (table_fanout_panic_propagates) passes on the fork branch. Behavior changes only when a writer subtask dies mid-file: the write now fails loudly instead of persisting a desynced file. Adds light_encode_roundtrip: a cell-exact round-trip regression test over the light delta-encoding scheme sets (dict-cardinality boundaries incl. 255/256/257 and the u16 edge, near-unique + long out-of-line strings, null-dominated sparse, constants, sliced and jumbo multi-block batches), which also refutes the deterministic scheme-corruption alternative for the q10 divergence (0 mismatches at levels 1/2/3 + FULL). * style: cargo fmt on light_encode_roundtrip test * feat(vortex): upgrade fork pins to Vortex 0.79.0 (#11942) Moves the vortex pins from the TableStrategy-hotfix branch to lukim/spiceai-54-vortex-0.79.0 (spiceai/vortex#74): upstream 0.79.0 merged into the spiceai-54 line. 0.79.0 natively carries the #8672 panic-propagation fix plus #8474 (BitBufferMut overflow), #8667 (constant detection hardcoded into the cascading compressor), and #8406 (dict-layout probes with the configured compressor — benefits the restricted light delta scheme sets). Arrow requirement is unchanged (58.3). Port surface (all mechanical): - vortex::array::arrow / vortex::dtype::arrow -> vortex::arrow (the arrow integration moved to the new vortex-arrow crate) - DType::to_arrow_schema() -> session.arrow().to_arrow_schema(dtype) - Arc<dyn Datum>::try_from(scalar) -> scalar.to_arrow_datum() (ToArrowDatum) - Int/Float/StringConstantScheme removed from the light delta scheme sets: constant detection is built into the 0.79 cascade (#8667), so level 1 is now Sparse-only and levels 2-3 are Sparse+Dict(+numeric); docs updated. Tests: cayenne delta_encoding 8/8; light_encode_roundtrip 0 mismatches at 0.79; vortex-datafusion 256/256; clippy clean (cayenne + vortex-datafusion). * chore(vortex): repoint pins at merged spiceai-54 (vortex#73 + vortex#74 landed) * review: drop Box::leak for pool column names; fix stale level-1 scheme comment * feat(cayenne): swap the light delta-encoding path to string Zstd only The auto light level (2) becomes a single string::ZstdScheme pass instead of Sparse+Dict. Measured on the 417k-row round-trip harness (release, 16 CH-benCH-shaped columns): zstd-only 103.1 MB vs Sparse+Dict 172.5 MB (-40%) vs the FULL BtrBlocks+FSST cascade 115.8 MB (-11%), at statistically equal light-path encode wall time (~100 ms; FULL 126 ms); dict+zstd measured only ~4% smaller than zstd-only, so the light path keeps the single-scheme set. Numeric columns stay canonical on the light path (compaction's FULL re-encode optimizes the long-lived artifact); level 3 keeps the rich dict+numeric(+Zstd) set as the explicit A/B rung. Level semantics stay monotonic: 1 = sparse-only detection, 2 = zstd pass, 3 = rich light. Byte-size relief matters most on I/O-bound venues (EBS-class CI runners) where transient-file inflation is hot-path write I/O; encode-CPU relief vs the FULL cascade is preserved. Round-trip harness mirrors the new sets and now prints per-level encode timing. * review: cap harness pools in CI mode; avoid per-row String clones; sync trunk * review: correct CI-default dataset row counts in harness docs * test(cayenne): de-tune size-sensitive compaction file-count bound The exact post-compaction count tracks the light delta-encoding's file sizes; the Zstd-only light set graduates one more file at the 1 MiB target than the prior Sparse+Dict set (7 vs 6 for these 20 appends). Assert the behavior under test (20 appends fold to a small handful, rows preserved) with headroom for encoding-size drift.
github-actions Bot
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jul 23, 2026
* fix(vortex): bump pins past yanked 0.76.0 TableStrategy silent-corruption bug (#11942) Upstream yanked vortex 0.76.0 for a latent correctness bug: TableStrategy's detached column-chunk fan-out hides panics from parallel column-writer subtasks, letting surviving writers finalize columns at different lengths silently (the file still reports full row count). Every Cayenne Vortex write (staged deltas, mem-tier checkpoints, compaction outputs) goes through this writer. Likely root cause of the SF1000 chbench_q10 c_city divergence that forced #11910 to revert #11826 (the SF1000 replication-convergence lever). Pins move to fork branch lukim/spiceai-54-tablestrategy-fix (68badd6899c82156a3a6d4c4cf9906051a674ece) = the prior pin + a clean cherry-pick of upstream vortex-data/vortex#8672; the upstream regression test (table_fanout_panic_propagates) passes on the fork branch. Behavior changes only when a writer subtask dies mid-file: the write now fails loudly instead of persisting a desynced file. Adds light_encode_roundtrip: a cell-exact round-trip regression test over the light delta-encoding scheme sets (dict-cardinality boundaries incl. 255/256/257 and the u16 edge, near-unique + long out-of-line strings, null-dominated sparse, constants, sliced and jumbo multi-block batches), which also refutes the deterministic scheme-corruption alternative for the q10 divergence (0 mismatches at levels 1/2/3 + FULL). * style: cargo fmt on light_encode_roundtrip test * feat(vortex): upgrade fork pins to Vortex 0.79.0 (#11942) Moves the vortex pins from the TableStrategy-hotfix branch to lukim/spiceai-54-vortex-0.79.0 (spiceai/vortex#74): upstream 0.79.0 merged into the spiceai-54 line. 0.79.0 natively carries the #8672 panic-propagation fix plus #8474 (BitBufferMut overflow), #8667 (constant detection hardcoded into the cascading compressor), and #8406 (dict-layout probes with the configured compressor — benefits the restricted light delta scheme sets). Arrow requirement is unchanged (58.3). Port surface (all mechanical): - vortex::array::arrow / vortex::dtype::arrow -> vortex::arrow (the arrow integration moved to the new vortex-arrow crate) - DType::to_arrow_schema() -> session.arrow().to_arrow_schema(dtype) - Arc<dyn Datum>::try_from(scalar) -> scalar.to_arrow_datum() (ToArrowDatum) - Int/Float/StringConstantScheme removed from the light delta scheme sets: constant detection is built into the 0.79 cascade (#8667), so level 1 is now Sparse-only and levels 2-3 are Sparse+Dict(+numeric); docs updated. Tests: cayenne delta_encoding 8/8; light_encode_roundtrip 0 mismatches at 0.79; vortex-datafusion 256/256; clippy clean (cayenne + vortex-datafusion). * chore(vortex): repoint pins at merged spiceai-54 (vortex#73 + vortex#74 landed) * review: drop Box::leak for pool column names; fix stale level-1 scheme comment * feat(cayenne): swap the light delta-encoding path to string Zstd only The auto light level (2) becomes a single string::ZstdScheme pass instead of Sparse+Dict. Measured on the 417k-row round-trip harness (release, 16 CH-benCH-shaped columns): zstd-only 103.1 MB vs Sparse+Dict 172.5 MB (-40%) vs the FULL BtrBlocks+FSST cascade 115.8 MB (-11%), at statistically equal light-path encode wall time (~100 ms; FULL 126 ms); dict+zstd measured only ~4% smaller than zstd-only, so the light path keeps the single-scheme set. Numeric columns stay canonical on the light path (compaction's FULL re-encode optimizes the long-lived artifact); level 3 keeps the rich dict+numeric(+Zstd) set as the explicit A/B rung. Level semantics stay monotonic: 1 = sparse-only detection, 2 = zstd pass, 3 = rich light. Byte-size relief matters most on I/O-bound venues (EBS-class CI runners) where transient-file inflation is hot-path write I/O; encode-CPU relief vs the FULL cascade is preserved. Round-trip harness mirrors the new sets and now prints per-level encode timing. * review: cap harness pools in CI mode; avoid per-row String clones; sync trunk * review: correct CI-default dataset row counts in harness docs * test(cayenne): de-tune size-sensitive compaction file-count bound The exact post-compaction count tracks the light delta-encoding's file sizes; the Zstd-only light set graduates one more file at the 1 MiB target than the prior Sparse+Dict set (7 vs 6 for these 20 appends). Assert the behavior under test (20 appends fold to a small handful, rows preserved) with headroom for encoding-size drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades the
spiceai-54line from yanked 0.76.0 to upstream 0.79.0. Contains #73 (the TableStrategy hotfix cherry-pick) as its base; 0.79.0 carries that fix natively plus vortex-data#8474 (BitBufferMut overflow), vortex-data#8667 (constant-detection hardening), and vortex-data#8406 (dict-layout probe with the configured compressor — directly benefits the restricted light scheme sets Spice registers). Arrow requirement is unchanged (58.3), so no arrow bump on the consumer side.Context: spiceai/spiceai#11942.
Conflict resolutions (7 files)
vortex-layout/src/layouts/table.rs→ upstream (contains fix: TableStrategy currently hides panic in spawned column writer tasks. vortex-data/vortex#8672 natively + the 0.78 TableStrategy/StructStrategy split)vortex-arrow/*(moved out ofvortex-array/src/arrow/) → upstream; the fork'sd41e094cfwas 0.76 deprecation catch-up, supersededvortex-datafusion/src/persistent/sink.rs→ upstream; the fork's fix: Ensure target file size is properly respected in vortex sink #33 target-file-size sink patch lives on in spice2's in-repocrates/vortex(thevortex-datafusionactually consumed by spiced).github/workflows/ci.yml→ union (fork-only runner steps guarded byrepository != 'vortex-data/vortex'+ upstream'senable-sccache)benchmarks-website/→ upstream removal acceptedFork patches verified surviving the merge
vortex-layout/src/scan/{layout,split_by}.rsexecutor.rs:336,optimizer/kernels.rs:282)Tests
cargo nextest run -p vortex-layout: 205/205 pass.cargo test -p vortex-layoutshows 7-9 failures from cross-test global-session interference — also present at the pristine0.79.0tag (verified in a clean worktree); upstream CI masks it via nextest's process-per-test isolation. Not introduced by this merge.cargo checkacross all consumed crates: clean.Sequencing
#73 first (unblocks spiceai/spiceai#11943 today), then this; spice2 pin bump to this branch follows as its own PR (import-path port:
vortex::array::arrow→vortex::arrow).