Skip to content

v0.7.0 - tabular data into typed columns, every cell a HyperCast verdict

Choose a tag to compare

@buvinghausen buvinghausen released this 07 Oct 23:32
· 97 commits to master since this release
594c2b0

HyperTabular's first release. It reads delimited text (CSV, TSV, any single-byte ASCII separator) and workbooks (XLSX and ODS) a batch at a time into typed columns, with a HyperCast verdict for every cell. It starts at 0.7.0 so its version matches HyperCast 0.7.0, the release it is built on. That is the first HyperCast with the interop surface and the typed doors every binding here reads cells through. The full list is in CHANGELOG.md.

Install

Language Package
Rust cargo add hypertabular
C# (.NET 11) dotnet add package HyperTabular
Java (JDK 25) implementation("io.github.skunkwerkx:hypertabular:0.7.0")
Go go get github.com/SkunkWerkx/HyperTabular/go@v0.7.0
Swift .package(url: "https://github.com/SkunkWerkx/HyperTabular", from: "0.7.0")
Python (3.11+) pip install hypertabular
Ruby (3.3+) gem install hypertabular
PHP (8.2+) composer require skunkwerkx/hypertabular

Highlights

  • One native core with no standard library, no allocation and no panic. The binding allocates the input, one value array and one verdict array per column, and the table that locates each cell. The core fills them in one call per batch, so the native boundary is crossed once per few thousand rows, not once per cell. rust/check-core.sh checks all three properties on every PR:

    • the library's imports must match an allow-list, so malloc fails it;
    • every export is under dtolnay's no-panic;
    • a counting allocator sees zero allocations across whole inputs.

    The C ABI has 14 exports, hypertabular_version among them. The library carries none of HyperCast's symbols and links beside libhypercast in one program.

  • Delimited text. A forward-only reader driven by a declared Dialect and a declared column plan; nothing is sniffed. A SIMD scanner (AVX2 and PCLMULQDQ on x86-64, NEON and PMULL on arm64) finds the cells, picked at run time. Text cells are spans into the input, not copies. The reader does about 575 MB/s on linux-x64.

  • Workbooks: XLSX (transitional, strict, .xlsm) and ODS. The core opens the container itself, ZIP64 included, with its own zip reader and inflater, because neither miniz_oxide nor zlib-rs passes the no-panic proof. Each numeric cell is converted according to its number format, through HyperCast's typed doors:

    • an exact decimal;
    • an Excel serial snapped to the time the writer stored;
    • a time of day;
    • a duration.

    The 1900 and 1904 date systems are both handled. Hidden sheets are listed and read. Encrypted packages and OLE files are refused by name. A 300 000-row Excel workbook reads in about 0.6 s on linux-x64.

  • A bad value is a verdict, a broken file is an error. A cell that doesn't cast is a HyperCast Fault (Empty, Malformed or OutOfRange, with the offending span), and the read goes on. A structural error is raised only after every intact row before it has been delivered. Examples are a record of the wrong width, or input that ends inside a quoted cell.

  • Eight packages, one shape. Every binding mirrors the Rust API and takes HyperCast's verdict, format and option types from its own package rather than copying them:

    • a batch per read();
    • values a whole column at a time or a cell at a time;
    • line(row) and each cell's raw text;
    • Workbook / Sheet.

    Per language:

    • C#: LibraryImport, plus Native AOT.
    • Java: FFM, plus GraalVM Native Image.
    • Go: cgo, with the core linked in.
    • Swift: a static library.
    • Python: PyO3, as abi3 wheels.
    • Ruby: a Magnus extension in precompiled platform gems, falling back to Fiddle.
    • PHP: ext-ffi.
  • iOS and Mac Catalyst, in C#, Swift and Go, matching HyperCast and HyperUuid 0.7.0. The core is linked in from static archives built and attested beside everything else:

    • C#: ios-arm64, iossimulator-arm64, and maccatalyst on arm64 and x64.
    • Swift: HyperTabularCoreApple.xcframework.
    • Go: ios, iossimulator and maccatalyst builds, selected by build tag.

    CI runs the suites in an iOS simulator and as a Mac Catalyst process on every PR, and links an iOS device build of each.

  • WebAssembly, wherever a binding's toolchain can link the core. The core imports nothing, so each wasm build links the same archive.

    Language How
    Rust The suite runs under wasmtime on wasm32-wasip1, and in the browser on wasm32-unknown-unknown.
    C# Blazor WebAssembly on .NET 11; the package links the core itself.
    Go TinyGo, -target=wasm or wasip1.
    Swift swift.org's WebAssembly SDK.
    Python A Pyodide 314 wheel; the whole pytest suite runs inside Pyodide under Node and in Chrome.
    Ruby The hypertabular-wasm gem, linked into a ruby.wasm interpreter with rbwasm build.
    Java An in-process GraalWasm backend (-Dhypertabular.backend=wasm). It is also the automatic fallback where the jar has no native build.

    PHP is not built for WebAssembly, for the same reasons as HyperCast.

  • Platforms.

    • Tested on real hardware: linux-x64, linux-arm64, linux-musl-x64, linux-musl-arm64, osx-arm64, win-x64 and win-arm64.
    • osx-x64 is built and tested at the core.
    • Plus iOS, Mac Catalyst and WebAssembly as above.
  • A shared conformance corpus. 33 delimited cases and 198 workbook cases, replayed byte for byte by the Rust suite and by every binding. The workbooks include files written by Excel for Windows and the web, LibreOffice and Google Sheets.

  • Fuzzed. Four libFuzzer targets, about four million executions before release, with nothing found in the core:

    • delimited text;
    • raw XLSX/ODS containers;
    • workbook XML;
    • the inflater, checked against zlib-rs.

Verifying

Every native library and static archive is built by CI and carries a build-provenance attestation:

gh attestation verify go/staticlib/linux_amd64/libhypertabular.a \
  --repo SkunkWerkx/HyperTabular --signer-repo SkunkWerkx/.github

The Python wheels and the crate are attested by the forge the same way, so they also need --signer-repo SkunkWerkx/.github. The gems and the NuGet package are attested by this repo's own release.yml, so --repo SkunkWerkx/HyperTabular alone verifies them.