Commit fa35a45
authored
feat(bootstrap)!: source hyperd from the PyPI tableauhyperapi wheels (#254)
## Motivation
`hyperdb-bootstrap` fetched `hyperd` from Tableau's Hyper **Java API
zip**, whose
filename embeds an opaque `build_id` (e.g. `r07abb490`) that **cannot be
derived
from the version**. Discovering it meant scraping the public releases
page — and
that scraper (`src/scrape.rs`, behind `--latest`) has been **broken for
three-plus
releases without anyone noticing**, because its tests ran against a
synthetic
fixture rather than the live page. Two independent defects:
- the heading regex expects `<h3>VERSION [DATE]</h3>`, but Docusaurus
renders
`0.0.26479 <!-- -->[September 3 2026]`, which `\s*` cannot span;
- the build-id capture hardcodes `(rc[a-z0-9]+)`, while every build id
since
`0.0.24457` has been `r` + hex.
The PyPI `tableauhyperapi` wheels carry the same engine behind a **fully
constructible** URL, and **PyPI publishes a sha256 per file**:
```text
https://files.pythonhosted.org/packages/py3/t/tableauhyperapi/tableauhyperapi-{version}-py3-none-{wheel_tag}.whl
```
So `--latest` is **deleted, not fixed**: with a constructible URL and
published
digests there is nothing left for it to do.
## The bytes are unchanged
The strongest evidence this migration is faithful — the `hyperd`
extracted from
the `macosx_13_0_arm64` wheel is **bit-identical to the `hyperd` in the
Java zip
for the same release**, sha256
`aef5c81970bb4d84d06fb9513d5ffd722526fce779632a0c5f63d87b6450e478`. To
be precise
about what is and isn't established here: that cross-envelope equality
was
established upstream of this PR and I did not re-derive it. What I
verified
locally is that the wheel path produces exactly that binary:
```text
$ shasum -a 256 .hyperd/current/hyperd
aef5c81970bb4d84d06fb9513d5ffd722526fce779632a0c5f63d87b6450e478
$ stat -f%z .hyperd/current/hyperd
277836448
$ file .hyperd/current/hyperd
.hyperd/current/hyperd: Mach-O 64-bit executable arm64
$ .hyperd/current/hyperd --version
Hyper version main.0.0.26479.r96880f6a
```
Same build, different envelope. Note the `--version` line corroborates
this
independently: the wheel's binary self-reports build `r96880f6a`, which
is exactly
the `build_id` the Java zip carries for `0.0.26479` (see
`chore/post-rc-cleanup`'s
`8c99d20`). It is the same engine build reached by a different filename.
This is *not* a claim that the wheel matches the engine currently on
`main`
(`0.0.26359`, build `r07abb490`) — that is a different build. The
equality is
same-release, cross-envelope.
Both binaries report `minos 13.0`, so the `macosx_13_0` wheel tag is
**not** a
raised support floor — no contributor loses support. Wheels are
~3.6–4.5% smaller
than the Java zips.
Bumping the pin no longer means downloading four ~80 MB archives and
hashing them
by hand — the digests come off the JSON API. They are **still
committed**: a hash
in git is an attestation independent of the host serving the bytes.
## Breaking changes
Public API removed:
| Removed | Replacement |
|---|---|
| `PinnedRelease::build_id`, `InstalledHyperd::build_id` | `.version` is
the only release identifier |
| `PinnedRelease::version_tag()` | `.version` |
| `VersionSource::ScrapeLatest`, the `scrape` module | none — deleted |
| `Error::Http`, `Error::HttpStatus`, `Error::ScrapeFailed` | none —
served the scraper only |
| CLI `--latest`, `--build-id` | `--version X` alone is now a complete
source |
| `regex`, `reqwest`, `rustls` deps | none — no in-process HTTP client
remains |
`url::build_download_url` is now **fallible** (`Result<String, Error>`);
a platform
with no pinned wheel tag is `Error::MissingWheelTag` rather than a
guess.
`PinnedRelease::wheel_tag_for(Platform)` is new.
The install layout is keyed on the version alone: `<dest>/0.0.26479/`
(was
`<dest>/0.0.26479.r96880f6a/`), and `current/VERSION` now contains just
`0.0.26479`. Nothing in the repo reads that file programmatically.
Dropping `reqwest`/`rustls` also **retires the rustls crypto-provider
workaround**
the CHANGELOG records as a past breaking change (`rustls-no-provider`
plus a
`OnceLock` installing ring). Verified `aws-lc-rs` is absent
workspace-wide and
that `hyperdb-bootstrap` no longer reaches `reqwest`, `rustls`, or
`regex` at all.
## Versioning: breaking, released as `1.0.0-rc.2`
**Decided — no reviewer action needed.** The break is now marked
honestly *and*
the version is pinned, which are two independent things:
- The subject is `feat(bootstrap)!:` and the migration commit carries a
`BREAKING CHANGE:` footer enumerating every removed item. The break is
real
and is recorded as such.
- An empty follow-up commit, `chore: release 1.0.0-rc.2`, carries a
`Release-As: 1.0.0-rc.2` footer, which pins the next release inside the
`1.0.0-rc` line.
Without that footer the `!` would compute **2.0.0**: the workspace is
already on
`1.x`, and `bump-minor-pre-major` in `release-please-config.json` is
gated on
`version.isPreMajor` (`major < 1`), so it does not apply at
`1.0.0-rc.1`.
`Release-As:` wins regardless — release-please's
`DefaultVersioningStrategy`
returns a `CustomVersionUpdate` from the `RELEASE AS` note *before* it
reads the
breaking-change tally. This is the same mechanism, and the same
empty-commit
shape, that produced `1.0.0-rc.1` (`7bf2dff`), and it is what
[`docs/GITHUB_OPERATIONS.md` →
Pre-releases](https://github.com/tableau/hyper-api-rust/blob/main/docs/GITHUB_OPERATIONS.md#pre-releases)
documents.
Simulated against `release-please@17.11.2` (the version
`release-please-action@v5` pins) using its real
`parseConventionalCommits` and
`DefaultVersioningStrategy`: `1.0.0-rc.1` → **`1.0.0-rc.2`** under a
merge
commit *and* under a squash merge. Dropping the footer from the same
input
yields `2.0.0-rc.1`, which is the outcome this pin exists to prevent.
> **Do not merge a release PR that says anything other than
`1.0.0-rc.2`.**
> #253 merges first, and release-please will run on that push and open a
> `chore(main): release …` PR computed from #253's commits alone —
before this
> PR's `Release-As:` footer is on `main`. That PR will show the wrong
version.
> It updates itself once this PR lands; leave it alone until then.
## Why wheel tags live in the pin file
They are deliberately *not* hardcoded in Rust. They are not guaranteed
stable
across releases (arm64 wheels only exist from `0.0.19484`; a future
macOS floor
bump would change `macosx_13_0_arm64`), and a wrong tag is a **silent
404** on one
platform only. Keeping them as pin data makes any such change a visible
pin edit,
and `build.rs` now fails the build if any supported platform lacks one.
(Empirically the four tags are unchanged from `0.0.19484` through
`0.0.26479`.)
## Stronger `verify`
`hyperdb-bootstrap verify` now cross-checks every pinned digest against
the digest
PyPI publishes for that exact wheel filename, on top of HEAD-ing the
four URLs —
so it validates the **exact pinned bytes** rather than merely that the
CDN serves
something at that path, and it catches a stale `[wheel_tag]` explicitly:
```text
$ cargo run --release -p hyperdb-bootstrap --bin hyperdb-bootstrap -- verify
verifying hyperd 0.0.26479...
OK macos-arm64 [200] .../tableauhyperapi-0.0.26479-py3-none-macosx_13_0_arm64.whl
digest matches PyPI
OK macos-x86_64 [200] .../tableauhyperapi-0.0.26479-py3-none-macosx_10_11_x86_64.whl
digest matches PyPI
OK linux-x86_64 [200] .../tableauhyperapi-0.0.26479-py3-none-manylinux2014_x86_64.whl
digest matches PyPI
OK windows-x86_64 [200] .../tableauhyperapi-0.0.26479-py3-none-win_amd64.whl
digest matches PyPI
all platforms reachable with matching digests.
```
## npm dual pin
`npm-build-publish.yml` carries its own independent pin (this is why
`0.7.1` once
shipped npm with a different engine than crates.io). It is migrated to
the wheel
URL, `HYPERD_BUILD_ID` is gone, and the matrix gains `hyperd-wheel-tag`.
`verify-npm-hyperd-pin.py` keeps its version + digest cross-check and
**gains a
wheel-tag cross-check**, since the tag is a new drift vector.
Empirically confirmed (not assumed) that the matrix `hyperd-sha256`
values are
digests of the **downloaded archive**, not the extracted binary — the
step hashes
`hyperd-archive.whl` — so they are the wheel digests and equal the
toml's
`[sha256]`.
I did **not** make the workflow read the toml directly. The matrix must
be static
YAML, so it would need a `tomllib` call inside the per-platform matrix
job, which
runs on a Windows runner under git-bash where `python3` may not be on
`PATH` — an
untestable portability risk for no correctness gain over the guard.
**Recommended
as a follow-up** alongside converting the matrix to a `setup`-job JSON
output.
## Verification
Run on macOS arm64 (Apple Silicon):
- `cargo build -p hyperdb-bootstrap` and `cargo build --workspace` —
clean
- `cargo test -p hyperdb-bootstrap` — 26 lib + 4 integration + 1 doc, 0
failed
(up from 12 lib + 2 integration; +16 runnable tests)
- `cargo test --workspace` — **1584 passed, 0 failed**, against the
freshly
downloaded engine
- `make test` — **1519 passed, 0 failed**
- `cargo fmt --all -- --check` and
`cargo clippy --workspace --all-targets --all-features -- -D warnings` —
clean
- `npx markdownlint-cli2` — zero new findings; the 5 in `AGENTS.md` /
`DEVELOPMENT.md` are the pre-existing MD040 backlog, identical before
and after
- `verify-npm-hyperd-pin.py` passes against the new toml, and **fails
correctly**
when a wheel tag is perturbed
- The workflow's `find`-based extraction was replayed against the real
wheel entry
names: `HYPER_DIR=hyperd-raw/tableauhyperapi/bin/hyper`, `hyperd` found,
`LICENSE*`/`NOTICE*` glob matches `dist-info/`
`extract.rs` is **unchanged** apart from doc comments — its "skip one
optional
top-level directory, then require a `lib/hyper` or `bin/hyper` pair"
logic already
absorbs the `tableauhyperapi` wrapper. New tests pin that against the
real wheel
entry names, including the Windows `hyperd.exe` + `crashdumper.exe`
case.
### Not verified here
- **Only macOS arm64 was executed.** The other three platforms rest on
CI. Their
URLs and digests *are* machine-verified (all four HEAD 200 with matching
PyPI
digests), and I downloaded the Windows and Linux wheels to confirm their
`bin/hyper/` contents (`hyperd.exe` + `crashdumper.exe`; `hyperd`) and
digests —
but no `hyperd` was run on Linux or Windows.
- The engine links only system frameworks (`otool -L`) and `bin/hyper/`
contains
no shared libraries on any of the three wheels I inspected, so the npm
shared-library copy loop finds nothing to copy — as before.
---
Repeated here as a trailing footer so the version is pinned on the merge
commit
too: this repo merges with `merge_commit_message = PR_BODY`, so the body
below
becomes the merge commit's body. Same value as the footer on
`chore: release 1.0.0-rc.2`, so it is redundant rather than conflicting.
Release-As: 1.0.0-rc.224 files changed
Lines changed: 1450 additions & 695 deletions
File tree
- .agents/skills/update-hyperd-release
- .claude/skills/update-hyperd-release
- .github
- scripts
- workflows
- hyperdb-bootstrap
- src
- bin
- tests
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
5 | | - | |
6 | | - | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
19 | 26 | | |
20 | 27 | | |
21 | 28 | | |
| |||
36 | 43 | | |
37 | 44 | | |
38 | 45 | | |
| 46 | + | |
39 | 47 | | |
40 | 48 | | |
41 | 49 | | |
42 | 50 | | |
43 | 51 | | |
44 | | - | |
45 | 52 | | |
46 | 53 | | |
47 | 54 | | |
| |||
51 | 58 | | |
52 | 59 | | |
53 | 60 | | |
54 | | - | |
55 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
56 | 72 | | |
57 | 73 | | |
58 | 74 | | |
59 | 75 | | |
60 | | - | |
| 76 | + | |
61 | 77 | | |
62 | 78 | | |
63 | 79 | | |
| |||
72 | 88 | | |
73 | 89 | | |
74 | 90 | | |
75 | | - | |
76 | | - | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
77 | 94 | | |
78 | 95 | | |
79 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
26 | 30 | | |
27 | | - | |
28 | 31 | | |
29 | 32 | | |
30 | 33 | | |
| |||
92 | 95 | | |
93 | 96 | | |
94 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
95 | 102 | | |
96 | 103 | | |
97 | 104 | | |
98 | 105 | | |
99 | | - | |
| 106 | + | |
| 107 | + | |
100 | 108 | | |
101 | 109 | | |
102 | 110 | | |
103 | 111 | | |
104 | 112 | | |
105 | | - | |
| 113 | + | |
| 114 | + | |
106 | 115 | | |
107 | 116 | | |
108 | 117 | | |
109 | 118 | | |
110 | | - | |
| 119 | + | |
| 120 | + | |
111 | 121 | | |
112 | 122 | | |
113 | 123 | | |
114 | 124 | | |
115 | | - | |
| 125 | + | |
| 126 | + | |
116 | 127 | | |
117 | 128 | | |
118 | 129 | | |
| |||
154 | 165 | | |
155 | 166 | | |
156 | 167 | | |
| 168 | + | |
157 | 169 | | |
158 | 170 | | |
159 | 171 | | |
160 | | - | |
161 | | - | |
162 | | - | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
163 | 178 | | |
164 | | - | |
| 179 | + | |
165 | 180 | | |
166 | | - | |
| 181 | + | |
167 | 182 | | |
168 | 183 | | |
169 | 184 | | |
| |||
175 | 190 | | |
176 | 191 | | |
177 | 192 | | |
178 | | - | |
179 | | - | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
180 | 198 | | |
181 | 199 | | |
182 | 200 | | |
| |||
195 | 213 | | |
196 | 214 | | |
197 | 215 | | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
198 | 220 | | |
199 | 221 | | |
200 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
5 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
49 | | - | |
| 50 | + | |
| 51 | + | |
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
16 | 20 | | |
17 | 21 | | |
18 | 22 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
384 | 384 | | |
385 | 385 | | |
386 | 386 | | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
391 | 390 | | |
392 | 391 | | |
393 | 392 | | |
| |||
420 | 419 | | |
421 | 420 | | |
422 | 421 | | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
428 | 425 | | |
429 | 426 | | |
430 | 427 | | |
| |||
435 | 432 | | |
436 | 433 | | |
437 | 434 | | |
438 | | - | |
439 | | - | |
440 | | - | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
441 | 439 | | |
442 | 440 | | |
443 | 441 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
144 | | - | |
145 | | - | |
| 144 | + | |
| 145 | + | |
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
150 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
151 | 152 | | |
152 | 153 | | |
153 | 154 | | |
| |||
0 commit comments