test(self-packaging #46): integration tests for round-trip behaviours#56
Merged
Merged
Conversation
This was referenced May 22, 2026
Part of #40. Implements integration tests for the spike's documented round-trip behaviours, plus a fix surfaced by the tests. New file: test/integration/test_self_packaging.py (8 tests, ~24 sec wall on Linux x86_64 debug build): #2 pack_produces_bundled_executable `flapi pack --in <fixture> --out <new>` succeeds, output exists, exec bits preserved (Unix), file size >= host binary size. #4 info_reports_bundle_contents `<bundled> info` shows EOCD offset, bundle size, and lists every fixture entry by name. #5 self_hosting_rebundle The bundled binary can pack a new bundled binary (`<bundled> pack ...`), which itself has a valid bundle. #6 repack_idempotence Three rounds of packing using each previous output as host binary -- file size identical across rounds. #7 truncated_bundle_falls_back_to_filesystem Chop 1 KiB off the tail. `info` reports "none". A `--validate-config -c <fixture>/flapi.yaml` invocation exits cleanly (no SIGSEGV/SIGABRT) -- the spike safety-net behaviour. #8 unpack_restores_all_entries `<bundled> unpack --to <dir>` writes every entry; CSV bytes round-trip byte-for-byte. pack_refuses_secret_files_by_default `.env` in input -> non-zero exit, "secret" in error message, AND no half-written output binary on disk. pack_with_allow_secrets_bundles_env_files --allow-secrets override bundles `.env` despite the deny list; `info` lists it. Implementation fix (src/pack.cpp): `Pack()` previously copied the multi-GB host binary BEFORE walking the input tree and validating the secret deny list. A rejected `.env` therefore left a half-written 1.7 GB file in the user's tmp dir. Re-ordered: walk + validate first, then copy + append. Surfaced by `test_pack_refuses_secret_files_by_default`. Spike behaviours #1, #3, #9 (filesystem-mode + bundled-mode HTTP + read_csv('embed://...') endpoint) require server-lifecycle test infrastructure plus resolving bundled-mode config-path lookup (template.path resolution from inside an `embed://` config) -- deferred to a follow-up issue. The DuckDB-side embed:// integration is already proven in-process by the unit tests in #44. Closes #46 (partial -- server-lifecycle behaviours deferred).
c018420 to
d8f5ca1
Compare
jrosskopf
added a commit
that referenced
this pull request
May 22, 2026
Part of #40. Closes #49. Adds four new CI jobs that exercise the self-packaging surface on every platform flapi builds for: pack-smoke-linux-amd64 ubuntu-24.04 pack-smoke-linux-arm64 ubuntu-24.04-arm pack-smoke-macos macos-latest pack-smoke-windows windows-latest Each job: - downloads the platform's `flapi` artifact from the existing windows-build / linux-build / osx-universal-build jobs - builds a tiny fixture tree (flapi.yaml + one endpoint + sample SQL) - runs `flapi pack --in fixture --out out-a` - runs `out-a info`, asserts the entry list contains the fixture files - runs `out-a unpack --to extracted`, diffs files byte-for-byte - runs a second `flapi pack ... --out out-b` with the same `SOURCE_DATE_EPOCH=1700000000` and asserts `sha256(out-a) == sha256(out-b)` -- the reproducible-build invariant baked into archive_io (#41) The macOS leg additionally: - runs `otool -l` to confirm the unbundled binary carries the reserved `__FLAPI/__bundle` segment from link time - runs `codesign --verify --strict out-segment` after the default reserved-segment pack -- the notarisation precondition this whole approach was built for - runs `flapi pack --macos-append --out out-append` and confirms `info` still discovers the bundle (ad-hoc legacy path still works, signature is intentionally invalid -- we do not run codesign verify here) - runs an oversized-payload pack (32 MiB into a 16-MiB segment) and confirms it exits non-zero with both "FLAPI_RESERVED_BUNDLE_MIB" and "reserved|exceeds" in the error message The Windows job uses pwsh; pack/info/unpack/sha256 logic mirrors Linux via PowerShell idioms. `create-release` now `needs` the four new jobs in addition to the existing build + smoke matrix, so a broken pack on any platform blocks the release. The existing `integration-tests` job (which already runs on ubuntu-24.04 amd64) auto-picks up `test_self_packaging.py` (#56) and `test_env_overrides.py` (#57) via pytest discovery -- no change needed there. Note: on macOS we deliberately do NOT assert reproducibility because the codesign step may include non-deterministic data; on Linux and Windows the host-bytes + appended-archive layout is reproducible by construction.
jrosskopf
added a commit
that referenced
this pull request
May 22, 2026
* ci: cross-platform self-packaging smoke jobs Part of #40. Closes #49. Adds four new CI jobs that exercise the self-packaging surface on every platform flapi builds for: pack-smoke-linux-amd64 ubuntu-24.04 pack-smoke-linux-arm64 ubuntu-24.04-arm pack-smoke-macos macos-latest pack-smoke-windows windows-latest Each job: - downloads the platform's `flapi` artifact from the existing windows-build / linux-build / osx-universal-build jobs - builds a tiny fixture tree (flapi.yaml + one endpoint + sample SQL) - runs `flapi pack --in fixture --out out-a` - runs `out-a info`, asserts the entry list contains the fixture files - runs `out-a unpack --to extracted`, diffs files byte-for-byte - runs a second `flapi pack ... --out out-b` with the same `SOURCE_DATE_EPOCH=1700000000` and asserts `sha256(out-a) == sha256(out-b)` -- the reproducible-build invariant baked into archive_io (#41) The macOS leg additionally: - runs `otool -l` to confirm the unbundled binary carries the reserved `__FLAPI/__bundle` segment from link time - runs `codesign --verify --strict out-segment` after the default reserved-segment pack -- the notarisation precondition this whole approach was built for - runs `flapi pack --macos-append --out out-append` and confirms `info` still discovers the bundle (ad-hoc legacy path still works, signature is intentionally invalid -- we do not run codesign verify here) - runs an oversized-payload pack (32 MiB into a 16-MiB segment) and confirms it exits non-zero with both "FLAPI_RESERVED_BUNDLE_MIB" and "reserved|exceeds" in the error message The Windows job uses pwsh; pack/info/unpack/sha256 logic mirrors Linux via PowerShell idioms. `create-release` now `needs` the four new jobs in addition to the existing build + smoke matrix, so a broken pack on any platform blocks the release. The existing `integration-tests` job (which already runs on ubuntu-24.04 amd64) auto-picks up `test_self_packaging.py` (#56) and `test_env_overrides.py` (#57) via pytest discovery -- no change needed there. Note: on macOS we deliberately do NOT assert reproducibility because the codesign step may include non-deterministic data; on Linux and Windows the host-bytes + appended-archive layout is reproducible by construction. * fix: LocateBundle probes the macOS reserved segment too Previously LocateBundle(path) only did the EOF tail scan. Only LocateBundleInSelf() probed the __FLAPI/__bundle Mach-O section first. That asymmetry meant flapi info / flapi unpack -- both of which call LocateBundle directly against a given binary -- couldn't find bundles that flapi pack wrote into the reserved segment on macOS. Caught by CI: pack-smoke-macos reported Packed 3 entries (683 bytes) into out-segment Bundle: none (filesystem mode) The pack succeeded (3 entries went into the section), but info checked only the EOF tail and saw nothing. Fix: move the section-probe into LocateBundle(path), where every caller benefits. LocateBundleInSelf becomes a thin wrapper. The fall-through to EOF tail is preserved so --macos-append bundles (legacy ad-hoc layout) remain discoverable. Found by CI on PR #59 (#49 cross-platform smoke jobs). * fix(pack): tolerate codesign failure on --macos-append + correct pwsh array-match Two follow-up fixes surfaced by CI on PR #59: 1. macOS: `flapi pack --macos-append` always tried to codesign the output, which fails with "main executable failed strict validation" because the appended ZIP after __LINKEDIT puts the binary outside what codesign considers signable. That's the documented trade-off -- append-mode output is explicitly not notarisable. Now we warn and continue; only the reserved-segment path treats codesign failure as fatal. 2. Windows pack-smoke: `$info -notmatch "x"` on a PowerShell array returns the filtered subset, not a boolean -- so the negation in the if statement was always truthy. Pack and info both worked correctly (info actually listed flapi.yaml, sqls/...); the test logic was the bug. Join the captured lines into a scalar string before -notmatch. Re-running the macOS leg should now show pack-smoke-macos passing through the --macos-append step. Re-running the Windows leg should get all the way to the reproducibility check. Found by CI on PR #59 (#49 cross-platform smoke jobs). * fix(pack): missing <iostream> for std::cerr warning on --macos-append CI on macos-latest failed to build src/pack.cpp: error: no member named 'cerr' in namespace 'std' Caused by the previous fix that warns instead of throws on the --macos-append codesign-failure path. <ostream> doesn't drag std::cerr in; we need <iostream> explicitly. * ci: rerun (flake retry on integration-tests' sql-injection corpus)
This was referenced May 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of epic #40. Stacked on #55 -> #54 -> #53 -> #52 -> #51.
Summary
End-to-end integration tests against the real
flapibinary,covering the spike's documented round-trip behaviours. Plus an
implementation fix surfaced by the test for the secret deny list.
test/integration/test_self_packaging.py-- 8 tests,pytest, runs in ~24 s on Linux x86_64 debug build.
src/pack.cpp-- validate input tree before copying themulti-GB host binary, so a rejected
.envdoesn't leave ahalf-written bundle on disk.
Coverage map vs. the spike's 9 behaviours
flapi packproduces a bundled binarytest_pack_produces_bundled_executableflapi inforeports EOCD offset + entriestest_info_reports_bundle_contentstest_self_hosting_rebundletest_repack_idempotence(3 rounds)test_truncated_bundle_falls_back_to_filesystemflapi unpackdumps the archivetest_unpack_restores_all_entriesread_csv('embed://...')in an endpoint templatePlus two non-spike invariants:
test_pack_refuses_secret_files_by_default--.envin inputcauses non-zero exit AND no half-written output binary.
test_pack_with_allow_secrets_bundles_env_files----allow-secretsbypasses the deny list.
Why three behaviours are deferred
Behaviours #1, #3, #9 all need server-lifecycle test infrastructure
(subprocess start, health-wait, hit HTTP endpoints, teardown). #3 and
#9 additionally need the bundled binary to load its embedded config
when run from a clean cwd -- which depends on
ConfigManager'stemplate-path resolution behaving sensibly when the loaded config
came from
embed://flapi.yaml. That's a non-trivial integrationquestion worth its own scope.
The DuckDB
read_csv('embed://...')proof-of-life already runsend-to-end in #54's unit tests against the in-process DuckDB
instance, so #9 is verified at the layer that matters most.
Filesystem-mode HTTP is exercised by every existing tavern test.
I'll file the server-lifecycle integration tests as a follow-up
issue once #51-#55 land.
Implementation fix in src/pack.cpp
Before this PR,
Pack()did:Re-ordered to:
Now
.envrejection is fail-fast, no garbage left behind. Thetest_pack_refuses_secret_files_by_defaulttest enforces it.Test plan
Pack() ordering, which is exercised by
test_pack_idempotenceand
test_pack_refuses_secret_files_by_default).Closes #46 (partial -- server-lifecycle behaviours deferred to a
follow-up).