Skip to content

test(self-packaging #46): integration tests for round-trip behaviours#56

Merged
jrosskopf merged 1 commit into
mainfrom
feature/gh-46-integration-roundtrip
May 22, 2026
Merged

test(self-packaging #46): integration tests for round-trip behaviours#56
jrosskopf merged 1 commit into
mainfrom
feature/gh-46-integration-roundtrip

Conversation

@jrosskopf
Copy link
Copy Markdown
Contributor

Part of epic #40. Stacked on #55 -> #54 -> #53 -> #52 -> #51.

Summary

End-to-end integration tests against the real flapi binary,
covering the spike's documented round-trip behaviours. Plus an
implementation fix surfaced by the test for the secret deny list.

  • New file: test/integration/test_self_packaging.py -- 8 tests,
    pytest, runs in ~24 s on Linux x86_64 debug build.
  • Fix to src/pack.cpp -- validate input tree before copying the
    multi-GB host binary, so a rejected .env doesn't leave a
    half-written bundle on disk.

Coverage map vs. the spike's 9 behaviours

Spike # Behaviour This PR
1 filesystem mode serves endpoints deferred -- existing tavern suite already covers filesystem-mode HTTP
2 flapi pack produces a bundled binary test_pack_produces_bundled_executable
3 bundled binary serves endpoints from clean cwd deferred -- needs bundled-mode config-path resolution work
4 flapi info reports EOCD offset + entries test_info_reports_bundle_contents
5 self-hosting (bundled binary packs new bundled binary) test_self_hosting_rebundle
6 re-pack idempotence test_repack_idempotence (3 rounds)
7 1-KiB-truncated bundle falls back to filesystem mode test_truncated_bundle_falls_back_to_filesystem
8 flapi unpack dumps the archive test_unpack_restores_all_entries
9 read_csv('embed://...') in an endpoint template deferred -- in-process proof-of-life already in #54's unit tests

Plus two non-spike invariants:

  • test_pack_refuses_secret_files_by_default -- .env in input
    causes non-zero exit AND no half-written output binary.
  • test_pack_with_allow_secrets_bundles_env_files -- --allow-secrets
    bypasses the deny list.
$ FLAPI_BUILD_TYPE=debug uv run pytest test_self_packaging.py -v
============================== 8 passed in 24.38s ==============================

Why three behaviours are deferred

Behaviours #1, #3, #9 all need server-lifecycle test infrastructure
(subprocess start, health-wait, hit HTTP endpoints, teardown). #3 and
#9 additionally need the bundled binary to load its embedded config
when run from a clean cwd -- which depends on ConfigManager's
template-path resolution behaving sensibly when the loaded config
came from embed://flapi.yaml. That's a non-trivial integration
question worth its own scope.

The DuckDB read_csv('embed://...') proof-of-life already runs
end-to-end in #54's unit tests against the in-process DuckDB
instance, so #9 is verified at the layer that matters most.
Filesystem-mode HTTP is exercised by every existing tavern test.

I'll file the server-lifecycle integration tests as a follow-up
issue once #51-#55 land.

Implementation fix in src/pack.cpp

Before this PR, Pack() did:

1. copy 1.7 GB host bytes to <out>
2. walk input dir
3. throw PackError on .env
4. leave a 1.7 GB half-written file in the user's tmp dir

Re-ordered to:

1. walk input dir (validates against deny list)
2. write archive in memory
3. copy host bytes
4. append

Now .env rejection is fail-fast, no garbage left behind. The
test_pack_refuses_secret_files_by_default test enforces it.

Test plan

Closes #46 (partial -- server-lifecycle behaviours deferred to a
follow-up).

Part of #40. Implements integration tests for the spike's documented
round-trip behaviours, plus a fix surfaced by the tests.

New file: test/integration/test_self_packaging.py (8 tests, ~24 sec
wall on Linux x86_64 debug build):

  #2 pack_produces_bundled_executable
       `flapi pack --in <fixture> --out <new>` succeeds, output exists,
       exec bits preserved (Unix), file size >= host binary size.

  #4 info_reports_bundle_contents
       `<bundled> info` shows EOCD offset, bundle size, and lists every
       fixture entry by name.

  #5 self_hosting_rebundle
       The bundled binary can pack a new bundled binary
       (`<bundled> pack ...`), which itself has a valid bundle.

  #6 repack_idempotence
       Three rounds of packing using each previous output as host
       binary -- file size identical across rounds.

  #7 truncated_bundle_falls_back_to_filesystem
       Chop 1 KiB off the tail. `info` reports "none". A
       `--validate-config -c <fixture>/flapi.yaml` invocation exits
       cleanly (no SIGSEGV/SIGABRT) -- the spike safety-net behaviour.

  #8 unpack_restores_all_entries
       `<bundled> unpack --to <dir>` writes every entry; CSV bytes
       round-trip byte-for-byte.

  pack_refuses_secret_files_by_default
       `.env` in input -> non-zero exit, "secret" in error message,
       AND no half-written output binary on disk.

  pack_with_allow_secrets_bundles_env_files
       --allow-secrets override bundles `.env` despite the deny list;
       `info` lists it.

Implementation fix (src/pack.cpp): `Pack()` previously copied the
multi-GB host binary BEFORE walking the input tree and validating the
secret deny list. A rejected `.env` therefore left a half-written
1.7 GB file in the user's tmp dir. Re-ordered: walk + validate first,
then copy + append. Surfaced by `test_pack_refuses_secret_files_by_default`.

Spike behaviours #1, #3, #9 (filesystem-mode + bundled-mode HTTP +
read_csv('embed://...') endpoint) require server-lifecycle test
infrastructure plus resolving bundled-mode config-path lookup
(template.path resolution from inside an `embed://` config) -- deferred
to a follow-up issue. The DuckDB-side embed:// integration is already
proven in-process by the unit tests in #44.

Closes #46 (partial -- server-lifecycle behaviours deferred).
@jrosskopf jrosskopf force-pushed the feature/gh-46-integration-roundtrip branch from c018420 to d8f5ca1 Compare May 22, 2026 15:20
@jrosskopf jrosskopf marked this pull request as ready for review May 22, 2026 16:10
@jrosskopf jrosskopf merged commit e0d1966 into main May 22, 2026
17 checks passed
@jrosskopf jrosskopf deleted the feature/gh-46-integration-roundtrip branch May 22, 2026 16:10
jrosskopf added a commit that referenced this pull request May 22, 2026
Part of #40. Closes #49.

Adds four new CI jobs that exercise the self-packaging surface on
every platform flapi builds for:

  pack-smoke-linux-amd64   ubuntu-24.04
  pack-smoke-linux-arm64   ubuntu-24.04-arm
  pack-smoke-macos         macos-latest
  pack-smoke-windows       windows-latest

Each job:
- downloads the platform's `flapi` artifact from the existing
  windows-build / linux-build / osx-universal-build jobs
- builds a tiny fixture tree (flapi.yaml + one endpoint + sample SQL)
- runs `flapi pack --in fixture --out out-a`
- runs `out-a info`, asserts the entry list contains the fixture files
- runs `out-a unpack --to extracted`, diffs files byte-for-byte
- runs a second `flapi pack ... --out out-b` with the same
  `SOURCE_DATE_EPOCH=1700000000` and asserts
  `sha256(out-a) == sha256(out-b)` -- the reproducible-build invariant
  baked into archive_io (#41)

The macOS leg additionally:
- runs `otool -l` to confirm the unbundled binary carries the
  reserved `__FLAPI/__bundle` segment from link time
- runs `codesign --verify --strict out-segment` after the default
  reserved-segment pack -- the notarisation precondition this whole
  approach was built for
- runs `flapi pack --macos-append --out out-append` and confirms
  `info` still discovers the bundle (ad-hoc legacy path still works,
  signature is intentionally invalid -- we do not run codesign verify
  here)
- runs an oversized-payload pack (32 MiB into a 16-MiB segment) and
  confirms it exits non-zero with both "FLAPI_RESERVED_BUNDLE_MIB"
  and "reserved|exceeds" in the error message

The Windows job uses pwsh; pack/info/unpack/sha256 logic mirrors
Linux via PowerShell idioms.

`create-release` now `needs` the four new jobs in addition to the
existing build + smoke matrix, so a broken pack on any platform
blocks the release.

The existing `integration-tests` job (which already runs on
ubuntu-24.04 amd64) auto-picks up `test_self_packaging.py` (#56) and
`test_env_overrides.py` (#57) via pytest discovery -- no change
needed there.

Note: on macOS we deliberately do NOT assert reproducibility because
the codesign step may include non-deterministic data; on Linux and
Windows the host-bytes + appended-archive layout is reproducible by
construction.
jrosskopf added a commit that referenced this pull request May 22, 2026
* ci: cross-platform self-packaging smoke jobs

Part of #40. Closes #49.

Adds four new CI jobs that exercise the self-packaging surface on
every platform flapi builds for:

  pack-smoke-linux-amd64   ubuntu-24.04
  pack-smoke-linux-arm64   ubuntu-24.04-arm
  pack-smoke-macos         macos-latest
  pack-smoke-windows       windows-latest

Each job:
- downloads the platform's `flapi` artifact from the existing
  windows-build / linux-build / osx-universal-build jobs
- builds a tiny fixture tree (flapi.yaml + one endpoint + sample SQL)
- runs `flapi pack --in fixture --out out-a`
- runs `out-a info`, asserts the entry list contains the fixture files
- runs `out-a unpack --to extracted`, diffs files byte-for-byte
- runs a second `flapi pack ... --out out-b` with the same
  `SOURCE_DATE_EPOCH=1700000000` and asserts
  `sha256(out-a) == sha256(out-b)` -- the reproducible-build invariant
  baked into archive_io (#41)

The macOS leg additionally:
- runs `otool -l` to confirm the unbundled binary carries the
  reserved `__FLAPI/__bundle` segment from link time
- runs `codesign --verify --strict out-segment` after the default
  reserved-segment pack -- the notarisation precondition this whole
  approach was built for
- runs `flapi pack --macos-append --out out-append` and confirms
  `info` still discovers the bundle (ad-hoc legacy path still works,
  signature is intentionally invalid -- we do not run codesign verify
  here)
- runs an oversized-payload pack (32 MiB into a 16-MiB segment) and
  confirms it exits non-zero with both "FLAPI_RESERVED_BUNDLE_MIB"
  and "reserved|exceeds" in the error message

The Windows job uses pwsh; pack/info/unpack/sha256 logic mirrors
Linux via PowerShell idioms.

`create-release` now `needs` the four new jobs in addition to the
existing build + smoke matrix, so a broken pack on any platform
blocks the release.

The existing `integration-tests` job (which already runs on
ubuntu-24.04 amd64) auto-picks up `test_self_packaging.py` (#56) and
`test_env_overrides.py` (#57) via pytest discovery -- no change
needed there.

Note: on macOS we deliberately do NOT assert reproducibility because
the codesign step may include non-deterministic data; on Linux and
Windows the host-bytes + appended-archive layout is reproducible by
construction.

* fix: LocateBundle probes the macOS reserved segment too

Previously LocateBundle(path) only did the EOF tail scan. Only
LocateBundleInSelf() probed the __FLAPI/__bundle Mach-O section
first. That asymmetry meant flapi info / flapi unpack -- both of
which call LocateBundle directly against a given binary -- couldn't
find bundles that flapi pack wrote into the reserved segment on
macOS.

Caught by CI: pack-smoke-macos reported

  Packed 3 entries (683 bytes) into out-segment
  Bundle: none (filesystem mode)

The pack succeeded (3 entries went into the section), but info
checked only the EOF tail and saw nothing.

Fix: move the section-probe into LocateBundle(path), where every
caller benefits. LocateBundleInSelf becomes a thin wrapper. The
fall-through to EOF tail is preserved so --macos-append bundles
(legacy ad-hoc layout) remain discoverable.

Found by CI on PR #59 (#49 cross-platform smoke jobs).

* fix(pack): tolerate codesign failure on --macos-append + correct pwsh array-match

Two follow-up fixes surfaced by CI on PR #59:

1. macOS: `flapi pack --macos-append` always tried to codesign the
   output, which fails with "main executable failed strict
   validation" because the appended ZIP after __LINKEDIT puts the
   binary outside what codesign considers signable. That's the
   documented trade-off -- append-mode output is explicitly not
   notarisable. Now we warn and continue; only the
   reserved-segment path treats codesign failure as fatal.

2. Windows pack-smoke: `$info -notmatch "x"` on a PowerShell
   array returns the filtered subset, not a boolean -- so the
   negation in the if statement was always truthy. Pack and info
   both worked correctly (info actually listed flapi.yaml, sqls/...);
   the test logic was the bug. Join the captured lines into a scalar
   string before -notmatch.

Re-running the macOS leg should now show pack-smoke-macos passing
through the --macos-append step. Re-running the Windows leg should
get all the way to the reproducibility check.

Found by CI on PR #59 (#49 cross-platform smoke jobs).

* fix(pack): missing <iostream> for std::cerr warning on --macos-append

CI on macos-latest failed to build src/pack.cpp:
  error: no member named 'cerr' in namespace 'std'

Caused by the previous fix that warns instead of throws on the
--macos-append codesign-failure path. <ostream> doesn't drag
std::cerr in; we need <iostream> explicitly.

* ci: rerun (flake retry on integration-tests' sql-injection corpus)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-packaging #6: Integration tests — round-trip parity (spike behaviours 1-9)

1 participant