Skip to content

feat(self-packaging #45): flapi pack / info / unpack subcommands#55

Merged
jrosskopf merged 1 commit into
mainfrom
feature/gh-45-pack-info-unpack
May 22, 2026
Merged

feat(self-packaging #45): flapi pack / info / unpack subcommands#55
jrosskopf merged 1 commit into
mainfrom
feature/gh-45-pack-info-unpack

Conversation

@jrosskopf
Copy link
Copy Markdown
Contributor

Part of epic #40. Stacked on #54 which is stacked on #53 -> #52 -> #51.

Summary

Closes the loop on self-packaging. After #41-#44 the runtime can
serve a bundled config tree; this PR adds the producer so the
same binary that serves bundles also creates them. The same artifact
is server + packager.

  • New library: src/{include/,}pack.cpp
  • CLI: argparse subparsers for pack / info / unpack in
    src/main.cpp. Zero-subcommand invocation
    (./flapi -c flapi.yaml) still runs the server
    , so existing
    operators see no change.

Library API

PackResult Pack(in_dir, out_path, PackOptions{});
int        PrintBundleInfo(ostream&);
int        PrintBundleInfoForPath(binary, ostream&);
UnpackResult UnpackBundle(dst_dir);
UnpackResult UnpackBundleFromPath(binary, dst_dir);
bool       IsSecretExcluded(rel_path);

Pack() flow:

  1. Copy host bytes from options.host_binary_override (defaults to
    GetSelfPath()). If the host already has a trailing ZIP, only the
    pre-bundle prefix is copied -- re-pack is idempotent.
  2. Recursive walk of in_dir. Files matching the default secret
    deny list (*.env at any depth, secrets/ segment at any
    depth, *.pem, *.key) cause PackError unless
    options.allow_secrets is set.
  3. archive_io::WriteArchive (feat(self-packaging #41): libarchive + archive_io RAII wrapper #51) with SOURCE_DATE_EPOCH stamping
    and sorted entries; appended to the output.
  4. Best-effort exec-bit copy from host -> output (Unix only).

CLI

$ ./flapi --help
Subcommands:
  info     Print bundle info (offset, size, entries) for this binary.
  pack     Package a flapi config tree into a self-contained executable.
  unpack   Dump the bundle of this binary into a directory.

$ ./flapi pack --in examples --out /tmp/flapi-prod --allow-secrets
Packed 17 entries (12534 bytes) into /tmp/flapi-prod

$ /tmp/flapi-prod info
Binary: /tmp/flapi-prod
Bundle offset: 1751413104
Bundle size:   12534 bytes
Entries (17):
  flapi.yaml (1024 bytes)
  sqls/customers.yaml (412 bytes)
  ...

$ /tmp/flapi-prod unpack --to /tmp/unpacked
Unpacked 17 entries to /tmp/unpacked

Subcommand handlers return early; the server-mode code path is
only reached when no subcommand was given.

The host_binary_override testability seam

The sanitiser-instrumented test binary is ~1.7 GB. Copying it for
every test case would push runtime past several minutes per case.
PackOptions::host_binary_override lets tests substitute a 4-KiB
fake host (with the EOCD signature scrubbed so the locator can't
mistake it for an existing bundle). Production code never touches
this knob.

Tests (10 cases / 38 assertions)

# Test What it asserts
1 IsSecretExcluded default deny list + non-matches (envoy.conf, foo.pemmed)
2 Pack produces a bundled binary entry count, output exists, EOCD locatable
3 Pack round-trip extract bundle slice -> ReadArchive -> entries match input
4 secret rejection .env in input -> PackError; output not written
5 --allow-secrets bypasses the deny list, bundles 5 entries
6 re-pack idempotence pack -> pack -> pack with stacked outputs, file size stable
7 host prefix preservation bytes 0..host_size byte-for-byte match the host file
8 PrintBundleInfo no bundle exit 1, output contains "none"
9 PrintBundleInfoForPath lists flapi.yaml, sqls/customers.sql, data/cities.csv
10 UnpackBundleFromPath 4 entries restored to disk, bytes equal
Filters: [pack]
All tests passed (38 assertions in 10 test cases)

Smoke test (real flapi binary, not test binary)

  • flapi --help shows three subcommands.
  • flapi info on unbundled binary -> "Bundle: none (filesystem mode)", exit 1.
  • flapi pack --in <dir> with .env in input -> single-line stderr error, exit 1.
  • flapi pack ... --allow-secrets -> Packed N entries (M bytes), exit 0.
  • <bundled> info -> offset (~1.6 GB into the host), size, entry list.
  • <bundled> unpack --to <dir> -> all files restored.

Test plan

  • 10 new pack tests green (38 assertions).
  • Smoke test against the actual flapi binary covers all three
    subcommands + happy + error paths.
  • Existing argparse usage unchanged when no subcommand is given;
    ./flapi -c flapi.yaml still launches server mode.
  • CI cross-platform once stack lands.

What's not in this PR (deferred)

Closes #45. Part of #40. Stacked on #54 -> #53 -> #52 -> #51.

Part of #40, depends on #41-#44. Closes the loop on self-packaging:
the same binary that serves bundled content now also produces it.

Library (src/{include/,}pack.cpp):
- Pack(in_dir, out_path, options) -> PackResult
    1. host bytes from `options.host_binary_override` (default
       `GetSelfPath()`) -- if the host already has a trailing ZIP,
       only the pre-bundle prefix is copied, so re-pack is idempotent
    2. recursive walk of `in_dir`; refuses files matching the default
       secret exclude list unless `options.allow_secrets` is set
    3. WriteArchive (#41) with SOURCE_DATE_EPOCH stamping and sorted
       entries, appended to the output
    4. best-effort exec-bit copy from host -> output (Unix only)
- PrintBundleInfo / PrintBundleInfoForPath -- EOCD offset, size,
  entry list of either GetSelfPath() or an explicit binary
- UnpackBundle / UnpackBundleFromPath -- restore entries to a dst dir
- IsSecretExcluded -- default deny list:
    *.env at any depth, secrets/ segment at any depth, *.pem, *.key

CLI (src/main.cpp):
- argparse subparsers for `pack`, `info`, `unpack`
- zero-subcommand invocation (`./flapi -c flapi.yaml`) still runs the
  server, preserving the contract for existing operators
- subcommand handlers return early -- never reach server startup
- pack reports `Packed N entries (M bytes) into <path>`
- unpack reports `Unpacked N entries to <path>`
- error paths emit a single line to stderr and exit 1

The host_binary_override knob (PackOptions) is the testability seam:
the sanitiser-instrumented test binary is ~1.7 GB and copying it for
every test case would push runtime past several minutes per test.
Tests inject a 4-KiB fake host with the EOCD signature scrubbed so
the locator can't mistake it for an existing bundle.

Tests (test/cpp/pack_test.cpp, 10 cases / 38 assertions):
- IsSecretExcluded: default deny list + non-matches
- Pack: entry count, output exists, EOCD locatable, bundle round-trip
- exclude enforcement: PackError on .env, --allow-secrets bypass
- idempotence: pack -> pack -> pack with stacked outputs, size stable
- host prefix preservation: byte-for-byte match against the fake host
- PrintBundleInfo: returns non-zero + "none" for unbundled
- PrintBundleInfoForPath: lists entries from a bundled binary
- UnpackBundleFromPath: 4 entries restored, byte-for-byte equal

Smoke test (real flapi binary, not test binary):
- `flapi --help` shows three subcommands
- `flapi info` on unbundled binary reports "none" with exit 1
- `flapi pack --in <dir>` with a .env in the input refuses with exit 1
- `flapi pack ... --allow-secrets` bundles, exit 0
- `<bundled> info` shows offset / size / entry list
- `<bundled> unpack --to <dir>` restores all files

Closes #45.
@jrosskopf jrosskopf force-pushed the feature/gh-45-pack-info-unpack branch from 9d7a4f2 to e49ce46 Compare May 22, 2026 14:25
@jrosskopf jrosskopf marked this pull request as ready for review May 22, 2026 15:19
@jrosskopf jrosskopf merged commit 5b9bc92 into main May 22, 2026
17 checks passed
@jrosskopf jrosskopf deleted the feature/gh-45-pack-info-unpack branch May 22, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Self-packaging #5: flapi pack / info / unpack subcommands

1 participant