Go SDK: airflow-go-pack tool for self-contained bundle binaries#67156
Draft
jason810496 wants to merge 5 commits into
Draft
Go SDK: airflow-go-pack tool for self-contained bundle binaries#67156jason810496 wants to merge 5 commits into
jason810496 wants to merge 5 commits into
Conversation
- Add `messages_test.go` to test message decoding and encoding functionalities. - Introduce `serde.go` for serialization of various data types to Airflow's format. - Create `serde_test.go` to validate serialization logic and ensure correctness. - Implement `server.go` to handle communication with the supervisor and manage task execution. - Add `task_runner.go` to execute tasks based on received startup details and handle success/failure.
…proved configuration flexibility
- Added new command `airflow-go-pack` to build a self-contained Airflow bundle from a Go package. - Introduced `inspect` command to print the embedded manifest and optionally the source from a bundle. - Implemented `dump-bundle-spec` functionality to output the bundle specification in JSON format. - Created `bundlefooter` package to manage appending source and metadata to the binary with a defined trailer. - Added tests for bundle footer operations and manifest rendering to ensure correctness. - Updated Justfile for building and packing example DAG bundles.
…fter "--" and update example bundle to support dynamic DAG naming
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ADR 0004 settled on a self-contained executable as the Go bundle
artefact: one binary that carries its own source and manifest in a
footer, runs under both the go-plugin and the coordinator protocol,
and exposes its bundle spec for introspection. This PR ships the
tool that produces that artefact (
airflow-go-pack) and the bundleserver hooks it needs.
What
cmd/airflow-go-pack/pack.godrives the build: it invokes theuser's Go toolchain to compile their bundle main, then appends the
source tree and manifest to the resulting binary via the new
internal/bundlefooterpackage.cmd/airflow-go-pack/inspect.goreads the footer back out — printthe embedded manifest, optionally dump the source — so operators can
verify what a binary contains without running it.
cmd/airflow-go-pack/main.gowires the subcommands and--dump-bundle-spec(the introspection contract from ADR 0001Option D); arg parsing treats everything after
--as build flagsto pass through to
go build.bundle/bundlev1/bundlev1server/server.golearns to detect aself-contained binary and serve its embedded source/manifest to the
scanner instead of expecting a ZIP container.
example/bundle/Justfileis updated to build viaairflow-go-pack,and
example/bundle/main.gonow reads its DAG id from an-ldflagsvariable so the test fixture can produce multiple bundles from one
source tree.
Was generative AI tooling used to co-author this PR?