Go SDK: Dag/Task specs and downstream wiring in the authoring API#67155
Draft
jason810496 wants to merge 3 commits into
Draft
Go SDK: Dag/Task specs and downstream wiring in the authoring API#67155jason810496 wants to merge 3 commits into
jason810496 wants to merge 3 commits into
Conversation
- Add `messages_test.go` to test message decoding and encoding functionalities. - Introduce `serde.go` for serialization of various data types to Airflow's format. - Create `serde_test.go` to validate serialization logic and ensure correctness. - Implement `server.go` to handle communication with the supervisor and manage task execution. - Add `task_runner.go` to execute tasks based on received startup details and handle success/failure.
…proved configuration flexibility
1 task
Draft
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On hold until "Defining Dag in foreign language" out.
Why
The Go authoring API today exposes
AddDag(name)andAddTask(fn)with no way to set the things a real DAG needs — schedule, tags,
queue, retries, or downstream relationships — so any non-trivial Go
DAG has to reach around the registry. This PR adds the missing
configuration surface and the downstream argument so users can author
multi-task DAGs in pure Go without escape hatches.
What
bundlev1/registry.go:AddDagnow takes an optionalDagSpec(schedule, description, tags, etc.) and
AddTasknow takes anoptional
TaskSpec(queue, retries, …) plus a downstream task-idlist. The registry materialises both into the serialised DAG so the
coordinator sees the same shape Python sees.
pkg/execution/serde.golearns to encode the new spec fields anddownstream edges into Airflow's task/DAG payload.
example/bundle/main.go) updates to the newsignatures so it doubles as a worked example of the API.
registry_test.go,serde_test.go,integration_test.go,and
worker/runner_test.gocover the new spec/downstream paths.Next
Was generative AI tooling used to co-author this PR?