ts-sdk: minimal Node.js TypeScript SDK for Apache Airflow (alpha)#66289
Draft
shivaam wants to merge 2 commits into
Draft
ts-sdk: minimal Node.js TypeScript SDK for Apache Airflow (alpha)#66289shivaam wants to merge 2 commits into
shivaam wants to merge 2 commits into
Conversation
Introduces ts-sdk/, a Node.js / TypeScript Task SDK that lets users author and run Airflow tasks as TypeScript handlers in a Node worker process, dispatched via the Edge Executor (AIP-69) through the Task Execution API (AIP-72). Workers are standalone Node processes that talk to Airflow over HTTPS only — no DB access, no in-cluster Python required. Marked alpha (`0.1.0-alpha.0`) — a foundation for follow-up PRs. Includes: - `registerTask` + `startWorker` public API (src/index.ts). - Edge API client: register, heartbeat, fetch job, ack. HS512 JWT signing per request with configurable issuer (defaults to "airflow" to match Airflow's `[api_auth] jwt_issuer` default). - Execution API client with two-token lifecycle: the `Refreshed-API-Token` response header swaps the workload-scoped token from `command.token` for an execution-scoped token on `/run`, and the same mechanism refreshes execution tokens near expiry. Tracks Airflow PR apache#60108 (merged 2026-04-23). - Worker poll loop with graceful SIGTERM drain — the worker waits for the in-flight task to complete naturally rather than killing it, making it K8s and Lambda safe. - Heartbeat circuit breaker that aborts after consecutive failures. - Failsafe ack: every fetched edge job is ack'd even if the execute path crashes, so edge state always converges. - 71 vitest unit tests across worker, edge-client, execution-client, registry, errors, jwt, retry. Includes coverage for the `Refreshed-API-Token` swap (`tests/execution-client.test.ts`). - 4 reference DAG fixtures + 4 reference workers under `integration/`, plus a pure-JavaScript consumer canary (`integration/workers/js-consumer.mjs`) that imports the built `dist/` directly with stock node — proves the package works for consumers who don't write TypeScript. - TESTING.md: end-to-end recipe runnable from a single host terminal via `docker exec`, with separate "any Airflow" and "set up a local Airflow with breeze" sections. - LICENSE and NOTICE shipped in the npm tarball (matches go-sdk's precedent), enforced via `package.json`'s `files` allowlist. - `packageManager: pnpm@10.28.1` declared so corepack auto-installs the right pnpm. Build: - Plain `tsc` build (no bundler) — pure library, no tree-shaking needed. ESM-only, Node 22+ only. Strict tsconfig with `noUncheckedIndexedAccess`, `verbatimModuleSyntax`, `moduleDetection: force`, `NodeNext` module resolution. - `src/generated/edge.ts` and `src/generated/execution.ts` are produced by `openapi-typescript` from the YAML/JSON specs in `airflow-core/docs/_api/` (committed to keep the diff reviewable and `pnpm install` reproducible — matches `ui/openapi-gen/` precedent). Excluded from the prek `insert-license` hook for the same reason. Intentionally deferred (TODO markers in src/worker.ts): - Per-TI heartbeat - XCom / Variables / Connections - Log forwarding to Airflow UI - Subprocess isolation per task - Concurrency > 1 Manually verified end-to-end against `breeze start-airflow --executor edgeexecutor` (Airflow 3.3-dev / edge3 3.5-dev): happy path, handler-throws → failed, no-handler-registered → failed, SIGTERM-during-task → drained-success. Pure-JS canary also verified.
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-2-test
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Two regexes flagged by GitHub Advanced Security as potentially polynomial
on adversarial input. Real exposure is bounded (base64 padding is at
most 2 chars, AIRFLOW__EDGE__API_URL is operator-controlled config),
but the safer alternatives are also simpler:
- src/utils/jwt.ts: replace the manual base64 → base64url conversion
(`/=+\$/`, `/+/g`, `/\//g`) with Node's built-in `.toString("base64url")`.
- src/worker-options.ts: replace the `/\/edge_worker\/v1.*\$/` prefix
strip with indexOf/slice. No regex backtracking either way.
Behavior preserved. 71/71 unit tests still pass; verified end-to-end
against breeze (happy path / failure / missing-handler all green).
Member
|
If you want to discuss adding new SDK - it needs definitely (at the very least) devlist discussion and most likely new AIP. |
Contributor
Author
Thanks Jarek. The devlist thread is here: https://www.mail-archive.com/dev@airflow.apache.org/msg22351.html On the AIP, I think we just need a dev list discussion as TS SDK is implementing the same interfaces that go & java sdk use. I have also discussed this via slack on @ashb but we can discuss further on the dev mailing list. Happy to write one if needed. |
Member
|
Ah cool. Catchig up :) .. Still good to keep it as Draft pr until the discussion concludes :). I might chime-in there soon. |
Contributor
Author
Sounds good |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new Node.js / TypeScript Task SDK at
ts-sdk/for running Airflow tasksas TypeScript handlers. Workers run as standalone Node processes and connect to
Airflow over HTTPS via the Edge Executor (AIP-69) and Task Execution API
(AIP-72) — no DB access, no in-cluster Python required.
Marked alpha (
0.1.0-alpha.0) — a foundation for future PRs to build on.What's in
registerTask+startWorkerpublic APIRefreshed-API-Tokenheader — tracks Airflow PR Two-token mechanism for task execution to prevent token expiration while tasks wait in executor queues #60108)Intentionally deferred (TODOs in
src/worker.ts:21-24)Testing
Unit:
pnpm test— 71/71 pass.End-to-end against
breeze start-airflow --executor edgeexecutor. Full recipe ints-sdk/TESTING.md.
Scenarios verified:
core_integrationcore_failurecore_missing_handlercore_sigterm_inflightPure-JS consumer canary (
integration/workers/js-consumer.mjs) also verified —proves the SDK works for users who don't write TypeScript.
Notes for reviewers
ts-sdk/) isdeferred to a follow-up. The only CI hook needed for this PR is the prek
insert-licenseexclude forts-sdk/src/generated/, which is in the diff.src/generated/edge.ts,execution.ts) are produced byopenapi-typescriptfrom the YAML/JSON specs inairflow-core/docs/_api/.Committed (not generated at install time) so the diff is reviewable —
matches
ui/openapi-gen/precedent.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines