lakeflow-sensor

The dependency-evaluation companion to lakeflow-controller. It watches LakeFlow custom resources (lakeflow.io/v1alpha1) across the cluster and, when a downstream's declared upstreams have all completed, creates the downstream's Argo Workflow from the template the controller rendered.

lakeflow-controller deliberately stops at rendering the WorkflowTemplate for a Dependency-triggered LakeFlow — it does not decide when that workflow should run. lakeflow-sensor owns that decision. It is read-only toward LakeFlow resources: it reads LakeFlow.status, evaluates dependencies, and creates Argo Workflows — nothing else.

What problem it solves

Triggering a workflow once its upstreams finish sounds like a job for Argo Events: an EventSource watches resources and a Sensor fires a trigger. That works for stateless, fire-on-event automation. It does not work for data-pipeline dependency triggering, because a Sensor is event-driven and therefore forgetful:

• Task failure is the norm, not the exception. Upstreams fail, get retried, get resubmitted for backfill. An event stream of "X completed" has no notion of an execution cycle, so it can't express "fire downstream only once every upstream has a completion belonging to the same logical run."
• It can't reason about re-execution. Retry a single upstream and an event-driven Sensor sees a fresh completion and fires the downstream prematurely — mixing the just-retried upstream with stale completions from the others.
• It's operationally opaque. When someone asks "why didn't daily-report trigger last night?" there is no authoritative answer to read back — only logs to reconstruct.

lakeflow-sensor is the opposite design point: status-driven, stateful, and explicitly debuggable. It evaluates cycles (the latest completion of each upstream, released like a CountDownLatch only when all are present), groups upstreams that share a dependency so a partial retry can't fire downstream on an inconsistent mix of runs, and records every decision in MySQL so each trigger — and each non-trigger — has a queryable reason.

For the cycle algorithm, dependency-group consistency, manual-trigger policies, and label propagation, see the design docs.

Dependency tree

Dependencies form a tree, not a flat list, and the tree decides what happens when a downstream's upstreams are not all of the same trigger type (scheduled vs. retry/resubmit). Upstreams that share a grandparent are treated as one group; within a group the trigger types must be consistent, while different groups stay independent. So a partial retry of one subtree is allowed to fire, but mixing a freshly-retried upstream with stale, scheduled siblings in the same group is rejected — which is what stops a single re-execution from triggering the downstream on an inconsistent set of runs.

Automatic namespace discovery

The sensor discovers namespaces to watch automatically — there is no static list to maintain. Any namespace carrying the label lakeflow.io/created-by=lakeflow-controller (configurable via kubernetes.namespaceLabelSelector) is picked up, and ownership is spread across pods by consistent hashing so the watch set scales horizontally and rebalances as replicas come and go. On taking ownership a pod re-syncs upstream completions and re-enqueues any previously-blocked downstream, so nothing is missed across restarts.

Prerequisites

• A Kubernetes cluster with Argo Workflows installed
• lakeflow-controller deployed (provides the CRD and templates)
• A MySQL database for durable state
• kubectl, and for building, Go 1.24+ and Docker

This repo depends on github.com/pzhenzhou/lakeflow-controller for the LakeFlow API types via a replace directive pointing at ../lakeflow-controller, so clone both repos side by side.

Build and run

make build                 # build to bin/lakeflow-sensor-srv (cmd/watcher/main.go)
make test                  # build, then run tests with coverage
make run-dev               # run locally with pprof + metrics enabled

# Docker / deploy (OVERLAY=local|test|staging|prod)
make docker-build IMG=${IMG_NAME}
make build-deploy-test

Debuggability

Every decision is recorded, so "why did/didn't it trigger" is one call away (HTTP API under /api/v1):

curl http://lakeflow-sensor:8080/api/v1/explain/prod/daily-report   # full evaluation + block reasons
curl http://lakeflow-sensor:8080/api/v1/verify/prod/daily-report    # end-to-end verification

License

Copyright 2025.

Licensed under the Apache License, Version 2.0. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0.