Snakemake-based execution layer + content-addressed manifests + Dask substrate

[EiffL]

Status: Draft. Branch supersedes the Dagster + Postgres engine with a Snakemake-based one whose entire contribution is a content-addressed manifest layer plus a vendored Dask executor plugin. See redesign.md for the full spec and design_review.md for the requirements it answers.

TL;DR

Snakemake owns DAG resolution, parallelism, retries, and dry-run. Dask owns task dispatch, per-task resource matching, and bin-packing into worker capacity. We own one Python function (write_manifest), a Snakefile generator, the container build/hash/wrap module, an in-process cluster_for_run() bootstrap, and a vendored Snakemake executor plugin that hands rules to Dask. Net diff vs. main: −9.7k / +4.4k LOC while delivering more of design_review.md than the previous engine did.

Design principles guiding this iteration

1. Snakemake does what Snakemake does

DAG construction, topological execution, parallelism, retries, dry-run, profiles, locking — all delegated to upstream. We do not write code for any of it.

2. We own only the integrity layer

The single property design_review.md calls non-negotiable — agents cannot fake outputs — becomes a consequence of how the system is built, not a policy enforced by a process boundary:

• Every rule declares its .lightcone-manifest.json as an output. Snakemake re-runs any rule whose manifest is missing.
• The manifest is content-addressed: data_version = sha256(output_dir), code_version = sha256(recipe ‖ container_image ‖ decisions), input_versions chain to upstream manifests.
• lc verify recomputes hashes and validates the chain — no orchestrator dependency.
• lc status walks manifests offline. Snakemake's .snakemake/metadata/ is treated as a cache and never read.

3. User-facing surface does not change

lc run, lc status, lc verify keep their semantics. The Snakefile is implementation detail — generated on every lc run from astra.yaml, never edited by hand.

4. Clean-slate replacement, no dual-engine

No backwards-compatibility flag, no migration command, no shim. The Dagster / Postgres / runner / slurm-info / targets modules and their tests are deleted, not deprecated.

5. Containers are ours, end-to-end

We build images deterministically from Containerfiles with content-addressed tags (lc-<name>-<sha256>) and pre-wrap each recipe with the explicit runtime call at Snakefile-generation time. Snakemake's container: directive is intentionally unused, so the manifest's container_image field is the strong evidence of what actually ran.

6. One execution substrate everywhere — Dask

A user inside an salloc/sbatch who runs lc run gets routed through Dask automatically; same plugin and bootstrap path covers the workstation case and a pre-existing scheduler. lightcone.engine.dask_cluster.cluster_for_run returns a scheduler address for the run's lifetime:

Trigger	Behavior
DASK_SCHEDULER_ADDRESS set	Connect to the existing scheduler. We don't own its lifecycle.
SLURM_JOB_ID set	Start an in-process LocalCluster(n_workers=0) bound to the driver's hostname; srun --ntasks=\$NNODES --ntasks-per-node=1 dask worker \$ADDR launches one persistent worker per node, each advertising the node's full cpus/memory/gpus as Dask resources.
Neither	LocalCluster() sized to the local machine.

The scheduler is always in-process — its lifetime equals the run's lifetime. No service to manage, no orphaned schedulers if the driver crashes, no lc cluster lifecycle commands.

Snakemake dispatches each rule via our own executor plugin at src/snakemake_executor_plugin_dask/: client.submit(_run_shell, cmd, resources={cpus, memory, gpus}, pure=False). Per-rule cpus_per_task / mem_mb / gpus_per_task translate 1:1 to per-task Dask resources, and the scheduler bin-packs tasks into workers up to each worker's advertised budget.

Why Dask, not slurm-jobstep or slurm:

• --executor slurm is the wrong shape — head-node sbatch submission, with the Snakemake docs warning that running it inside an active SLURM job leads to unpredictable behavior. Our users want pilot-job semantics: one big salloc, many tasks dispatched within it.
• --executor slurm-jobstep is the right shape but its repo description states it is "meant for internal use by snakemake-executor-plugin-slurm" — using it standalone is an off-label code path the maintainers do not commit to keeping working. It also inherits known footguns (SLURM 22.05+'s --cpus-per-task non-inheritance, the long-standing one-core dispatch issue) that would land in our user docs.
• Dask is pip install everywhere, gives us persistent workers within a run (sub-second task dispatch), exposes a live dashboard, and the Dask scheduler's failure modes are well understood. The trade we accept is that resource matching is advisory rather than cgroup-enforced — for minutes-to-hours recipes whose mem_mb is declared in astra.yaml, this is acceptable, and SLURM's per-allocation memory cgroup is the backstop.
• Flux was evaluated and rejected: richer scheduling, but module load flux (or building flux-core from source) on every host is install friction we don't want outside Perlmutter.

If standalone slurm-jobstep becomes a maintained, user-facing executor, this is worth revisiting. Until then, owning ~350 LOC of well-scoped Dask plumbing is the better trade.

7. No service to manage

The old lc cluster start/attach/stop lifecycle is gone with no replacement. The Dask scheduler is in-process; SLURM workers are bounded by the user's salloc. No Postgres, no scheduler daemon, no daemon-per-project.

What's in the diff

Added

• redesign.md, design_review.md — spec + requirements
• src/lightcone/engine/snakefile.py — generator (Jinja → .lightcone/Snakefile + sidecar JSON)
• src/lightcone/engine/manifest.py — write_manifest(), sha256_dir(), code_version helpers
• src/lightcone/engine/verify.py — chain integrity check
• src/lightcone/engine/dask_cluster.py — cluster_for_run() context manager
• src/snakemake_executor_plugin_dask/ — vendored Snakemake executor plugin
• Manifest, Snakefile, status, verify, container, dask-plugin, dask-cluster test suites

Removed

• engine/assets.py, engine/runner.py, engine/slurm_info.py, engine/targets.py, engine/io_manager.py
• All Dagster, Postgres scaffolding and corresponding tests
• The `lc cluster` command surface

Kept (re-purposed)

• `dask` and `distributed` — no longer behind `dagster-dask`'s shim, now driven directly by our vendored Snakemake executor plugin and `cluster_for_run` bootstrap.

Test plan

• `uv run pytest` — 206 tests passing (+ 1 opt-in `slow` marker for the real `LocalCluster` smoke)
• `uv run ruff check src/ tests/`
• `uv run mypy` clean on touched modules (only pre-existing `astra.helpers` stub gap remains)
• End-to-end `lc run` on a sample project locally (`LocalCluster` path)
• End-to-end `lc run` inside a Perlmutter `salloc` (srun-workers path)
• `lc verify` against manifests produced by Dask-driven runs

Open questions / not in this PR

• Heterogeneous nodes — first version assumes all nodes in an allocation share the same `cpus_per_node` / `mem_per_node`. If a site needs heterogeneous shapes we'll need to launch workers per node-type.
• Memory enforcement is advisory — Dask's `resources={"memory": ...}` is a soft scheduling constraint, not a cgroup cap. SLURM's per-allocation memory cgroup is the backstop on HPC.
• Worker-per-node sizing — fixed at one worker per node with `--nthreads = cpus_per_node`. Could expose tunables if a workload wants more workers per node, but starting simple.
• External input hashing default — `(mtime, size)` for now; `--strict-inputs` for sha256 mode.
• slurm-jobstep revisit — if the upstream plugin gets a stable, user-facing standalone-executor contract, the case for swapping out the Dask layer becomes strong.

🤖 Generated with Claude Code

[github-actions]

✅ Eval Results

Metric	Value
Score	1.00
Build complete	✅
Cost	$1.39
Turns	61
Duration	534s
lightcone-cli	0.1.dev1+g9d7bd2242 (9d7bd224)
Results	Download

Graders

✅ spec_valid (1.00)
✅ all_materialized (1.00)

Full output

80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:06 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:06 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:07 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:07 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:07 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:07 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:08 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:08 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:08 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:08 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:09 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:25:10 httpx HTTP Request: POST https://proxy.app.daytona.io/toolbox/8d5f472b-4be1-4dcc-80ab-63ac65abb96d/files/bulk-upload "HTTP/1.1 200 OK"
10:34:06 lightcone.eval.sandbox Deleted sandbox for trial build-snae-0
  snae trial 0: score=1.00 complete

lightcone-cli: 0.1.dev1+g9d7bd2242 (HEAD 9d7bd224)

  Eval Results: Scores  
┏━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Task ┃     Score     ┃
┡━━━━━━╇━━━━━━━━━━━━━━━┩
│ snae │ 1.00 +/- 0.00 │
│      │ pass@k: 100%  │
└──────┴───────────────┘

   Eval Results: Cost &   
         Duration         
┏━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Task ┃ Cost / Duration ┃
┡━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ snae │      $1.39      │
│      │      534s       │
└──────┴─────────────────┘

Total: 1 trials, $1.39, 534s

Results saved to: eval-results/build-9d7bd224/results.json

[cailmdaley]

Two questions on the cluster execution section, since they seem to pull in different directions:

1. Headline vs. body. The opening philosophy and bottom line both frame this as "Snakemake + manifests, not Dagster + Dask + Postgres" — but the cluster section reintroduces Dask as the inner execution substrate via a custom snakemake-executor-plugin-dask. Worth making explicit up front that Dask stays (just in a much thinner role), since right now the doc reads as if it were deleted.

2. Is Dask actually necessary? Snakemake's own executor abstraction seems to cover the cases:

   • local for laptop/workstation (--cores N)
   • slurm-jobstep for inside an existing allocation — the pilot-job pattern, dispatches each rule as srun --jobid=$SLURM_JOB_ID …
   • slurm for per-rule sbatch

   Profiles pick between them; user surface stays uniform.

   For the multi-node-within-one-allocation case in particular: can't Snakemake handle this in non-Slurm mode directly (e.g. local with high --cores, or slurm-jobstep) rather than going through a Dask scheduler + workers? Adding a Dask orchestration layer underneath Snakemake feels like it's in tension with the doc's own opening principle ("Snakemake does what Snakemake does, we don't reimplement"). Curious what the Dask layer is doing that Snakemake's native executors can't.

[EiffL]

@cailmdailey, from Claude:

What I found

slurm-jobstep exists and does roughly what the reviewer described — but with significant qualifications:

1. Officially marked "internal use only." The repo description says verbatim: "meant for internal use by
   snakemake-executor-plugin-slurm." The blessed deployment is: main slurm plugin runs on a head node, submits each rule via sbatch, and the inner sbatch wrapper invokes slurm-jobstep to do the actual srun -n1 --cpu-bind=q …. The maintainers do not officially endorse using it as a standalone executor for the pilot-job pattern.
2. You can use it standalone — the code reads SLURM_JOB_ID and dispatches srun -n1 --cpu-bind=q [cmd], which is exactly the pilot-job behavior. People in the community do this. But you're off the supported path; nothing prevents the maintainers from breaking that use case.
3. The main slurm executor explicitly warns against running inside an allocation. Snakemake docs say: "Running Snakemake within an active SLURM job can lead to unpredictable behavior." The intended pattern is head-node submission, not in-allocation dispatch.
4. Known footgun: SLURM 22.05+ broke --cpus-per-task inheritance. srun no longer inherits cpus_per_task from the surrounding sbatch/salloc — it must be re-requested or set via SRUN_CPUS_PER_TASK. This is a real correctness issue if the executor doesn't handle it explicitly. There's an open issue (snakemake-executor-plugin-slurm#41) about this.
5. Open issue: "Executing with --slurm always runs tasks with 1 core" (#2447). Suggests the per-rule resource translation has rough edges in production.
6. Robustness signals — package is on bioconda, active maintenance, version in the last 12 months, no Snyk vulnerabilities. No public production-scale references I could find. Maturity is "fine for the documented use case", less reassuring for the off-label pilot-job use.

Honest read: the reviewer's principle is right (don't reimplement what Snakemake does), but the specific recommendation oversells slurm-jobstep's maturity for standalone pilot-job use. The strongest version of the critique would be: try slurm-jobstep first, keep the Dask layer as a fallback only if slurm-jobstep proves unreliable in your testing. That's a defensible middle ground — the redesign would prove out the in-tree path before committing to bespoke code.