feat: fx effects pipeline — replace legacy resolver by sini · Pull Request #475 · denful/den

sini · 2026-04-23T20:21:11Z

Summary

Replace den's legacy recursive resolver with an algebraic effects pipeline (via nix-effects). The entire resolution lifecycle — from policy routing to aspect compilation to module collection — is now driven by composable effect handlers.

Remove legacy resolver — delete adapters.nix, resolve.nix, statics.nix (~450 lines). Fx handlers own everything: include resolution, parametric args, transitions, constraints, dedup.
Four-concern separation — decompose den.ctx into den.policies (directed entity edges), den.stages (scoped behavior bindings), den.schema (entity types), and den.aspects (reusable behavior).
Effectful bootstrap — pipeline setup is itself effectful via resolve-policy and resolve-target effects, making the entire lifecycle interceptable.
Per-policy named handlers — policies can declare handlers that get scope.provide'd into transition sub-computations (Layer 2 foundation).
den.ctx compat shim — always-imported, forwards den.ctx.* to den.stages.* with deprecation warnings. Covers scoped behavior + into forwarding.

481 tests passing (22 new). Net: -149 lines across 197 files.

Detailed walkthrough

Simplified Review Guide — plain-language explanation of the architecture, handler vocabulary, context propagation, and what changed from main.

Commits

Commit	What
remove legacy resolver	Delete adapters/resolve/statics, gut parametric, fx handlers own resolution
ctx-as-data transitions	scope.provide + __scopeHandlers, deferred includes, fan-out identity
edge cases	Fan-out dedup, cross-providers, perCtx, constraint cascade
parametric type separation	__fn/__args replace __functor, scope.provide replaces scope.stateful
four-concern separation	den.stages + den.policies + entity consolidation, remove den.ctx
code quality	Decompose oversized functions, naming, comments, efficiency
policies rename + compat	den.relationships → den.policies, ctx compat shim
effectful bootstrap	resolve-policy/resolve-target effects, policy handlers, depth guards, tests

Next steps (separate PRs)

aspect-chain removal — 3 files still use fromAspect/aspect-chain; replace with item.resolved
Generic fan-out isolation — current fix scopes to targetClass == "flake"; needs hierarchical dedup
Layer 2: per-policy named effects — the handlers field is wired; next is user-facing documentation and batteries that use it
Vestigial removal — parametric.nix, take.nix, perHost-perUser.nix once downstream migrates

Test plan

481/481 CI tests pass (nix develop -c just ci)
Downstream validation against:
- theutz/slashfiles
- drupol/infra
- sjcobb2022/nix-den
  -- all hosts work with ctx compat shim -- even fixed another bug the owner was facing 🥇
Review by @vic

Documentation overhaul

Rendered documentation can be previewed at: https://den.json64.dev/

🤖 Generated with the aid of Claude Code

sini · 2026-04-23T20:36:07Z

TODO:
Docs revamp - changed some things, need another update/rework
Followup work on policies w/ ACL example - need a demo template exercising them in a rich capacity
Diagram generation -- regression, need fixing
~~Have some brave users actually test this branch~~

TODO more like DONE

sini · 2026-04-23T22:51:40Z

Working on some straggling edge-cases around the flake / flake-packages / flake-system integration. Otherwise the core is ready for review, only minor fixes pending.

sini · 2026-04-24T00:11:54Z

I was hoping to dodge it -- but alas. Larger change coming in replacing forwards with an effect handler. :)

vic

I did a quick review up to my energy posibilities. Added a few comments, maybe we can also run a code-simplification run on code after forwards are done.

Ping me when I can review again.

reviewing lib

… add debug logging

…on drop

…oard paste (Ctrl+V)

The subtree collection added in 8659af2 walked all descendant scopes for every simple route. Non-flake routes (e.g. flake-parts template class routes) don't need subtree collection and it caused two bugs: packages collision with the builtin to-packages route, and missing scope args (tux) when user-scope test modules were pulled into the flake-parts route context where adaptArgs discards entity-specific args. Restrict collectFromSubtree to flake routes only. Non-flake routes fall back to single-scope lookup, matching pre-8659af23 behavior.

pnpm 11 requires Node.js v22.13+. The GitHub runner ships v20 which lacks the node:sqlite builtin that pnpm 11 depends on.

pnpm 11 blocks dependency build scripts by default. Add pnpm-workspace.yaml with onlyBuiltDependencies whitelist for esbuild and sharp, which need native compilation.

onlyBuiltDependencies was removed in pnpm 11. The replacement is allowBuilds with a map of package names to booleans.

…s null policy.instantiate can now collect class modules from any scope level (e.g., flake-system scope for perSystem class collection) instead of dropping to mainModule when no entity scope matches.

Three fixes for collect-perSystem to work correctly: 1. flake-parts.nix: merge duplicate perSystem attrs (nix-unit settings + class wiring) into single block; use evalClassModules for files class since files perSystem doesn't expose .imports 2. resolve.nix: guard mainModule fallback with `spec ? mainModule` so non-entity instantiate specs (which lack mainModule) get empty list instead of crashing when no walked modules found 3. resolve.nix: gate findHostScopeId single-child fallback to entity specs only — non-entity specs (collect-perSystem) must fall through to sourceScopeId to collect from full flake-system subtree, not just the host subtree

Replace old den.ctx/provides.forward with policy.route and den.schema.flake-parts. Each custom class (treefmt, devshell, packages, files, tests) has a route policy delivering into the flake-parts class with adaptArgs for perSystem args. Uses den.lib.aspects.resolve for the flake-parts class resolution (separate pipeline). Test helpers (tux/igloo) provided via adaptArgs closure over config.flake.nixosConfigurations. Pipeline extensions: - resolve.nix: sourceScopeId fallback for non-entity instantiate - resolve.nix: intoPath support and hasOutput check for intoPath - route/apply.nix: collectSubtree flag for non-flake routes Template outputs: packages (cowsay, htop, hola, write-files), devShells.default, treefmt, nix-unit (1 test), files/README.md, nixosConfigurations.igloo. 757/757 CI tests pass.

Replace den.ctx/provides.forward with policy.route and den.schema.flake-parts. Each custom class (treefmt, devshell, packages, files, tests) has its own file with a route policy. Pipeline changes: - resolve.nix: sourceScopeId fallback for non-entity instantiate specs - resolve.nix: merge all instantiate outputs into single module via recursiveUpdate (reverts 6118a7c grouping which broke freeform unique-type merge for multi-key outputs) - route/apply.nix: collectSubtree flag for non-flake routes Template uses den.lib.aspects.resolve for the flake-parts class. Test helpers (tux/igloo) provided via adaptArgs closure. All outputs verified: packages (cowsay, htop, hola, write-files), devShells.default, treefmt, nix-unit (1/1), files/README.md, nixosConfigurations.igloo. 757/757 CI tests pass.

stale

Replace dagre with elkjs for layout with configurable node placement strategies (LINEAR_SEGMENTS default). Add scope-based grouping with host containers and nested user sub-groups. Implement interactive collapse/expand via double-click on scope nodes with edge rerouting and hidden node count badges. Add edge hover highlighting and node click to highlight connected edges. Pipe edges excluded from layout. Update diagram-demo to include fleet views (pipe-flow, scope-topology, aspect-matrix, policy-resolution, pipe-sequence, fleet-dag, fleet-ir) and text summaries matching fleet-demo capabilities.

drupol · 2026-05-09T16:02:09Z

Massive :) congratulations!

@vic

## Summary Replace den's legacy recursive resolver with an algebraic effects pipeline (via nix-effects). The entire resolution lifecycle — from policy routing to aspect compilation to module collection — is now driven by composable effect handlers. - **Remove legacy resolver** — delete adapters.nix, resolve.nix, statics.nix (~450 lines). Fx handlers own everything: include resolution, parametric args, transitions, constraints, dedup. - **Four-concern separation** — decompose `den.ctx` into `den.policies` (directed entity edges), `den.stages` (scoped behavior bindings), `den.schema` (entity types), and `den.aspects` (reusable behavior). - **Effectful bootstrap** — pipeline setup is itself effectful via `resolve-policy` and `resolve-target` effects, making the entire lifecycle interceptable. - **Per-policy named handlers** — policies can declare `handlers` that get `scope.provide`'d into transition sub-computations (Layer 2 foundation). - **`den.ctx` compat shim** — always-imported, forwards `den.ctx.*` to `den.stages.*` with deprecation warnings. Covers scoped behavior + into forwarding. **481 tests passing** (22 new). Net: -149 lines across 197 files. ## Detailed walkthrough [Simplified Review Guide](https://gist.github.com/sini/dea443d26352b3cbd667d88ee4b34508) — plain-language explanation of the architecture, handler vocabulary, context propagation, and what changed from main. ## Commits | Commit | What | |--------|------| | remove legacy resolver | Delete adapters/resolve/statics, gut parametric, fx handlers own resolution | | ctx-as-data transitions | scope.provide + __scopeHandlers, deferred includes, fan-out identity | | edge cases | Fan-out dedup, cross-providers, perCtx, constraint cascade | | parametric type separation | __fn/__args replace __functor, scope.provide replaces scope.stateful | | four-concern separation | den.stages + den.policies + entity consolidation, remove den.ctx | | code quality | Decompose oversized functions, naming, comments, efficiency | | policies rename + compat | den.relationships → den.policies, ctx compat shim | | effectful bootstrap | resolve-policy/resolve-target effects, policy handlers, depth guards, tests | ## Next steps (separate PRs) - **aspect-chain removal** — 3 files still use `fromAspect`/`aspect-chain`; replace with `item.resolved` - **Generic fan-out isolation** — current fix scopes to `targetClass == "flake"`; needs hierarchical dedup - **Layer 2: per-policy named effects** — the `handlers` field is wired; next is user-facing documentation and batteries that use it - **Vestigial removal** — `parametric.nix`, `take.nix`, `perHost-perUser.nix` once downstream migrates ## Test plan - [x] 481/481 CI tests pass (`nix develop -c just ci`) - [x] Downstream validation against: - [theutz/slashfiles](https://github.com/theutz/slashfiles) - [drupol/infra](https://github.com/drupol/infra) - [sjcobb2022/nix-den](https://github.com/sjcobb2022/nix-den) -- all hosts work with ctx compat shim -- even fixed another bug the owner was facing 🥇 - [ ] Review by @vic ## Documentation overhaul Rendered documentation can be previewed at: https://den.json64.dev/ 🤖 Generated with the aid of [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Michael Utz <theutz@users.noreply.github.com>

sini force-pushed the feat/fx-pipeline branch from 168d607 to 7167f88 Compare April 23, 2026 20:24

sini force-pushed the feat/fx-pipeline branch 3 times, most recently from 2fb2263 to aac3d51 Compare April 23, 2026 21:09

vic added allow-ci allow all CI integration tests ai-assisted labels Apr 23, 2026

sini force-pushed the feat/fx-pipeline branch 2 times, most recently from 3de71c4 to d3060d7 Compare April 23, 2026 22:50

sini force-pushed the feat/fx-pipeline branch 3 times, most recently from 0910c3f to 3928b87 Compare April 23, 2026 23:51

vic previously requested changes Apr 24, 2026

View reviewed changes

sini force-pushed the feat/fx-pipeline branch from 3928b87 to c0d9dc5 Compare April 24, 2026 02:06

sini marked this pull request as draft April 24, 2026 03:06

sini force-pushed the feat/fx-pipeline branch 4 times, most recently from 35706cb to 2c20b4b Compare April 24, 2026 04:29

sini marked this pull request as ready for review April 24, 2026 04:45

drupol reviewed Apr 24, 2026

View reviewed changes

Comment thread modules/aspects/provides/os-class.nix Outdated

drupol reviewed Apr 24, 2026

View reviewed changes

Comment thread modules/aspects/provides/os-user.nix Outdated