feat: typed-handle migration, four uniqueness pillars, fumadocs docs#16
Merged
Conversation
NodeRef is a stable Arc<str>+epoch handle returned by add_node; every public API accepts str | NodeRef. @dagron.flow records call structure into a DAG; @task is signature-preserving (ParamSpec) and takes effect=Effect.{PURE,READ,WRITE,NETWORK,NONDETERMINISTIC}. FlowFuture[T] / NodeResult[T] are generic; ExecutionResult overloads type lookups by FlowFuture key. stubgen emits Literal[] overloads for typed string-keyed lookup. DAGExecutor(enforce_effect_isolation) serializes nondeterministic tasks. AST-scan warns on impure-looking PURE tasks. effects_of(dag) reads tags from node metadata.
Solid.js / Jane-Street-Incremental style auto-tracked reactive engine in py_src/dagron/reactive.py. Pure Python; weakref observer sets; glitch-free batch() context. ~10us to recompute one branch out of 10k after upstream signal mutation. Distinct from the existing dagron.execution.reactive.ReactiveDAG which wraps a pre-built DAG.
dagron.contentcache.ContentCache: filesystem-as-index CAS at ~/.cache/dagron/cas (or $DAGRON_CACHE_DIR). Atomic temp+rename writes, magic-byte header, sharded <aa>/<bb>/<rest>.cache layout. Independent processes share intermediates without coordination. Pluggable Hasher protocol; default_hash (pickle+blake2b) and numpy_hash bundled. compute_or_cached is effect-aware -- skips WRITE/NETWORK/NONDETERMINISTIC. fingerprint_function hashes co_code+co_consts+co_freevars so source mutation invalidates.
dagron.trace.TraceWriter appends per-node JSONL records; payloads stored in the Phase 6 ContentCache keyed by output fingerprint, so identical values across runs deduplicate. TraceReader reads back; replay(source, at=t) reconstructs per-node ReplayedNode state at any past wall-clock instant. Pure/READ replay byte-identically; WRITE/NETWORK/NONDETERMINISTIC are flagged replayable=False but their logged values are still surfaced. Re-recorded retries take the latest value up to the cutoff. Honors $DAGRON_TRACE_DIR.
New guide page docs/pages/guide/typed-and-reactive.mdx walks through NodeRef, @flow, generic typing, effects, reactive engine, content cache, and replay. New API reference docs/pages/api/utilities/ modern-api.mdx documents every new public symbol. Sidebar wired up. CHANGELOG bumped with [Unreleased] section listing all additions.
Side-by-side migration target — old docs/ stays live until verified.
Built on Next.js 16 + Fumadocs 16 + Tailwind v4, mirrors taskito/docs/
layout (src/app/, src/components/{ui,...}, src/lib/). 54 MDX files
migrated via scripts/migrate.mjs (drops sidebar_position+slug, removes
@site imports, converts :::tip to <Callout>). Reusable components:
ui primitives (Button, CodePanel, SectionHeader), MDX-globals
(DagDiagram, StatusBadge, EffectBadge, FeatureCard/Grid, ApiSignature,
ParamTable), client-side themed Mermaid. meta.json per directory
preserves the old sidebar order. Build passes; pnpm lint + types:check
clean. Deploy still GitHub Pages /dagron via DOCS_BASE_PATH env.
…k mermaid Strip the # Heading that duplicated the frontmatter title across 53 MDX files, plus 2 hand-fixes where titles diverged. Move typed-and-reactive into its own root section so it appears in the sidebar dropdown alongside Guide and API. Add api/index.mdx so the API root has a landing page (without it, fumadocs hid it from the dropdown). DagDiagram now remaps light-mode hex colors to dark equivalents via the original DARK_COLOR_MAP — diagrams stay legible in dark mode. FeatureCards use h-full + flex column so a row of cards has consistent height; link footer sits tight under description (no mt-auto gap).
Remove the old docusaurus tree under docs/, rename docs-next/ → docs/. Update .github/workflows/docs.yml: pnpm + Next.js static export with DOCS_BASE_PATH=/dagron, artifact path docs/out (was docs/build). Update .pre-commit-config.yaml biome hook to run pnpm exec biome check inside the new docs/ structure. Anchor /lib/ in root .gitignore to repo root so it doesn't catch docs/src/lib/.
Split the docs biome hook into docs-biome (lint) + docs-types (fumadocs-mdx typegen + tsc --noEmit), so MDX edits trigger the types check too. Add `pnpm check` script that runs both, and `pnpm lint:fix` for the auto-fix flavour.
The new docs-biome and docs-types pre-commit hooks invoke `cd docs && pnpm exec ...`. The lint job had no Node/pnpm, so both hooks failed with "pnpm: command not found". Add pnpm/action-setup + actions/setup-node + a frozen-lockfile install before pre-commit runs.
test_reactive_bench.py uses pytest-benchmark which isn't a CI dependency — add it to the existing --ignore list. Force UTF-8 when writing the synthesized mypy snippet in test_mypy_reveal_types (Windows default CP1252 broke the em-dash) and replace the em-dash with a plain hyphen for belt-and-braces.
test_two_pure_nodes_run_in_parallel asserted elapsed < 0.15s; macOS CI runner blew that with 0.158s. Replace the wall-clock check with a threading.Barrier(2) + max_active counter — deterministic: both tasks must enter the critical section together for the assertion to hold, otherwise the barrier times out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two coordinated efforts on one branch:
Direction A — typed-handle migration (Phases 1-3)
NodeRefstable handle returned byadd_node; every public API acceptsstr | NodeRef. Per-node epoch detects remove-then-readd viaStaleNodeRefError. Existing string-based callsites unchanged.@dagron.flowTawazi-style compose API: write a Python function, the call structure becomes the DAG. Pythonic, no string IDs.FlowFuture[T],NodeResult[T],ExecutionResult.__getitem__overloads typed byFlowFuture[T]key.dagron.stubgenemitsLiteral[\"...\"]overloads so even string-keyed lookups become statically typed.Direction B — four uniqueness pillars (Phases 4-7)
Effect.PURE/READ/WRITE/NETWORK/NONDETERMINISTIC) with AST-scan heuristic andDAGExecutor(enforce_effect_isolation=True).dagron.reactive) — Solid.js / Jane-Street-IncrementalstyleSignal/Computed/Watcherwith auto-tracked deps. ~10 µs to recompute one branch out of 10 000 after upstream signal mutation.dagron.contentcache) — Nix-flake-style. Filesystem is the index; two CI workers share intermediates without coordination.dagron.trace) — append-only JSONL traces + payload-deduped CAS.replay(at=t)reconstructs any past run state.Docs migration (bonus)
byteveda.github.io/dagron/).scripts/migrate.mjs(admonitions →<Callout>,:::tipsyntax converted, frontmatter normalized, duplicate H1s stripped).ui/primitives (Button,CodePanel,SectionHeader), MDX globals (DagDiagram,StatusBadge,EffectBadge,FeatureCard/Grid,ApiSignature,ParamTable), client-side themedMermaidwith dark-mode color remap.meta.json.docs.ymlworkflow usespnpm+DOCS_BASE_PATH=/dagron; pre-commit gainsdocs-biome(lint) +docs-types(typegen + tsc).Test plan
cargo test --workspace— 308 Rust tests passuv run pytest tests/python/ --ignore=tests/python/test_benchmarks.py— 831 Python tests pass (685 pre-existing + 146 new across NodeRef / flow / typing / effects / reactive / contentcache / replay)cargo clippy --workspace --all-targets -- -D warnings— cleancargo fmt --all --check— clean.venv/bin/ruff check py_src/ tests/— cleanuv run mypy py_src/dagron/ tests/— cleancd docs && pnpm install && pnpm check— biome + types passcd docs && DOCS_BASE_PATH=/dagron pnpm build— static export succeeds (60 pages)Docsworkflow on master rebuilds and republishes to GitHub Pages