Skip to content

feat: implement value resolution env spec - #2294

Merged
yohamta0 merged 7 commits into
mainfrom
feature/value-resolution-env
Jun 18, 2026
Merged

feat: implement value resolution env spec#2294
yohamta0 merged 7 commits into
mainfrom
feature/value-resolution-env

Conversation

@yohamta0

@yohamta0 yohamta0 commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • implement Spec 006 environment value resolution semantics
  • add black-box Spec 006 conformance fixtures and tests
  • mark Spec 006 as Implemented and update schema descriptions

Testing

  • make conformance CONFORMANCE_TEST_TARGET=./conformance/spec006_env
  • env DAGU_BIN=.local/bin/dagu make test TEST_TARGET=./conformance/spec006_env
  • go test package set: internal/cmn/value, internal/core/spec/types, internal/core/spec, internal/core, internal/runtime, internal/runtime/builtin/command, internal/runtime/builtin/docker
  • jq empty internal/cmn/schema/dag.schema.json
  • git diff --check

Summary by cubic

Implements Spec 006 env value resolution with ordered, top-to-bottom evaluation, strict env name validation, and shell-style expansion that preserves undefined references. Also fixes Windows env precedence by treating env names case-insensitively during merges, and stabilizes tests by improving store.MarkDispatchIndexReconcileDueForTest behavior.

  • New Features

    • Evaluate env entries in YAML source order for DAG, steps, and containers; preserve map-form order via github.com/goccy/go-yaml/parser.
    • Support ${env.NAME}, ${NAME}, $NAME, braced numeric ${1}, and POSIX ${VAR:-...}; resolve when defined and keep undefined values literal; single-quoted and unsupported text like $1a stay literal.
    • Chain scoping top-to-bottom; step/container env can read consts, runtime params, root env, and predecessor outputs; emit notices with exact field paths (including simple $VAR in run and container.env).
  • Migration

    • Env names must match ^[A-Za-z_][A-Za-z0-9_]*$; invalid names and malformed KEY=value entries now fail.
    • Undefined env references are kept literal (no OS/shell fallback) and may produce notices; validation does not require runtime-only sources.
    • Manifests relying on unordered map env evaluation should be reviewed for the new top-to-bottom behavior.
    • Run conformance: make conformance CONFORMANCE_TEST_TARGET=./conformance/spec006_env.

Written for commit e1e3caf. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features

    • Added comprehensive environment variable support with ordered evaluation and validation
    • Implemented multiple expression forms: ${env.NAME}, $NAME, ${NAME}, and shell-style operators
  • Tests

    • Added extensive conformance test suite for environment value resolution covering valid declarations, invalid cases, and runtime behavior
  • Documentation

    • Updated specification documentation marking environment variable resolution feature as implemented
    • Enhanced schema documentation with supported expression forms

@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Implements spec 006 (Value Resolution Env) end-to-end. Adds ValidEnvName for identifier validation, refactors EnvValue.UnmarshalYAML and ManifestDecoder to use AST-based parsing that preserves YAML source order for env mappings, extends the evaluation pipeline to return ordered evaluatedEnvEntry objects, adds positional/numeric $1 token support, introduces env-specific unresolved-reference notice reporting, and delivers 14 conformance fixtures plus TestValidate/TestRuntime suites. The spec is marked Implemented.

Changes

Spec 006: Env Value Resolution

Layer / File(s) Summary
Env name validator
internal/cmn/value/names.go, internal/cmn/value/template.go
Introduces ValidEnvName compiled-regex helper and switches supportedStrictBinding to use it instead of the previous inline pattern.
AST-based env YAML parsing and map-order preservation
internal/core/spec/types/env.go, internal/core/spec/manifest_decoder.go, internal/core/spec/loader.go
Refactors EnvValue.UnmarshalYAML to switch on goccy AST node types; adds preserveEnvMappingOrder pipeline to ManifestDecoder.Unmarshal; rewrites decodeDocuments to use parser.ParseBytes instead of a streaming decoder loop.
Evaluation pipeline with ordered entries and name validation
internal/core/spec/variables.go, internal/core/spec/dag.go, internal/core/spec/step.go
Expands evaluatePairs to return evaluatedEnvEntry objects, adds validateEnvPair; updates buildEnvs to use loadEnvEntriesFromEnvValue and buildStepEnvs to validate names via ValidEnvName.
Positional/numeric token expansion and preserve-on-undefined
internal/cmn/value/template.go, internal/cmn/value/envscope.go, internal/cmn/value/expand.go, internal/cmn/value/expand_test.go, internal/cmn/value/pipeline_test.go
Extends the substitution regex to match $<digits>, adds validVariableTokenName / numericVarContinues helpers, and changes undefined-variable handling to unconditionally preserve unresolved tokens rather than gating on ExpandOS.
Unresolved env-expansion notice reporting
internal/cmn/value/notices.go, internal/core/value_notices.go
Adds ReportUnresolvedEnvExpansionNotices and simpleEnvExpansionKey; wires env-specific notice helpers for root DAG env, step env, container env, and handler env into ReportValueReferenceNotices, skipping env field paths in the generic reference loop.
Unit tests for ordering, notices, and container eval
internal/core/spec/types/env_test.go, internal/core/spec/value_reference_notices_test.go, internal/runtime/builtin/docker/eval_test.go
Adds TestEnvValue_MapFormPreservesSourceOrder, three value_reference_notices_test cases for map order / earlier-scope scoping / DAG scope pass-through, and EnvEntriesEvaluateSequentially in docker eval tests.
Schema description updates
internal/cmn/schema/dag.schema.json
Updates env property descriptions at DAG, step, and container levels to enumerate supported expression forms and document source-order evaluation.
Conformance testdata fixtures
conformance/spec006_env/testdata/*
Adds 14 YAML fixtures covering valid env forms (top-to-bottom, map, root/step sources, shell boundary, direct execution, single-quoted, shell-style, braced non-env text), invalid declarations, missing references, container ordering, and runtime source validation.
Conformance test suite and spec status
conformance/spec006_env/value_resolution_env_test.go, specs/006-value-resolution-env.md, specs/README.md
Adds TestValidate and TestRuntime with exact output assertions for all resolution scenarios; marks spec 006 as Implemented.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • dagucloud/dagu#2034: Modifies the decodeDocuments function in internal/core/spec/loader.go for multi-document YAML parsing, which this PR replaces with a parser.ParseBytes-based approach.
  • dagucloud/dagu#2285: Touches internal/cmn/value/envscope.go for shell/escaped-dollar preserve behavior, directly overlapping with this PR's positional $1 capture additions to the same function.
  • dagucloud/dagu#2290: Introduces the ValueReferenceNoticeSink framework in internal/cmn/value/notices.go that this PR extends with ReportUnresolvedEnvExpansionNotices.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 28.26% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The pull request title 'feat: implement value resolution env spec' directly and accurately summarizes the main objective of the changeset, which is to implement Spec 006 environment value resolution semantics.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The pull request description includes a summary of changes, testing procedures, and details about the implementation.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/value-resolution-env

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
conformance/spec006_env/value_resolution_env_test.go (1)

12-233: Use stretchr/testify assertions or document an approved exception for conformance harness pattern.

This test suite uses a custom harness pattern (harness.NewRunner, result.ExpectExitCode(), etc.) instead of stretchr/testify assertions. While this pattern is systematic across the conformance test suite (spec002, spec004, spec005, spec006), it violates the project guideline: **/*_test.go: Use stretchr/testify assertions for Go tests. Most other test files in the project (internal/launcher, internal/upgrade, internal/workspace, etc.) follow the testify standard. Either migrate the assertions to testify (e.g., require.Equal(), assert.Contains()) or clarify whether the conformance harness is an approved exception to be documented in the guidelines.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@conformance/spec006_env/value_resolution_env_test.go` around lines 12 - 233,
The TestValidate and TestRuntime test functions use a custom harness pattern
(harness.NewRunner, result.ExpectExitCode, result.ExpectStdout,
result.ExpectStderr, result.ExpectStderrContains, dagu.ExpectNoFile,
dagu.ExpectFileContent) instead of stretchr/testify assertions as required by
project guidelines. Either migrate all these custom assertion calls to use
testify's require or assert functions (e.g., require.Equal, assert.Contains)
throughout both test functions, or update the project guidelines documentation
to explicitly approve the conformance harness pattern as an exception to the
testify requirement.

Source: Coding guidelines

internal/cmn/value/pipeline_test.go (1)

595-599: ⚡ Quick win

Rename the test case to reflect preserved behavior.

POSIXDefaultExpanded now asserts the opposite behavior ("${UNDEFINED:-default}" is preserved). Renaming (e.g., POSIXDefaultPreserved) will prevent confusion.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/cmn/value/pipeline_test.go` around lines 595 - 599, The test case
name `POSIXDefaultExpanded` is misleading because it asserts that the POSIX
default expansion syntax is preserved (not expanded) when the variable is
undefined. Rename this test case to `POSIXDefaultPreserved` or similar to
accurately reflect the preserved behavior being tested, where the input
`"${UNDEFINED:-default}"` remains unchanged in the output.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/cmn/value/expand.go`:
- Around line 146-149: The documentation comment for the expandWithShellContext
function is outdated and does not reflect the current behavior. The current code
in lines 146-149 preserves undefined variables regardless of the ExpandOS flag,
but the function's doc block still describes ExpandOS=true as applying POSIX
undefined handling. Update the doc comment block for expandWithShellContext to
accurately describe that undefined variables are now preserved for their owning
runtime regardless of the ExpandOS setting value.

---

Nitpick comments:
In `@conformance/spec006_env/value_resolution_env_test.go`:
- Around line 12-233: The TestValidate and TestRuntime test functions use a
custom harness pattern (harness.NewRunner, result.ExpectExitCode,
result.ExpectStdout, result.ExpectStderr, result.ExpectStderrContains,
dagu.ExpectNoFile, dagu.ExpectFileContent) instead of stretchr/testify
assertions as required by project guidelines. Either migrate all these custom
assertion calls to use testify's require or assert functions (e.g.,
require.Equal, assert.Contains) throughout both test functions, or update the
project guidelines documentation to explicitly approve the conformance harness
pattern as an exception to the testify requirement.

In `@internal/cmn/value/pipeline_test.go`:
- Around line 595-599: The test case name `POSIXDefaultExpanded` is misleading
because it asserts that the POSIX default expansion syntax is preserved (not
expanded) when the variable is undefined. Rename this test case to
`POSIXDefaultPreserved` or similar to accurately reflect the preserved behavior
being tested, where the input `"${UNDEFINED:-default}"` remains unchanged in the
output.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3e029e5d-8850-4509-be5c-765db946a0b1

📥 Commits

Reviewing files that changed from the base of the PR and between a770e2e and c4e4fbb.

📒 Files selected for processing (36)
  • conformance/spec006_env/testdata/braced_non_env_text.yaml
  • conformance/spec006_env/testdata/container_env_ordering.yaml
  • conformance/spec006_env/testdata/direct_execution_env_expansion.yaml
  • conformance/spec006_env/testdata/env_forms_order.yaml
  • conformance/spec006_env/testdata/env_top_to_bottom.yaml
  • conformance/spec006_env/testdata/invalid_env_declaration_name.yaml
  • conformance/spec006_env/testdata/invalid_env_declaration_shape.yaml
  • conformance/spec006_env/testdata/invalid_env_list_entry.yaml
  • conformance/spec006_env/testdata/missing_env_references.yaml
  • conformance/spec006_env/testdata/root_env_sources.yaml
  • conformance/spec006_env/testdata/shell_run_boundary.yaml
  • conformance/spec006_env/testdata/shell_style_env_expressions.yaml
  • conformance/spec006_env/testdata/single_quoted_env_references.yaml
  • conformance/spec006_env/testdata/step_env_sources.yaml
  • conformance/spec006_env/testdata/validate_missing_runtime_sources.yaml
  • conformance/spec006_env/value_resolution_env_test.go
  • internal/cmn/schema/dag.schema.json
  • internal/cmn/value/envscope.go
  • internal/cmn/value/expand.go
  • internal/cmn/value/expand_test.go
  • internal/cmn/value/names.go
  • internal/cmn/value/notices.go
  • internal/cmn/value/pipeline_test.go
  • internal/cmn/value/template.go
  • internal/core/spec/dag.go
  • internal/core/spec/loader.go
  • internal/core/spec/manifest_decoder.go
  • internal/core/spec/step.go
  • internal/core/spec/types/env.go
  • internal/core/spec/types/env_test.go
  • internal/core/spec/value_reference_notices_test.go
  • internal/core/spec/variables.go
  • internal/core/value_notices.go
  • internal/runtime/builtin/docker/eval_test.go
  • specs/006-value-resolution-env.md
  • specs/README.md

Comment thread internal/cmn/value/expand.go

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="internal/cmn/value/template.go">

<violation number="1" location="internal/cmn/value/template.go:130">
P2: Braced numeric vars can be skipped incorrectly when followed by identifier chars. `${1}a` should expand `${1}` then append `a`, but current boundary check preserves `${1}` unchanged.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread internal/cmn/value/template.go Outdated
@yohamta0
yohamta0 merged commit 7f9ffbf into main Jun 18, 2026
10 checks passed
@yohamta0
yohamta0 deleted the feature/value-resolution-env branch June 18, 2026 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant