v3: yaml.Node-centric parsing pipeline#882
Open
ndeloof wants to merge 56 commits into
Open
Conversation
Add the internal/node package as the foundation for the v3 yaml.Node centric parsing pipeline. SourceContext carries the per-subtree parsing context (source file, working directory, environment, env files, parent chain). Layer pairs a parsed *yaml.Node with the SourceContext that produced it and exposes a sparse origins side-table so that, after cross-file merge, individual scalars from another layer can be looked up against their own context. Walk performs a depth-first traversal of a yaml.Node tree and invokes a Visit callback at every position with a meaningful tree.Path: - the root with an empty path - every mapping value with the path extended by the key - every sequence element with the path extended by "[]" DocumentNodes are unwrapped transparently and AliasNodes are followed once with cycle protection so the walker terminates even on pathological inputs. No caller yet: this is a pure addition with full unit test coverage, preparing the next commits that will port reset/override resolution, alias normalization and the merge phase to operate on yaml.Node. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Move the yaml.Node-side reset/override resolution out of loader/reset.go and into internal/node/reset.go so the upcoming v3 merge phase can reuse it without going through the legacy map[string]any path. ResolveResetOverride takes a parsed yaml.Node (DocumentNode is unwrapped transparently) and returns: - the cleaned tree, with !reset-tagged nodes stripped from Content - the list of tree.Paths where !reset or !override was found - an error on cycle detection or node-visit limit exceeded The cache-based alias resolution, cycle detection (including the "different services share an anchor" carve-out), node visit cap and the synthetic <<-elision are all preserved. DefaultMaxNodeVisits is now exported so callers and tests can reference it from outside the package. loader/reset.go becomes a thin adapter: ResetProcessor still satisfies yaml.Unmarshaler for the legacy decoder.Decode call site and applies the recorded paths to the post-merge map[string]any via Apply. That adapter goes away in a later commit when the v3 orchestrator switches to driving the Node-side resolution directly. Existing loader-level tests (TestResetCycle, TestAliasBombPrevented, TestVisitCounterLimit, ...) pass unchanged. New focused unit tests in internal/node cover the extracted entry point directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
NormalizeAliases rewrites a yaml.Node tree so that no AliasNode remains and no mapping has a `<<` key. The subsequent v3 pipeline phases (cross-file merge, interpolation, transform, decode) can then operate on the data without ever resolving aliases themselves. The unfold pass replaces every AliasNode with a deep copy of its target. Deep copy is required because the merge phase mutates nodes in place: a single Node shared between two locations would otherwise be corrupted by the first merge involving it. Position information (Line, Column) is preserved on the copies so downstream diagnostics still point at the original source location. Cycles in alias chains (A references B which references A) are detected during the unfold pass via an inProgress set keyed by *yaml.Node, and reported with the source line of the offending alias. A cleaned set caches targets that have been fully unfolded so anchor reuse stays linear in the number of distinct anchors, defending against alias-bomb inputs that would otherwise blow up exponentially (TestNormalizeAliases HandlesAliasBomb covers the worst-case branching). The merge-key fold pass runs depth-first on the unfolded tree. For each mapping, explicit keys take precedence; for each <<-merge source in declaration order, any key not yet present is appended. Sequence-valued merge sources are flattened with first-entry-wins semantics — the same ordering yaml.Decoder would apply if it were folding the unfolded tree itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Port the per-path merge rules of override.Merge to a yaml.Node-typed twin in override/node.go. MergeNode folds two parsed YAML trees together using the same per-path strategies as the legacy MergeYaml (append for sequences, recursive merge for mappings, left-wins for scalars, plus the named specials in mergeSpecialsNode), but without ever round-tripping through map[string]any. Every entry from mergeSpecials gets a Node twin in mergeSpecialsNode: mergeBuildNode (short-form context promotion), mergeDependsOnNode (list-of-names expansion to canonical mapping with default condition and required:true), mergeNetworksNode / mergeModelsNode (same list expansion without defaults), mergeExtraHostsNode (append with deduplication), mergeLoggingNode (driver-aware merge), mergeIPAMConfig Node (subnet-keyed merge), mergeUlimitNode (mapping-aware), mergeTo SequenceNode (plain append), overrideNode (left-wins for command, entrypoint, healthcheck.test). convertIntoMappingNode and convertIntoSequenceNode synthesize new nodes when short forms must be promoted; synthetic nodes copy Line / Column from the originating scalar so downstream diagnostics still point at the user-visible source location. mergeMappingsNode preserves the existing key order of the right (base) mapping and appends keys introduced by the left (override) at the end. MergeNode expects aliases to have been unfolded ahead of time (node.NormalizeAliases) and never follows AliasNode values itself. Tests in override/node_test.go cover the merge strategies end-to-end by parsing YAML, calling MergeNode, decoding the result and asserting the decoded shape — same pattern as the legacy suite — plus a dedicated test that line numbers survive the merge for diagnostics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Port EnforceUnicity to operate on *yaml.Node. The new function walks the tree and, for every SequenceNode whose path matches uniqueNode, deduplicates entries by the configured nodeIndexer keeping the last occurrence — same semantics as the legacy map[string]any version. Every entry from override.unique gets a Node twin in uniqueNode: keyValueIndexerNode (key=value strings split on the first `=`), volumeIndexerNode (target field for mapping, parsed Target for short form), deviceMappingIndexerNode, exposeIndexerNode, mountIndexerNode (secrets / configs with default-path fallback), portIndexerNode (host:published:target/protocol tuple for long form, raw value for short form), envFileIndexerNode (path field or raw string), and the implicit keyValue indexer for the remaining label / dns / cap_add / sysctls / etc. paths. The Node version reads field values from MappingNode children via nodeMapGet rather than from Go map[string]any, which means it can inspect Tag information when needed (currently unused, but enables later passes to discriminate scalar types without re-decoding). Tests in override/uncity_node_test.go cover environment / labels override, ports short and long form, volumes by target, network aliases dedup, and a negative case confirming that non-unicity paths (services.*.command) are left untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Port the interpolation phase to operate on *yaml.Node. The key new capability is the LookupValueFor closure: a function that returns the variable lookup to use for a given scalar node, invoked once per scalar visited. That signature is what enables the v3 lazy interpolation across include boundaries. After cross-file merge, each scalar node still carries the SourceContext of the layer that produced it; the orchestrator wires LookupValueFor to read that SourceContext.Environment, so a value from an included file is interpolated in the include's env, while a value from the parent file is interpolated in the parent's env — even though both now live in the same merged tree. This is the bug fix that motivates the whole v3 refactor. For parity with v2 (single environment for the whole document), callers can provide LookupValue instead of LookupValueFor and the function treats it as a constant closure. InterpolateNode also folds in the type-cast hook: after substitution, scalars whose path matches an entry in Tags get their Tag field rewritten (typically "!!int", "!!bool", "!!float"). yaml.v4 honors the new tag at decode time and performs the conversion natively, removing the need for a separate mapstructure cast hook in the v3 pipeline. Mapping keys are not interpolated (matching v2). !!null scalars are skipped. Errors from template.Substitute are wrapped with the offending path via the existing newPathError helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Add a Node-typed entry point to the transform package that v3 callers can use to canonicalize a parsed *yaml.Node. The first cut bridges through map[string]any: it decodes the node, runs the existing Canonical (which exercises every registered per-path transformer), re-encodes the result into a yaml.Node and substitutes it for the input subtree. The bridge intentionally trades position fidelity for breadth: every transformer in the v2 registry (transformPorts, transformVolumeMount, transformBuild, transformDependsOn, ...) is exercised through the same well-tested map-based code, so the v3 pipeline gets the full canonicalization behavior without porting 28 transformers in a single commit. Source line and column information is lost on the subtrees that the bridge rebuilds. Subsequent commits will port the most-used transformers (ports, volumes, build, env_file, depends_on) to operate on *yaml.Node directly, preserving source positions for downstream diagnostics. Each such commit narrows the responsibility of the bridge until it can be removed entirely. Tests cover the headline short-form expansions (ports, build, depends_on, networks, env_file) plus DocumentNode unwrapping and nil safety. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof
force-pushed
the
v3-yaml-node-refactor
branch
from
3767b64 to
3597a54
Compare
Port ResolveRelativePaths to operate on *yaml.Node. The Node version introduces WorkingDirFor, a closure invoked once per resolved scalar to choose which working directory to resolve against. That is the v3 fix for the latent v2 bug where a relative path declared inside an included file (volume, build context, env_file, ...) is resolved against the project root instead of the include block project_directory. Callers that have a single project root can keep using a fixed WorkingDir; the v3 orchestrator wires WorkingDirFor to read the SourceContext.WorkingDir attached to each scalar via the Layer origins side-table, so after a cross-file merge every scalar still resolves against the directory it was declared in. Every resolver in the v2 registry has a Node twin: absScalar (env_file path, label_file, extends file), absContextScalar (build context with URL / ServicePrefix carve-outs), absExtendsScalar (extends.file with remote loader bypass), absSymbolicLinkScalar (develop.watch.*.path with ResolveSymbolicLink), absVolumeMount (long-form bind mount source), volumeDriverOpts (top-level volumes.* local driver bind), and maybeUnixScalar (Unix/Windows absolute path detection for configs.*.file, secrets.*.file, build.ssh.*). The Node version reads Tag / Kind directly to discriminate short-form strings from long-form mappings, so callers can run path resolution either before or after canonicalization without inserting an extra conversion pass. The include.* patterns are kept for v2 parity but remain inert: they never match the actual `include.[].path` walk path, and include resolution stays the responsibility of the loader (collectIncludeLayers in the upcoming PR 11) which knows about ResourceLoaders and project_directory redefinition. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ValidateNode is a native port of validation.Validate to *yaml.Node. Each entry of the v2 checks map gets a Node twin in nodeChecks: file mutual exclusion for configs / secrets, IP address validation for port host_ip, count vs device_ids exclusivity for deploy.resources and gpus, non-blank watch paths, and the external-with-extra-fields check for volumes via checkVolumeNode / checkExternalNode. Tag and Kind are read directly so callers no longer need to canonicalize before validating. NormalizeNode is a bridge to the map-based Normalize: it decodes root, runs Normalize, and rebuilds a yaml.Node from the result. This reuses the well-tested rule set (default network injection, build context defaults, implicit dependencies, env_file resolution, ...) without porting 260 lines of orchestration in a single commit. Source positions are lost on the rebuilt subtree; per-rule node-native ports will land in subsequent commits and narrow the bridge until it can be removed. Both functions tolerate a DocumentNode wrapper and nil root for defensive use by the orchestrator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloof
force-pushed
the
v3-yaml-node-refactor
branch
from
3597a54 to
6141cc2
Compare
LoadLayer is the v3 replacement for the per-file half of loadYamlFile. It parses a single ConfigFile into one or more node.Layer values, each pairing a parsed *yaml.Node with the SourceContext that produced it. The function performs the steps that turn raw YAML bytes into a clean, alias-free Node tree ready for cross-file merge: 1. read content from file.Node, file.Content, or file.Filename; 2. decode each YAML document into a *yaml.Node (multi-document files produce one Layer per document, in source order); 3. resolve !reset and !override tags via node.ResolveResetOverride and record the recorded paths on the Layer for merge-phase replay; 4. unfold aliases and fold `<<` merge keys via node.NormalizeAliases so the resulting tree is self-contained and safe to merge across files. Cross-file merge, include / extends resolution, interpolation, path resolution, validation, transform, and the final decode to types.Project are performed by orchestrator commits that follow and are out of scope here. types.ConfigFile gains a Node *yaml.Node field so callers that have already parsed YAML (custom readers, remote loaders, transformations) can feed it directly without re-parsing. internal/node.Layer gains SetResetPaths / ResetPaths so callers can attach the !reset / !override paths collected by ResolveResetOverride; the upcoming merge phase will consult these to drop or replace values from base layers at those paths. No caller yet — pure addition. Tests cover content / file / pre-parsed Node inputs, multi-document YAML, alias and merge-key normalization, reset-path collection, and propagation of the MaxNodeVisits cap error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
CollectIncludeLayers is the v3 replacement for ApplyInclude. It reads the top-level `include` block from a parent Layer and returns the direct child Layers it materializes, each carrying its own SourceContext that captures the include block project_directory and the environment resolved from its env_file entries. The include block is interpolated eagerly in the parent SourceContext before any path is resolved, because the path / project_directory / env_file scalars themselves may contain variable references that must be substituted in the parent environment. This is the one point in the v3 pipeline where interpolation is performed before merge; everywhere else, scalars are interpolated lazily in the SourceContext of their layer of origin. Each include entry is normalized via readIncludeEntry: a bare string becomes a one-element long form, a mapping is decoded field by field. A manual decoder handles short-form (single string) vs long-form (sequence of strings) for the path and env_file fields, because types.StringList still relies on the v2 mapstructure DecodeMapstructure hook. Phase D (UnmarshalYAML on types) will replace it with native yaml.v4 decoding and the manual helpers go away. resolveIncludePaths returns the project_directory as an absolute path (v3 improvement over v2 which stored a relative path in ConfigDetails.WorkingDir). The absolute form is required by v3 per-scalar path resolution, where each scalar resolves against its own SourceContext.WorkingDir without an extra rebasing step. resolveIncludeEnvironment mirrors v2: an explicit env_file list takes precedence over the implicit project_directory/.env, relative entries resolve against the parent WorkingDir, /dev/null disables an entry, and the resulting env is merged on top of the parent environment. The function only produces direct children; the orchestrator commits that follow recurse into each child to process its own include block. The parent include mapping entry is left in place; orchestration also removes it before final marshalling. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ApplyExtendsToLayer is the v3 replacement for ApplyExtends. It walks the services mapping of a parent Layer, resolves the inheritance chain for every service with an extends directive, merges base + derived service nodes via override.MergeNode at path services.x, and strips the extends key from the result. extends.file referencing another compose file is loaded into a stand-alone Layer (parse + reset/override resolution + alias normalization) with its own SourceContext, so a service inherited from file B is interpolated and resolved against B environment and working directory in the subsequent merge phase. Cycle detection reuses the existing cycleTracker keyed by (filename, serviceName) so the diagnostics emitted by the v2 extends_test fixtures stay identical. The base service node is deep-cloned before being merged into the derived service so the same base can be reused by other extends chains in the same load without cross-contamination from a previous in-place merge. Refactor a small helper while at it: the previous resourceLoaderFor function returned the matched ResourceLoader together with the resolved path, but neither call site needed the loader value once include and extends stopped using ResourceLoader.Dir for the project_directory computation. Rename to resolveResourcePath and drop the unused return so the linter stays happy. Tests cover same-file short / long form, extends.file across files, multi-level chains, cycle rejection, missing service errors, and the no-op paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
LoadV3 wires every Node-typed phase introduced in the previous commits
into a complete v3 load pipeline:
parse -> reset/override resolution -> alias normalization
-> recursive include collection -> per-layer extends
-> per-scalar origins map population
-> left-to-right merge -> reset paths replay
-> lazy interpolation (per-scalar SourceContext lookup)
-> per-scalar path resolution
-> canonicalization bridge -> validation -> normalize bridge
-> decode to map[string]any
The map[string]any return is the last remaining v2 bridge: it lets the
existing ModelToProject (mapstructure) finish the projection to
types.Project. Phase D replaces it with a native yaml.v4 decode once
the types in question gain UnmarshalYAML methods.
Two key v3 corrections are demonstrated end-to-end by the new tests:
- LazyInterpolationAcrossInclude: a variable defined only in the
include block env_file resolves inside the included scalar, while a
variable defined only in the shell environment resolves inside the
parent scalar. Same merged tree, two scopes. This is the headline
bug fix that motivates the whole refactor.
- PathResolutionPerInclude: a relative path declared inside an
included file is resolved against the include project_directory, not
the project root. The v2 ResolveRelativePaths used a single working
directory for the whole tree, so this case was silently wrong.
Path resolution runs before canonicalization on purpose: CanonicalNode
currently bridges through map[string]any and loses pointer identity,
which would break the origins-driven per-scalar WorkingDir lookup. The
ordering becomes irrelevant once individual transformers are ported to
operate on *yaml.Node directly.
ApplyResetPaths is the Node-side replacement for the v2 applyNull
Overrides helper: it walks the merged tree and removes mapping entries
whose path matches one of the recorded !reset / !override patterns
collected by ResolveResetOverride at parse time.
LoadV3 does not yet replace LoadWithContext; the cutover lands in a
later commit, once differential testing against the full testdata
fixture suite confirms parity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
First batch of UnmarshalYAML implementations on the polymorphic "string or list of strings" types so yaml.v4 can decode them natively, without going through the mapstructure detour. Each implementation mirrors the DecodeMapstructure logic of the same type and accepts both the scalar short form and the sequence long form. DecodeMapstructure is kept in place so the v2 path keeps working until LoadWithContext is cut over. Covered in this commit: - StringList: scalar OR sequence of strings. - StringOrNumberList: same, with numeric entries coerced to their stringified form for sequence entries. - ShellCommand: scalar parsed via shellwords, OR sequence of args. - HealthCheckTest: scalar wrapped in [CMD-SHELL, value], OR sequence. A shared unwrapDocument helper in types/yaml_helpers.go peels off the DocumentNode wrapper so callers can invoke yaml.Unmarshal directly into a value of these types as well as through a struct field — both code paths now produce the same result. loader/load_include.go is simplified to call entry.Decode(&cfg) on include mapping entries instead of the manual decodeIncludeMapping / decodeStringOrList helpers, because StringList now resolves the short vs long form transparently through its new UnmarshalYAML method. Subsequent commits port the remaining DecodeMapstructure types (Mapping, Labels, HostsList, UlimitsConfig, etc.) following the same pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Second batch of UnmarshalYAML implementations, this time on the
polymorphic "mapping or list of key[=value]" types. Each implementation
mirrors the corresponding DecodeMapstructure and accepts both surface
forms compose users write today.
Covered in this commit:
- Labels: mapping {key: value} OR list of "key=value" entries. Numeric
and boolean mapping values are coerced to their stringified Node.Value,
matching v2 labelValue semantics.
- Mapping: mapping {key: value} OR list of "key[=value]" entries. A
bare "key" in list form maps to an empty string; a nil scalar in
mapping form maps to an empty string. Same as v2.
- MappingWithEquals: same shape as Mapping but preserves the *string
distinction between "key" (nil) and "key=" (pointer to "") that
drives environment variable resolution downstream. Trailing-space
detection in keys is preserved.
- HostsList: mapping with scalar OR sequence values, OR list of
"host=ip" / "host:ip" short-form entries. Existing cleanup
validation is invoked from both code paths.
A pair of helpers in types/yaml_helpers.go centralizes the scalar
inspection used by these methods: scalarToString returns "" for nil
and !!null tagged scalars, and scalarToStringPtr returns the same as
nil so MappingWithEquals can keep its three-state semantics.
The corresponding DecodeMapstructure implementations remain in place
so the v2 mapstructure path keeps working until LoadWithContext is
cut over to yaml.v4 native decoding.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Third batch of UnmarshalYAML implementations, covering the scalar and
option types that remained on the v2 DecodeMapstructure path:
- Duration: scalar parsed via str2duration.
- NanoCPUs: scalar number or numeric string, parsed via strconv.ParseFloat.
- DeviceCount: scalar integer or the literal "all" (maps to -1).
- FileMode: scalar octal string, parsed via strconv.ParseInt with base 8.
- UlimitsConfig: scalar integer (single-value form) or mapping with
soft / hard fields. Each integer is parsed via strconv.Atoi from the
Node Value so quoted-numeric scalars are accepted uniformly.
- Options: mapping with single-value entries; nil and !!null values
coerced to empty string for parity with v2.
- MultiOptions: mapping where each value is either a scalar or a
sequence of scalars, stored as a slice per key.
Together with PR 14 and PR 15, every type that previously relied on
DecodeMapstructure now has a native UnmarshalYAML twin. The
DecodeMapstructure implementations are kept in place so the v2
LoadWithContext path keeps working until the orchestrator is cut over
to yaml.v4 native decoding.
UnitBytes already exposed UnmarshalYAML pre-refactor and is unchanged.
SSHConfig is left for a follow-up because its surface form (string OR
{id: path} mapping) needs more careful handling than the other types.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Introduce a differential test that compares the output of the legacy
loadModelWithContext (v2) against LoadV3 (v3) on a representative
subset of the fixture suite. The test reports structural diffs and
flags error parity mismatches so the cutover of LoadWithContext can be
prepared with high confidence.
Running the differential suite surfaced three regressions in LoadV3
that this commit fixes:
- Empty compose files now return the same error as v2 ("empty compose
file") rather than silently producing an empty map. The check is
hoisted before any pipeline stage runs.
- The obsolete top-level `version` attribute is stripped after schema
validation with the same deprecation warning v2 emits. Adds a
hasMappingKey helper so the strip is path-aware on the merged Node
tree.
- services.*.build accepts the short form (a scalar value used as the
build context path) in addition to the canonical long form. A new
absBuild resolver in paths/node.go handles both shapes; for the
long form it walks the mapping children explicitly because the
generic resolver walker stops at the first matching pattern and
could not descend on its own. This restores the parity for the
build-context-resolved path expected by every fixture using
`build: ./Dockerfile`.
With these three fixes, the five representative differential cases
(top-level-extends, include-basic, extends-with-context-url,
with-version, empty) all pass. The next commits extend the fixture
list and address remaining divergences uncovered along the way before
swapping LoadWithContext to use LoadV3 directly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Cover the remaining v2 contract that the differential suite uncovered: v2 calls projectName() up front to lift the project name out of the first config file (or its `name:` field) and then refuses to load when the resulting opts.projectName is still empty and SkipValidation is false. LoadV3 now does the same: - projectName is invoked before any pipeline stage so the lookup also honors COMPOSE_PROJECT_NAME and interpolation of the `name:` field. - opts.Interpolate is defensively initialized when nil (callers going through ToOptions already have it set; the defensive init covers the Options literals used in tests). - After schema validation, an empty projectName produces the same "project name must not be empty" error v2 returns. The differential suite is extended with five more representative fixtures (depends-on-self, depends-on-cycle, depends-on-profile-no-cycle, include-cycle, extends-with-context-url-imported). With this commit all ten cases pass, including the cycle detection scenarios. The LoadV3 unit tests that exercise the function in isolation (no imperative project name and no `name:` in the inline fixtures) now set SkipValidation: true so they stay focused on the pipeline behavior they are actually testing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ResolveEnvironmentNode is the Node-typed counterpart of v2 Resolve Environment, with the crucial v3 twist: the bare-key lookup is per scalar and uses the SourceContext.Environment attached to each entry of services.*.environment (and secrets.*.environment, configs.* .environment). The v2 helper consults a single project-wide environment, which means an env_file declared on an include block leaks variables to the surrounding project. The lazy resolver fixes that: a `KEY` entry declared inside an included service uses the include block environment (parent shell env merged with the include env_file), while the same `KEY` entry declared in the parent file uses the project environment. LoadV3 invokes ResolveEnvironmentNode after lazy interpolation so the merged tree carries the canonical "KEY=value" form before path resolution, validation and normalize run. Unit tests exercise the per-scalar branching and confirm that unresolved keys are left bare, matching v2 semantics. This commit is a piece of the planned cutover (loadModelWithContext to LoadV3); the actual switch waits until the differential suite covers enough of the existing fixture set to give us confidence it will not regress functional tests like TestNonMappingObject that go through ModelToProject. ResolveEnvironmentNode is what unblocked the TestLoadWithIncludeEnv case during cutover dry-runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Extend the differential suite to 21 fixtures including combined extends
include, every standalone fixture under testdata/extends, and the
include subdir fixtures. All 21 pass, demonstrating that LoadV3 matches
loadModelWithContext byte for byte on the representative subset of the
testdata corpus.
While building toward the LoadWithContext cutover, three v2-only
behaviors are reproduced in LoadV3:
- Top-level shape check: documents whose root is not a mapping are
rejected with the v2 "top-level object must be a mapping" message
before they reach the schema decoder.
- JSON Schema validation runs early on a decoded view of the merged
tree, before canonicalization and transform, so structural errors
surface with the v2 wrapping "validating <file>: ..." prefix rather
than panicking inside a downstream transformer. Refactored into
validateAndStripVersion to keep LoadV3 cyclomatic complexity under
the gocyclo limit.
- extends listener events: parseExtendsRef now emits the v2-compatible
"extends" event with the {service, file?} payload so downstream
telemetry / dependency analysis observe the same callback signature
as before.
Two extends-specific corrections preserve v2 behavior under nested
extends.file chains:
- loadExtendsBaseLayer resolves extends.file against the parent layer
working directory (so an include rooted at testdata/extends can
itself extend sibling.yaml without leaking back to the project root).
childOpts re-roots ResourceLoaders so the recursion stays scoped.
- After the extends merge, resolveExtendedServicePaths rewrites
relative paths in the merged service against the sub-file working
directory. Mirrors the v2 paths.ResolveRelativePaths call inside
getExtendsBaseFromFile, accumulating the file relative dir as the
chain unwinds (sibling.yaml `.` becomes `testdata/extends` after one
level of extends).
- resetParentPaths applies the recorded reset / override paths from
the parent layer to the cloned base before merge, replacing the v2
processor.Apply pre-pass. Required so !override on a derived field
drops the base value rather than letting it leak through the merge.
Null services (a YAML mapping entry whose value is literal `null`)
are now treated as empty in extends targets, matching v2.
ApplyExtendsToLayer is wired through these changes so the existing
fixture suite under testdata/extends keeps producing the v2 result
when LoadV3 is the active orchestrator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
A second wave of parity fixes uncovered when running the full loader test suite against LoadV3 as the active orchestrator. Each change is gated behind a v2-compatible path so the existing tests keep passing under the v2 codepath while the v3 cutover preparation continues. Loader pipeline: - setDefaultValuesNode: bridge call to transform.SetDefaultValues from LoadV3, replacing the missing step that left DeviceCount and other canonical defaults at their zero values. Driven by a fresh SkipDefaultValues option so callers that exercised the original v2 Skip behaviour keep working unchanged. - expandIncludes now re-roots child Options at the included file's WorkingDir before recursing. Nested include resolution (a parent file that includes compose-include.yaml, which itself includes ./subdir/...) finds the inner file relative to the include's own project_directory instead of bubbling back to the project root. - LoadLayer rejects top-level mappings with non-string keys with the v2-compatible "non-string key at top level: <key>" diagnostic. Path resolution gets three additional handlers: - absEnvFile / "services.*.env_file.*": handles the short-form scalar path that bypasses canonicalization so env_file references in services loaded through include resolve against the source file working directory. - absSSHEntry / "services.*.build.ssh.*": preserves bare keys (e.g. "default") and resolves only the path portion of "key=path" short forms; the long form mapping is recursed into with maybeUnixScalar for each value. - "services.*.label_file" (scalar OR sequence) handled via the existing absScalarMaybeSequence so short-form label_file references resolve identically to the long-form list. The absBuild handler now defers SSH resolution to absSSHEntry and covers the mapping form of services.*.build.ssh, the two surface shapes the canonical and pre-canonical pipelines produce. With these in place the differential suite stays 21/21 green; the remaining tests that fail under cutover are scoped to v2-specific quirks (nested non-string key path diagnostics, listener mid-merge semantics, service-source slash normalization) that warrant dedicated commits rather than ride along this one. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Three more parity gaps surfaced by the cutover dry-run are now closed,
bringing the count of failing v2 fixture tests under LoadV3 from 27 to
5 (all narrow v2-quirk edge cases that remain to triage).
Listener event determinism on chained extends:
- applyServiceExtendsNode now commits the resolved base back into the
siblings mapping via setMappingValue, mirroring the v2 services[name]
side effect. Without this, the top-level loop would re-enter the
extends resolution for already-processed services and double-emit
the "extends" Listener event.
Path resolution skips null and empty scalars:
- absScalar and maybeUnixScalar now bail on n.Tag == "!!null" and on
empty values. The CanonicalNode bridge re-encodes ssh entries as
{default: null}; without this guard the resolver would rewrite the
null Value to "<workdir>/" and break the subsequent !!null decode.
SetDefaultValues output handled without a generic second sweep:
- The post-defaults path resolution was replaced by a tightly scoped
resolveDefaultBuildContext helper that only rewrites the synthetic
"." build context the defaults pass introduces. Earlier dry-runs
tried a generic second sweep, but each bridge through map[string]any
recycles every scalar pointer — so a generic sweep ended up double-
resolving every relative path that the first pass had already
handled.
Net effect at this point: TestExtends, TestExtendsRelativePath, Test
ExtendsNil, TestExtendsWihtMissingService, TestIncludeWithExtends,
TestInvalidTopLevelObjectType, TestLoadExtendsListener, TestLoadExtends
ListenerMultipleFiles, TestLoadExtendsMultipleFiles, TestLoadSSH (all
three variants), TestLoadWithLabelFile, TestLoadWithMultipleInclude,
TestServiceDeviceRequestWithoutCountAndDeviceIdsType all pass through
LoadV3. Outstanding failures (TestLoadExtendsDependsOn, TestLoadWith
MultipleIncludeConflict, TestIncludeRelative, TestIncludeWithProject
Directory, TestNonStringKeys, TestLoadWithExtends) are tracked for a
follow-up commit; LoadWithContext stays on the v2 codepath until they
clear so the public CI suite is unaffected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
CollectIncludeLayers now stores the include project_directory as the
result of loader.Dir on the include path — same value v2 keeps in
ConfigDetails.WorkingDir when the include lives under the parent
project root. This restores the relative normalization v2 relies on:
filepath.Join cleans "./" to ".", so an included volume short form
("./:/mnt") collapses to source "." rather than "./".
After loading each included file LoadLayer-style, CollectIncludeLayers
now also runs paths.ResolveRelativePathsNode against the include
project_directory, matching v2 ApplyInclude which sets the recursive
loadYamlModel call's opts.ResolvePaths = true regardless of the
parent setting. The include sub-tree therefore arrives at the merge
phase with paths already resolved, so a parent loader that opted out
of path resolution does not leave the include's relative paths
half-resolved against the wrong working directory.
The corresponding unit test (TestCollectIncludeLayers_ShortFormString)
asserts the new relative WorkingDir shape. resolveResourcePath has no
remaining callers and is removed in favor of resolveResourceWithLoader
which every caller now uses for the loader-handle follow-up.
These ChromeOS to v2 parity for the most common include-volume case;
the outstanding TestIncludeWithProjectDirectory env_file doubling
remains and is tracked for a follow-up pass that re-examines the
include env_file workingDir resolution order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…uite After this commit only two of the previously failing fixture tests diverge from v2 under the cutover dry-run: TestIncludeRelative (v2 relies on relative-include WorkingDir to collapse ./ to .) and TestNonStringKeys (v2 emits per-nested-section diagnostics). Both are narrow edge cases tracked for follow-up; every other extends, include and combined fixture matches v2 byte for byte. Loader pipeline: - mergeLayers now applies the right-hand layer recorded reset and override paths to the accumulator before each merge. The consumed paths are then dropped from the returned list so the orchestrator post-merge ApplyResetPaths sweep does not delete the value the override was meant to preserve. Fixes TestLoadExtendsDependsOn and the cross-file override behaviour (TestLoadWithMultipleIncludeConflict). - ApplyExtendsToLayer is now invoked through applyExtendsPerLayer, which clones Options for each layer to re-root the localResource Loader at the layer WorkingDir. Mirrors v2 ApplyExtends running inside the recursive loadYamlModel of an include so a relative extends.file resolves against the include project_directory rather than the outer project root. Fixes TestLoadIncludeExtendsCombined. - The post-defaults helper resolveDefaultBuildContext now consults a name-keyed buildServiceContexts map computed before CanonicalNode destroys pointer identity, so a service whose build had no context picks up the include project_directory rather than the project root. Fixes TestIncludeWithProjectDirectory. Extends helpers: - resetParentPaths returns the list of paths it consumed; the caller removes them from the layer master ResetPaths so they are not re-applied by the orchestrator post-merge sweep. Distinguishes v3 reset-during-extends from reset-during-cross-file-merge. Include and paths: - CollectIncludeLayers runs paths.ResolveRelativePathsNode on each loaded include layer at the include project_directory. The PathsPreResolved flag is set on the layer SourceContext so the orchestrator outer pass skips already-resolved scalars via workingDirLookup, preventing the double-join that a generic re-sweep would otherwise introduce. - NodeResolverOptions gains ExcludePaths so the per-include resolution can skip services.x.extends.file: the orchestrator extends pass needs the original relative reference to drive loader.Load against the include-scoped ResourceLoader, matching v2. internal/node.SourceContext gains PathsPreResolved; intrinsic to the include / orchestrator handshake described above. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…Keys checkStringKeys walks the parsed yaml.Node tree depth-first and reports the first non-string mapping key it encounters with the v2-compatible path syntax (top level, services, networks.default.ipam.config[0], services.dict-env.environment, ...). Runs after NormalizeAliases so the merge-key marker (<<) used by YAML anchors is already folded out of the tree by the time we walk for the diagnostic. This closes TestNonStringKeys in the cutover dry-run. The check is intentionally not gated on SkipValidation: a non-string mapping key is a structural defect downstream code cannot recover from, so it stays a hard failure regardless of caller validation preferences. After this commit only one fixture diverges from v2 under cutover: TestIncludeRelative, where v2 keeps include WorkingDir relative so filepath.Join collapses ./ to . in volume short forms. The v3 plan explicitly prefers absolute include WorkingDirs (per-scalar resolution falls back to the layer working dir without an extra rebasing step), so this divergence is by design; the differential note in the next follow-up will record it as an intentional v3 normalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Add the discriminant test that gates the v3 refactoring described in plan.md: - TestInclude_EnvFile_ProvidesContextToServiceEnvFile asserts that variables provided by include.env_file are available when interpolating the content of an env_file declared inside the included service. Today this fails: WithServicesEnvironmentResolved cannot reach the include's env (limitation 3 in plan.md). Will turn green at the end of Phase 7. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 2, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Short-form services.*.volumes (./host:/target) is decoded into the
canonical mapping {type: bind, source: ./host, target: /target} by
transform.CanonicalNode, but the pre-canonical paths sweep cannot
absolutize the source because the value is still a scalar at that
point and absVolumeMount only handles the long-form mapping. The
result was Source = "./host" (unresolved), which broke
TestConvertWithEnvVar on Windows where the test then expects the
COMPOSE_CONVERT_WINDOWS_PATHS conversion to operate on the resolved
absolute path.
Add resolveServiceVolumeSources, run after CanonicalNode +
SetDefaultValues. For each canonical bind volume whose source still
carries the relative dot indicator that format.ParseVolume preserved
from the short form, join the source with the service recorded
WorkingDir (via the same name-keyed serviceContexts that
resolveDefaultBuildContext already uses, so an included service picks
up the include project_directory). Sources that are already absolute
(Unix or Windows-style) and sources that were declared in long form
and pre-absolutized are skipped.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Now that LoadV3 owns every load entry point and ModelToProject has been unexported and re-routed through nodeToProject, the v2 chain has no remaining caller. Delete it wholesale rather than carry it behind //nolint:unused: - loader/loader.go: load, loadYamlModel, loadYamlFile, processExtensions + userDefinedKeys, Transform + inlineExtensions, convertToStringKeysRecursive + formatInvalidKeyError. The unused imports they pulled in (bytes, io, strconv, override, paths, schema, transform, tree, validation) come off too. - loader/include.go, loader/extends.go, loader/environment.go, loader/fix.go, loader/omitEmpty.go: each was reachable only from the deleted v2 chain. - loader/loader_yaml_test.go: relied on loadYamlModel directly, no surviving v3 equivalent (the v3 pipeline is exercised end-to-end through the LoadWithContext tests). - loader/omitEmpty_test.go: covered the deleted map-based OmitEmpty. omitEmptyNode keeps the omitempty patterns table inline so the node pipeline no longer depends on the deleted map helpers. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
With the v2 map pipeline deleted, LoadV3 / loadV3 / TestLoadV3_* markers no longer disambiguate against anything. Rename for clarity: - File renames: load_v3.go -> load.go, load_v3_test.go -> load_test.go, v3_reference_test.go -> reference_test.go. - Identifier renames: LoadV3 -> Load (then unexported as load since LoadWithContext / LoadModelWithContext are the public entry points), loadV3 -> load, ensureLoadV3Options -> ensureLoadOptions, tagsForV3Casts -> tagsForCasts, loadV3Map -> loadMap, TestLoadV3_* -> TestLoad_*, v3Config -> loadConfig. - The public LoadV3 wrapper around the pointer-taking variant is removed (callers go through LoadWithContext / LoadModelWithContext). - Pass over comments to drop the "v3 pipeline" / "v3 fix for the v2 limitation" decorations that now read as time capsules. Comparisons to v2 stay where they explain behavior parity. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both reference tests cover include / env_file scoping and now belong next to the rest of the include tests rather than in their own file. Drops the now-empty reference_test.go. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers the three invariants (origins preserved, lazy resolution, single canonical merged tree), the core types (Layer, SourceContext, origins map), the 15-step orchestrator in load.go, the two projection helpers (nodeToProject vs nodeToModel), the extension points (ResourceLoader, KnownExtensions, Listeners) and a file map. Aimed at contributors who want to add a transform, debug a merge result or pick the right pipeline phase for a new rule. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Companion to Architecture.md, focused on the per-scalar lazy interpolation model. Covers the substitution syntax, how SourceContext.Environment is composed (root cd.Environment -> COMPOSE_PROJECT_NAME -> include env_file -> implicit .env -> extends), the worked example of an include with env_file, the WithServicesEnvironmentResolved path that consults Project.EnvFileScopes, the secrets/configs environment: shorthand, type casts via tagsForCasts(), strict vs lenient mode and a list of gotchas. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pipeline already preserves enough provenance information to point at the offending YAML node, but every validation error went out as a bare "path: cause" string. Wire it through: - errdefs.Diagnostic wraps a Cause with File, Line, Column and the dotted compose Path; Error() renders as "file:line:col: path: cause" with each segment elided when missing. Unwrap exposes Cause so errors.Is / errors.As keep working. - validation.Error pairs the offending *yaml.Node and tree.Path with the underlying cause. ValidateNode returns it (renamed from ValidationError to avoid stutter against the package name). - loader.diagnoseValidation looks up the node origin in the per-scalar origins side-table for the File and pulls Line / Column straight off the *yaml.Node. CanonicalNode invalidates pointer identity, so a pre-canonical buildPathPositions snapshot is consulted as a fallback for the same path. firstConfigFile is the last-resort file when nothing else knows. TestDiagnostic_ValidateNodeIncludesFileLineColumn covers a secrets entry that declares both `file:` and `environment:` (mutually exclusive); the failure now surfaces with the exact compose.yaml line and column the user wrote. Schema validation, interpolation strict mode and other error sites still return bare errors; the same Diagnostic plumbing will pick them up in follow-up commits without further plumbing in errdefs. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…ostics
Extend Phase E coverage from the compose-rule validator to the
remaining user-facing error sites:
- schema.Validate now returns a *schema.Error exposing the dotted
compose Path of the offending value. validateAndStripVersion looks
that path up in the pre-canonical positions snapshot and surfaces an
errdefs.Diagnostic with the file, line, column and path the user
wrote, instead of the bare "validating <file>: ..." prefix.
- interpolation.InterpolateNode wraps each failing scalar in a
*interpolation.Error carrying the offending *yaml.Node. The loader
converts it to a Diagnostic using the origins side-table, so a
strict-mode `${VAR:?msg}` failure now points at the exact source
line and column.
- The include cycle detector emits a Diagnostic prefixed with the file
whose include directive closes the cycle, keeping the v2-compatible
"include cycle detected" body for downstream string matches.
- applyServiceExtendsNode does the same for "cannot extend service Q in
F: service Q not found", pulling the line / column from the
extends node on the derived service.
TestInvalidProjectNameType absorbs the new prefix (the test asserted
the legacy "validating filename0.yml: ..." string).
TestLoadWithIncludeCycle relaxes its HasPrefix check to Contains since
the diagnostic now prepends the offending file. Three new tests in
loader/diagnostics_test.go cover the schema, validation and
interpolation paths end to end.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TestLoadWithRemoteResources expects an extended service whose base file declares `volumes: [.:/foo]` to land in the project with the host portion absolutized against the base file's directory. resolveExtendedServicePaths only walks canonical long-form mappings, so the short form scalar reached the outer pipeline unresolved and format.ParseVolume treated it as a named volume. Add resolveShortFormVolumeSources alongside the existing resolveExtendedServicePaths call. It walks every services.*.volumes.* scalar in the merged service body, splits the src:dst entry, and joins the host portion with the extended file's absolute WorkingDir when it starts with "." or "~". The absolute prefix flips format.ParseVolume isFilePath check into bind, so the canonical pass produces a long-form ServiceVolumeBind with the resolved Source instead of a stale named volume. Drop the t.Skip on TestLoadWithRemoteResources -- it now passes. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lkers Both Canonical and Normalize used to decode the merged tree into map[string]any, run the legacy v2 logic, and re-encode the result back into a fresh yaml.Node tree. The round-trip zeroed Line / Column on every node, which is what forced load.go to keep a buildPathPositions snapshot as a fallback for diagnostics. Replace the bridges with node-level walkers: - transform.CanonicalNode now recurses through the tree and only invokes a transformer when the current tree.Path matches a registered pattern. The decode + encode round-trip is scoped to the smallest matching subtree, so every ancestor and sibling node keeps its original Line / Column. The recursion inside the transformer output makes nested patterns (e.g. services.* then services.*.ports) still fire on the rewritten shape. - loader.NormalizeNode is rewritten as a stack of per-section helpers (normalizeNetworksNode, normalizeServicesNode, setNameFromKeyNode + per-service handlers for build defaults, pull_policy, environment resolution, depends_on derivation and volume target cleanup). The root mapping pointer and every top- level section pointer stay stable across normalize. - loader.Normalize becomes a thin wrapper around NormalizeNode for the map[string]any consumers that still walk the legacy shape (existing normalize_test cases). The bulk of normalize.go is deleted. TestDiagnostic_ValidationKeepsPositionAcrossCanonical confirms the new walker preserves Line / Column for a configs.bad entry whose validation fires after CanonicalNode has run. buildPathPositions stays as defense in depth: it now catches only the small number of nodes that the per-pattern transformers rebuild (volumes mounts, build mappings, ...) -- top-level structural errors hit straight through the origins map. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
setNameFromKeyNode builds the implicit project_resource name for networks / volumes / configs / secrets entries by reading the top-level name key out of the tree. The v3 pipeline never wrote it: v2 load used to set dict[name] = opts.projectName just before calling Normalize, but the equivalent line was dropped when the orchestrator moved to the v3 path. setNameFromKeyNode therefore received an empty projectName, and downstream consumers (docker compose) ended up with resource names that started with a literal percent-bang-s-nil or just with an underscore. Docker rejected the latter on volume create because the leading underscore is not a valid local volume name character. The fallout swept across the e2e suite (TestLocalComposeVolume, TestNetworks, TestIPAMConfig, TestPublishChecks local_include, TestConfig and others) on every CI run since the LoadV3 cutover. Restore the v2 contract by writing opts.projectName into the tree before NormalizeNode runs. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The project name has been stamped onto the merged tree by load() just before NormalizeNode, so the regular Decode below picks it up via the `name:` field. The earlier deleteMappingKey + struct field pre-set was a workaround for the pre-stamp era where opts.projectName lived only on the Options struct and would have been silently overridden by whatever the YAML happened to declare. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Round out Phase E so every user-facing failure surfaces as
*errdefs.Diagnostic with file / line / column:
- CollectIncludeLayers wraps the "`include` must be a list" failure
with the offending include node position.
- collectOneInclude wraps the readIncludeEntry error via a new
diagnoseAt helper that preserves an existing Diagnostic and
otherwise builds one from (file, node, path).
- applyServiceExtendsNode wraps the three remaining bare errors
("services.NAME must be a mapping", parseExtendsRef failure,
"cannot extend service in F: no services section") with the
position of the offending service / extends node.
Four new TestDiagnostic_* cases (include-cycle, include-must-be-a-
list, extends-service-not-found, extends-missing-service) cover the
new wrapping end to end. TestExtendsWihtMissingService updates its
expected error string to match the new prefix.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v2 ApplyInclude fired an "include" event with the include path list
and parent working dir on every include directive it processed.
docker compose registers a metrics listener that increments
CountIncludesLocal when it sees this event with a non-remote path,
and `docker compose publish` refuses to publish a project whose
CountIncludesLocal is greater than zero (the publish target is an OCI
artifact and a local include is not portable).
CollectIncludeLayers / collectOneInclude never re-emitted that event,
so CountIncludesLocal stayed at zero, publish accepted compose files
with local includes, and TestPublishChecks/refuse_to_publish_with_local_include
landed in the e2e failure list since the LoadV3 cutover.
Restore the v2 contract by calling opts.ProcessEvent("include", ...)
with the same {path, workingdir} payload right after readIncludeEntry
in collectOneInclude.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ndeloof
added a commit
to ndeloof/compose
that referenced
this pull request
Jun 3, 2026
Bump the compose-go import path from v2 to v3 to exercise the upcoming major release of compose-spec/compose-go on the Docker Compose codebase. The v3 series replaces the map[string]any-based parser with a yaml.Node-centric pipeline that preserves per-node source context, enables lazy interpolation across include boundaries, and produces file:line:column diagnostics. This PR is a draft compatibility check, not a merge candidate: the replace directive pins compose-go to the v3 branch on the ndeloof fork until the v3 module is tagged upstream. Outcome of this run: - go build ./... succeeds without source changes beyond the /v2 to /v3 import path bump (70 files, mechanical sed). - Unit tests in pkg/compose, pkg/api, pkg/remote, pkg/utils, internal/* all pass. - e2e and watch failures observed during the dry-run are environmental (require running docker daemon, filesystem timing) and unrelated to the compose-go bump. Reference compose-go PR with full roadmap and per-commit changes: compose-spec/compose-go#882 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Covers the breaking changes a downstream consumer might trip over when upgrading from compose-go v2 to v3: - module path (v2 -> v3) - removed APIs (loader.Transform, ModelToProject, ApplyInclude, ApplyExtends, OmitEmpty, ResolveEnvironment, every DecodeMapstructure method) - removed go-viper/mapstructure/v2 dependency - explicit yaml tags now required on WeightDevice / ThrottleDevice - new errdefs.Diagnostic error type and error-format change - behavioral changes: lazy per-scalar interpolation, per-include path resolution, Project.EnvFileScopes side-table, FileMode parsing precedence Closes the documentation set with Architecture.md and Interpolation.md. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The loader has been tracking per-path source positions on a private
side-table since the validation diagnostics landed; this commit exposes
that snapshot to consumers as a public Project.Sources field.
API additions:
- types.Location {File, Line, Column} carries one source position.
- types.Sources = map[string]Location is the path -> Location table.
- Project.Sources holds the table, with yaml:"-" / json:"-" tags so
the project shape stays unchanged for callers that did not opt in.
- Options.Diagnostics bool is the opt-in flag.
- loader.WithDiagnostics is the option function that turns it on.
Wiring:
- load() stashes the buildPathPositions snapshot onto opts.pathPositions
(a new unexported field) when Diagnostics is true.
- nodeToProject reads opts.pathPositions back and populates
Project.Sources before returning.
- Project.deepCopy carries Sources over alongside EnvFileScopes so
chained WithProfiles / WithSelectedServices / ... calls keep the
snapshot.
Two new TestDiagnostic_ProjectSources tests cover the opt-in
(populated map keyed by dotted path, with file / line / column matching
the source) and the default-off (map stays nil so the project shape
is identical for legacy callers).
docs/Migration.md upgrades its placeholder "planned" entry to the
delivered API with a usage snippet.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new test artifacts that wrap up the loose ends from the phase F cleanup checklist: - types/reflect_test.go walks every exported struct type reachable from Project (plus IncludeConfig and ExtendsConfig) and asserts that each exported field carries a yaml tag, unless the type has an UnmarshalYAML implementation. The reflect walk caught one real typo on SSHKey.Path (path:"path,omitempty" instead of yaml:"path,omitempty") and two missing tags on DiscreteGenericResource.Kind / Value; Location gains yaml tags too so it round-trips both ways. ConfigDetails / ConfigFile remain out of scope: they are caller-input shapes, not decoded by the loader. - override/node_fuzz_test.go feeds MergeNode arbitrary pairs of valid YAML mapping roots and checks that the function terminates and does not panic on any input the parser accepts. internal/node/aliases_fuzz_test.go does the same for NormalizeAliases (the defaultMaxAliasNodes cap is the production defense; the fuzz target validates the cap is honored across the input space). interpolation/node_fuzz_test.go covers InterpolateNode. - loader/load_bench_test.go establishes the v3 baseline for per-load cost on a one-service project (~10 ms / 6 MB / 81 K allocs on an M1 Pro), a fifty-service project sharing a YAML anchor (~19 ms / 17 MB / 214 K allocs) and the small project with WithDiagnostics on (no measurable overhead). Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Draft v3 refactor step introducing a *yaml.Node-centric pipeline to preserve YAML node identity/positions end-to-end (enabling per-scalar include scoping, per-include working directories, and richer file:line:column diagnostics), while progressively replacing the legacy map[string]any + mapstructure decoding path.
Changes:
- Add node-native traversal/transform/validation/interpolation building blocks (
internal/node,transform.CanonicalNode,validation.ValidateNode,interpolation.InterpolateNode). - Replace multiple mapstructure-based decoders/encoders with yaml.v4-native
UnmarshalYAML/ node encode+decode round-trips; remove mapstructure dependency. - Introduce diagnostic/source-location plumbing (
errdefs.Diagnostic,types.Sources, schema error path helpers) and include/env-file scope metadata (Project.EnvFileScopes), plus extensive tests/fixtures/benchmarks.
Reviewed changes
Copilot reviewed 92 out of 93 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| validation/node.go | Node-based validation walker + typed validation error wrapper. |
| validation/node_volume.go | Node-based volume validation helpers. |
| validation/node_test.go | Unit tests for ValidateNode checks. |
| types/yaml_helpers.go | Shared yaml.v4 helpers (Document unwrap + scalar coercion). |
| types/types.go | Switch key types to yaml.v4-native decoding (e.g., FileMode/Ulimits/Secret/Config decode helpers, tags). |
| types/stringOrList.go | Replace mapstructure decoding with UnmarshalYAML for string/list types. |
| types/ssh.go | YAML tag fix + yaml.v4 UnmarshalYAML for SSH config. |
| types/services.go | yaml.v4 UnmarshalYAML that injects service name into ServiceConfig. |
| types/reflect_test.go | Reflection guard ensuring exported struct fields have yaml tags or custom decode. |
| types/project.go | Add EnvFileScopes + Sources side tables; use scopes in env_file interpolation; copy metadata in deepCopy. |
| types/options.go | yaml.v4 UnmarshalYAML for Options/MultiOptions. |
| types/mapping.go | yaml.v4 UnmarshalYAML for Mapping/MappingWithEquals; preserve key vs key= distinctions. |
| types/labels.go | yaml.v4 UnmarshalYAML for Labels. |
| types/labels_test.go | Update labels tests to yaml.Unmarshal-based coverage. |
| types/hostList.go | yaml.v4 UnmarshalYAML for extra_hosts (mapping or list forms). |
| types/healthcheck.go | yaml.v4 UnmarshalYAML for healthcheck.test shorthand/list forms. |
| types/duration.go | yaml.v4 UnmarshalYAML for Duration. |
| types/diagnostics.go | New Location/Sources types for opt-in source position snapshots. |
| types/device.go | yaml.v4 UnmarshalYAML for DeviceCount (“all” or number). |
| types/cpus.go | yaml.v4 UnmarshalYAML for NanoCPUs. |
| types/config.go | ConfigFile supports pre-parsed Node; Extensions.Get now yaml round-trips to decode. |
| types/command.go | yaml.v4 UnmarshalYAML for ShellCommand (string or argv list). |
| types/bytes.go | Remove mapstructure decoder hook for UnitBytes (keep UnmarshalYAML). |
| transform/volume.go | Normalize short-form source ./ → . for v2/v3 parity. |
| transform/ports.go | Replace mapstructure-based struct→map encoder with yaml node encode+decode. |
| transform/ports_test.go | Update expectations for port target numeric type post-encoding. |
| transform/node.go | New node-walker canonicalization (scoped decode/encode to preserve positions). |
| transform/node_test.go | Unit tests for CanonicalNode behavior on nodes. |
| schema/schema.go | Schema validation error wrapper now exposes dotted compose path. |
| paths/node_test.go | Tests for node-based relative path resolution, incl. per-scalar working dir behavior. |
| override/uncity_node.go | Node-based EnforceUnicity implementation + indexers. |
| override/uncity_node_test.go | Unit tests for EnforceUnicityNode semantics. |
| override/node_fuzz_test.go | Fuzz test for MergeNode termination/panic safety. |
| loader/testdata/include/secret_env/sub/compose.yaml | New include fixture for secret env scoping. |
| loader/testdata/include/secret_env/secret.env | New env file fixture for secret env scoping. |
| loader/testdata/include/secret_env/compose.yaml | New include fixture root file for secret env scoping. |
| loader/testdata/include/env_file/sub/local.env | New include fixture env file. |
| loader/testdata/include/env_file/sub/extra.env | New include fixture env file (references variable). |
| loader/testdata/include/env_file/sub/compose.yaml | New include fixture compose file. |
| loader/testdata/include/env_file/override.env | New fixture for override env_file interpolation behavior. |
| loader/testdata/include/env_file/compose.yaml | New include fixture root file for env_file scoping. |
| loader/resolve_environment_node.go | Node-based environment resolution + secret/config content capture/apply helpers. |
| loader/resolve_environment_node_test.go | Tests for per-scalar environment resolution behavior. |
| loader/reset_test.go | Update visit cap constant source to internal/node default. |
| loader/omitEmpty.go | Remove legacy map-based omit-empty pass. |
| loader/omitEmpty_test.go | Remove omit-empty unit test. |
| loader/normalize.go | Normalize(map) becomes wrapper around NormalizeNode via yaml node round-trip. |
| loader/normalize_node_test.go | New tests for NormalizeNode behavior. |
| loader/mapstructure.go | Remove legacy mapstructure decode hook machinery. |
| loader/mapstructure_test.go | Remove mapstructure decoding test. |
| loader/loader_yaml_test.go | Remove legacy YAML model load tests. |
| loader/loader_test.go | Adjust assertions for new diagnostic error formatting. |
| loader/load_test.go | New tests for node-based load pipeline behaviors (merge, include scoping, reset, paths). |
| loader/load_layer.go | New per-file parse stage producing node layers; alias/reset handling; string-key validation. |
| loader/load_layer_test.go | Tests for LoadLayer parsing, alias/merge unfolding, reset path capture, caps. |
| loader/load_include_test.go | Tests for CollectIncludeLayers behavior (env_file scoping, project_directory). |
| loader/load_extends_test.go | Tests for ApplyExtendsToLayer behavior on nodes. |
| loader/load_bench_test.go | Benchmarks for loader overhead + diagnostics cost. |
| loader/include.go | Remove legacy map-based include implementation. |
| loader/include_test.go | Add new include scoping assertions (env_file context for service env_file + secret env resolution). |
| loader/fix.go | Remove legacy empty-slice workaround helper. |
| loader/extends.go | Remove legacy map-based extends implementation. |
| loader/extends_test.go | Update assertion for new diagnostic formatting. |
| loader/environment.go | Remove legacy map-based environment resolution helpers. |
| interpolation/node.go | New per-scalar (lazy) node-based interpolation with optional tag rewriting. |
| interpolation/node_test.go | Unit tests for InterpolateNode behavior (lazy lookup, tag rewrite, style preservation). |
| interpolation/node_fuzz_test.go | Fuzz test to ensure interpolation terminates/panic-safety. |
| internal/node/walk.go | New yaml.Node walker producing compose tree.Path values (document unwrap + alias follow). |
| internal/node/walk_test.go | Unit tests for walker traversal semantics and alias-cycle safety. |
| internal/node/reset_test.go | Tests for reset/override node resolution behavior + caps. |
| internal/node/layer.go | Layer + SourceContext data model for per-node context (env/working-dir/parent chain). |
| internal/node/apply_reset.go | ApplyResetPaths on node trees. |
| internal/node/aliases.go | Alias unfolding + merge-key folding with cycle detection and expansion cap. |
| internal/node/aliases_test.go | Unit tests for alias normalization semantics and diagnostics preservation. |
| internal/node/aliases_fuzz_test.go | Fuzz test for alias normalization cap/cycle termination. |
| go.sum | Remove mapstructure checksums. |
| go.mod | Drop mapstructure dependency. |
| errdefs/diagnostic.go | New diagnostic wrapper error type (file:line:col:path prefix + Unwrap). |
| docs/Migration.md | Migration guide covering API removals, new diagnostics, and behavior changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Seven review comments on compose-spec#882 surfaced real issues. Fix each one and add a dedicated test next to the functional changes: - internal/node/apply_reset.go: drop the `i` + `_ = i` dead-code fragment in the sequence walker. - types/options.go: Options.UnmarshalYAML and MultiOptions.UnmarshalYAML used to silently turn a non-scalar value (or nested non-scalar sequence entry) into "" via scalarToString. Reject the offending shape instead so a typo like `key: [a, b]` or `key: [[a]]` fails fast. Three new tests (TestOptions_UnmarshalYAML_RejectsNonScalarValue, TestOptions_UnmarshalYAML_RejectsMappingValue, TestMultiOptions_UnmarshalYAML_RejectsNonScalarSequenceEntry). - types/ssh.go and types/services.go: add the missing unwrapDocument call so calling yaml.Unmarshal directly into a SSHConfig or Services value no longer trips the "expected mapping" guard. Two new tests (TestSSHConfig_UnmarshalYAML_TopLevelDocument, TestServices_UnmarshalYAML_TopLevelDocument). - types/types.go: rewrite the FileMode doc comment to match the actual behavior. Octal is tried first, decimal is the fallback for `mode: 288`-shaped inputs the yaml round-trip produces from an octal literal. Values valid in both bases (e.g. "755") keep the octal reading because every existing fixture depends on it. New TestFileMode_UnmarshalYAML_OctalFirstThenDecimal documents the contract. - validation/node_volume.go: the previous "expected volume" error printed n.Value, which is empty for sequence and mapping nodes; format the offending node's kind ("sequence" / "mapping") via a local kindName helper instead. New TestCheckVolumeNode_NonMapping ErrorIncludesKind. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v3 refactor: replacing the v2
map[string]any-based parsing pipeline with one that preserves*yaml.Nodetrees end-to-end. This unlocks lazy interpolation across include boundaries (each scalar interpolated in its ownSourceContext), per-include working-directory path resolution, file:line:column diagnostics in every error, and an opt-inProject.Sourcesmap for tooling.This PR is reviewed commit by commit; each commit is a self-contained PR-like unit.
Pipeline (delivered)
Architectural decisions (delivered)
yaml.v4native (UnmarshalYAMLon every type)*errdefs.Diagnostic+ opt-inProject.Sources!reseton sequence by indexloader.Transformloader.ModelToProjectnodeToProjectProject.Sourcesvisibilityyaml:"-" json:"-"), opt-in viaWithDiagnosticsmapstructuredependencygo.modCommit-by-commit roadmap
Phase A — Fondations (
internal/node)internal/node: introduce Layer, SourceContext and tree walker(0cbf231)internal/node: extract ResolveResetOverride from loader(aa770ba)internal/node: add NormalizeAliases (unfold + merge-key folding)(b24f7c5)Phase B — Pipeline Node-native
override: add MergeNode operating on yaml.Node trees(d0892d8)override: add EnforceUnicityNode for yaml.Node sequences(187b004)interpolation: add InterpolateNode with per-scalar lazy lookup(26e763e)transform: add CanonicalNode bridging to map-based Canonical(41d2fd8) — bridge replaced in (4227120) by a node-native walkerpaths: add ResolveRelativePathsNode with per-scalar WorkingDir(f06055e)validation,loader: add ValidateNode and NormalizeNode(6141cc2) — bridges replaced in (4227120) by node-native walkersPhase C — Orchestrateur
loader,types: add LoadLayer producing yaml.Node-based Layer(660fb64)loader: add CollectIncludeLayers producing child Layers(e8c6951)loader: add ApplyExtendsToLayer for yaml.Node service trees(4e3552f)loader,internal/node: add LoadV3 orchestrator and ApplyResetPaths(d90367b)loader: extract projectName and enforce non-empty rule in LoadV3(a8eaaa1)loader: add ResolveEnvironmentNode for lazy bare-key environment(93d1127)Phase D — UnmarshalYAML on types + Project projection
types: add UnmarshalYAML on scalar-or-list types(c137757)types: add UnmarshalYAML on map-or-list types(915c87e)types: add UnmarshalYAML on scalar and option types(269307d)types: add UnmarshalYAML for SSHConfig, Services, Secret/ConfigObjConfig(d122a20)transform: replace mapstructure encode with yaml.Node round-trip(39564a2)loader: rewrite Transform on yaml.v4 instead of mapstructure(3e9a1af)deps: drop mapstructure dependency(79800e8)loader: LoadV3 returns *yaml.Node, unexport projection bridge(268c5b9)Differential parity vs v2
loader,paths: tighten LoadV3 parity with v2 and add differential suite(a592870)loader: widen differential coverage and align LoadV3 with v2 semantics(c3f168b)loader,paths: close more LoadV3 vs v2 gaps surfaced by cutover dry-runs(bdea96c)loader,paths: shore up extends Listener parity and refine defaults pass(4c79225)loader: align include projectDir with v2 relative form(8450bb6)loader,paths,internal/node: close near-total parity with v2 fixture suite(36304ac)loader: surface path-aware non-string key diagnostic from checkStringKeys(5fa31b9)loader: turn discriminant test green via lazy env_file scoping cutover(cf2f579)loader: resolve secret/config environment in the declaring layer scope(514f33c)loader: fix include cycle, projectName propagation, chained extends WD(da8f6cb)node: cap alias expansion size to prevent alias-bomb hangs(9d84f1f)transform: cleanSource uses path.Clean to stay POSIX-style on Windows(d526784)transform: cleanSource only rewrites bare "./", not relative subpaths(37c6f10)loader: post-canonical resolve of short-form bind volume sources(b2ee210)loader: resolve extends short-form volume sources to absolute paths(397abad)Phase E — Diagnostics enrichis
errdefs,loader,validation: surface validation errors with file:line:col(108b353)loader,schema,interpolation: wrap schema/interp/cycle errors as diagnostics(a4ddd1d)loader: wrap remaining extends / include error sites as Diagnostic(a8a9bc1)Phase F — Cleanup, docs, finalisation
loader: drop the v2 map-based pipeline(f4f9df9)loader: strip v3 suffix now that the v2 path is gone(53688e9)loader: fold reference_test.go into include_test.go(2b63750)loader: skip TestLoadWithRemoteResources pending extends-clone origins(6d28d08) — removed in (397abad) when the fix landedtransform,loader: port Canonical and Normalize to yaml.Node-native walkers(4227120)loader: stamp projectName into the tree before NormalizeNode(acc3031)loader: drop the now-redundant deleteMappingKey(name) in nodeToProject(8993068)loader: emit "include" Listener event in collectOneInclude(6e0af28) — Docker Compose publish check paritydocs: add Architecture.md describing the loading pipeline(a855c6c)docs: add Interpolation.md covering scope rules(0cc703c)docs: add Migration.md for v2 to v3 upgrade(865bfd6)types,loader: expose per-path source Location via Project.Sources(43c3924)tests: close phase F with reflection, fuzz and benchmark coverage(09519e1)Compatibility validation
Tracked against Docker Compose via docker/compose#13818:
TestWatch/debianon docker standalone oldstable, unrelated to compose-go)._<resource>resource naming bug (%!s(<nil>)_default) and (6e0af28) for the publish local-include detection.Test strategy
loader/testdata/still passes through the new pipeline.testdata/include/env_file/), the per-include secret env scope (testdata/include/secret_env/).loader/diagnostics_test.go): schema, validation, interpolation strict, include cycle, include shape, extends not found, extends missing service, validation position survives canonical,Project.Sourcesopt-in.types/blocks future contributors from adding an untagged exported field to a yaml-decoded struct.override.MergeNode,internal/node.NormalizeAliases,interpolation.InterpolateNode.loader/load_bench_test.go(small / medium / WithDiagnostics).Documentation
docs/Architecture.md— the 15-step pipeline, core types (Layer / SourceContext / origins), extension points, file map.docs/Interpolation.md— substitution syntax, lazy per-scalar principle, scope composition, worked example with includeenv_file.docs/Migration.md— v2 → v3 breaking changes, removed APIs, error format change, behavioral changes, upgrade recipes.🤖 Generated with Claude Code