Skip to content

Conversation

@markphelps
Copy link
Contributor

@markphelps markphelps commented Feb 6, 2026

Config Refactor

1. Clear Three-Phase Architecture (Parse → Validate → Complete)

Before (main): Config loading was a monolithic process where parsing, validation, and CUDA resolution were interleaved and hard to follow.

After: Clean separation of concerns:

  • Parse (parse.go) - Only YAML parsing, no side effects
  • Validate (validate.go) - Semantic validation with typed errors
  • Complete (config.go) - CUDA resolution, requirements loading

This makes the code easier to understand, test, and maintain.

2. Typed Error System

Before: Generic error returns with string formatting, hard to programmatically handle different error types.

After: Rich error types in errors.go:

  • ParseError - YAML parsing failures
  • SchemaError - Schema validation issues
  • ValidationError - Semantic validation failures
  • DeprecationWarning - Deprecated field usage
  • CompatibilityError - Version incompatibilities

Callers can use errors.As() to handle specific error types differently.

3. Separation of Raw Config from Completed Config

Before: Single Config struct used for both raw YAML and completed config, with nullable fields mixed with populated fields.

After:

  • configFile (unexported) - Raw YAML with pointer fields to distinguish "not set" from "zero value"
  • Config - Completed config with resolved values

4. Better Deprecation Handling

Before: Deprecation warnings were ad-hoc and mixed with errors.

After: ValidationResult separates errors from warnings:

type ValidationResult struct {
    Errors   []error
    Warnings []DeprecationWarning
}

Can optionally treat deprecations as errors with WithStrictDeprecations().

5. Functional Options Pattern for Validation

Before: Hard-coded validation behavior.

After: Flexible configuration:

ValidateConfigFile(cfg,
    WithProjectDir(dir),
    WithRequirementsFS(fsys),
    WithStrictDeprecations(),
)

6. Bug Fixes

  • Fixed Python version check for concurrency (now checks major version)
  • Fixed path handling for Windows compatibility (filepath.Join vs path.Join)
  • Removed dead code paths

7. Reduced API Surface

  • Unexported internal types (configFile, buildFile, etc.)
  • Unexported internal functions (validateCudaVersion, cudaBaseImageFor)
  • Removed unused code (Example, Version, Stats structs)

8. Better Testability

  • Each phase can be tested independently
  • WithRequirementsFS() allows testing without real filesystem
  • Typed errors make assertions clearer

9. Idiomatic Go

  • Lowercase error messages
  • Proper error wrapping with %w
  • Marker interfaces for error types
  • Functional options pattern
  • Replaces internal slices package with stdlib
  • Uses io.Reader where applicable instead of filenames as args for testability

@markphelps markphelps changed the title refactor(config): implement three-phase Parse → Validate → Complete architecture refactor(config)(wip): implement three-phase Parse → Validate → Complete architecture Feb 6, 2026
@markphelps markphelps force-pushed the config-refactor branch 2 times, most recently from 73957d3 to 7260c9b Compare February 6, 2026 21:23
@markphelps markphelps changed the title refactor(config)(wip): implement three-phase Parse → Validate → Complete architecture refactor(config)(wip): implement two-phase Parse → Validate+Complete architecture Feb 6, 2026
@markphelps markphelps changed the title refactor(config)(wip): implement two-phase Parse → Validate+Complete architecture refactor(config)(wip): implement three-phase Parse → Validate → Complete architecture Feb 6, 2026
…rchitecture

This refactors the config system to use a cleaner three-phase architecture:

1. **Parse**: YAML → ConfigFile (raw representation with pointer fields)
2. **Validate**: Pure validation with no mutations, returns structured errors
3. **Complete**: ConfigFile → Config with defaults and CUDA resolution

- errors.go: Structured error types (ParseError, SchemaError, ValidationError, etc.)
- config_file.go: Raw ConfigFile type mirroring YAML with pointer fields
- parse.go: Parse(), ParseBytes(), ParseReader(), FromYAML() functions
- validate.go: ValidateConfigFile() with functional options pattern
- complete.go: CompleteConfig() for transforming ConfigFile → Config
- schema.go: GenerateSchema() using github.com/invopop/jsonschema
- build_options.go: BuildOptions struct for CLI flags (alternative to globals)

- validator.go: Old schema validation (replaced by validate.go)
- validator_test.go: Old tests (replaced by validate_test.go)

- Load() returns LoadResult with Config, Warnings, and RootDir
- Complete() replaces ValidateAndComplete() for programmatic configs
- FromYAML() is now a test helper that doesn't run completion
- Removed GetConfig(), GetRawConfig(), loadConfigFromFile()

- Renamed JSON files for consistency (*_matrix.json → *_compatibility.json)
- Renamed env_variables.go → env.go
- Updated callers to use new APIs
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
Bug fixes:
- Fix major version check in validateConcurrency using splitPythonVersion
- Use filepath.Join instead of path.Join for OS-specific path handling
- Remove dead code (empty if !found block) in validateFrameworkCompatibility
- Remove unused schemaVersion constant

Dead code removal:
- Remove unused Version, Image, Stats structs from version.go
- Remove unused Example struct from config.go

Unexport internal code:
- ValidateCudaVersion → validateCudaVersion
- CUDABaseImageFor → cudaBaseImageFor
- ConfigFile/BuildFile/RunItemFile/etc → lowercase versions
- EnvironmentVariableDenyList → environmentVariableDenyList
- Use filepath.Join instead of path.Join in load.go for OS-specific paths
- Fix ValidateModelPythonVersion to check major version for concurrency
- Remove unused getter methods from config_file.go (GetPythonVersion,
  GetPythonRequirements, GetCUDA, GetCuDNN, GetMax)
- Standardize error messages to lowercase per Go conventions
- Use %w for error wrapping in findConfigPathInDirectory
markphelps and others added 4 commits February 9, 2026 09:45
- Fix golangci-lint issues: rename shadowed 'max' variable, use type conversion
- Remove duplicate ValidateModelPythonVersion function from config.go
- Remove redundant validation call from build.go (Load already validates)
- Update integration tests to match new ValidationError message format
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
@markphelps markphelps marked this pull request as ready for review February 9, 2026 15:29
@markphelps markphelps requested a review from a team as a code owner February 9, 2026 15:29
tempusfrangit
tempusfrangit previously approved these changes Feb 9, 2026
Copy link
Member

@tempusfrangit tempusfrangit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems pretty reasonable. Please make sure wip is removed from the PR title. I'd also like @michaeldwan 's review on this before we merge.

if len(c.Build.PythonPackages) > 0 {
c.Build.pythonRequirementsContent = reqs
} else if len(c.Build.PythonPackages) > 0 {
// Backwards compatibility: if using deprecated python_packages, populate requirements content
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's time to drop this and error. we're doing a TON of big restructuring and this has been deprecated for a while. We can do this as a separate PR before release. I expect 0.17.0 can include this PR and the future removal of deprecated things.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this sentiment. Let's delete more things!

@markphelps markphelps changed the title refactor(config)(wip): implement three-phase Parse → Validate → Complete architecture refactor(config): implement three-phase Parse → Validate → Complete architecture Feb 9, 2026
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
@tempusfrangit tempusfrangit merged commit 2c2259c into main Feb 9, 2026
81 of 88 checks passed
@tempusfrangit tempusfrangit deleted the config-refactor branch February 9, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants