Allow defaults to satisfy required values

[MuellerSeb]

Closes #52 #49

This merge request separates schema-declared requiredness from the effective
namelist input requirement. A valid operational default now satisfies
required, including recursively covered required components of one-level
derived values.

It also adds narrow mapping-valued defaults for derived types:

• scalar derived properties may define an object default;
• derived array items.default may define a broadcast object fallback;
• array-level lists of derived defaults remain unsupported.

Derived storage is initialized in this order:

component defaults and missing sentinels -> object/items default -> explicit input

The implementation keeps parsed/evaluated namelists sparse, does not add a
touched-state model, and preserves the existing one-level intrinsic-component
limit.

Changes

• Added shared property-requirement analysis that distinguishes:

  • whether a property is declared required;
  • whether defaults cover all required values;
  • whether every component is default-initialized;
  • whether namelist/setter input is still required;
  • which derived required components remain uncovered.

• Allowed intrinsic scalar and array defaults on properties listed in
  required.

• Added structural and value validation for derived object defaults:

  • keys are matched case-insensitively against declared components;
  • duplicate and unknown keys are rejected;
  • values use the component's type, enum, bounds, and string-length checks;
  • use-site object defaults replace inherited object mappings as a whole,
    while their selected values overlay component defaults.

• Allowed derived items.default mappings as broadcast fallbacks for every
  array element while retaining rejection of derived array-level defaults.

• Enforced the conservative derived presence policy:

  • optional derived fields with uncovered required components are rejected;
  • declared-required derived fields may use partial fallback defaults and
    require input only for the uncovered components;
  • derived fields with no uncovered required components are already enough
    set after initialization, including the vacuous no-required-component case.

• Updated schema-aware namelist evaluation:

  • defaults satisfy root and derived-component requiredness;
  • object/item defaults override component defaults;
  • explicit values override every default layer;
  • null values retain the current default or earlier explicit value;
  • absent defaults remain metadata rather than materialized states;
  • fixed-shape derived arrays validate untouched elements in deterministic
    Fortran element order.

• Updated generated Fortran:

  • initialization applies component defaults/sentinels before object/item
    overlays;
  • logical storage accepts an effective object/item default and still rejects
    missing logical initialization;
  • aggregate is_set and is_valid inspect only uncovered required
    components;
  • component is_set continues to report effective leaf presence;
  • setter dummies become optional when defaults satisfy the input requirement;
  • native argument order continues to use declared-required grouping for
    compatibility.

• Tightened runtime intrinsic-array default extent handling:

  • exact property-level lists without repeat/pad require an equal total
    extent;
  • repeat and pad defaults retain their minimum source-size rule and allow
    larger extents;
  • scalar items.default continues to broadcast to any positive extent.

• Updated f2py and generated Python wrappers:

  • Python parameters are regrouped by effective input requiredness;
  • newly optional low-level values receive explicit presence flags;
  • derived mappings remain partial and mapping-only, with uncovered required
    leaves reported by native validation.

• Updated Markdown generation to show both Declared required and
  Input required for fields and derived components, including component,
  object, and item default sources.

• Updated filled template precedence for derived components to:

  explicit override -> component example -> object/items default ->
  component default -> enum/type fallback
  

• Updated minimal templates to omit a derived field only when all components
  are default-initialized, not merely when required components are covered.

• Documented the operational-default policy and its intentional difference
  from annotation-only JSON Schema defaults.

• Extended and regenerated the committed derived-types example:

  • scalar period uses a partial object fallback;
  • periods uses a complete broadcast item fallback and becomes optional at
    the setter boundary despite remaining declared required;
  • generated Fortran, Markdown, f2py, and Python artifacts reflect the new
    policy.

Behavior Notes

• required remains meaningful schema metadata even when a default currently
  satisfies it. Removing that default in a later schema revision restores the
  input requirement.
• Object defaults are fallback mappings, not presence markers. An empty mapping
  is valid only when component defaults already cover every required component.
• A partial object default may cover some required components of a
  declared-required derived field; explicit input must cover the remainder.
• Optional derived values with uncovered required components remain unsupported
  because implementing "required if touched" needs per-instance touched state.
• A declared-required derived value with no required components is legal and
  vacuously enough set after configuration.
• Aggregate derived presence intentionally ignores optional non-defaulted
  components. Their component-level presence queries still report not set.
• Native Fortran setter argument order is preserved. Python-facing signatures
  place effective required parameters before optional parameters, so positional
  Python ordering can change when a default removes an input requirement.
• Nested derived values, component arrays, derived flexible-tail arrays,
  per-element derived array defaults, and touched-state tracking remain out of
  scope.

Tests

Added focused coverage for:

• required intrinsic scalars and arrays with complete defaults;
• exact, repeat/pad, and item-broadcast runtime extent policies;
• sparse default accounting and explicit/null override behavior;
• required derived components covered by component, object, or item defaults;
• partial fallbacks with uncovered required components;
• vacuously complete declared-required derived values;
• rejection of optional incomplete derived values;
• object-default key and component-constraint validation;
• inherited versus use-site-replaced object mappings;
• deterministic missing paths for untouched derived-array elements;
• generated Fortran initialization, setter optionality, and is_set/is_valid
  behavior;
• f2py presence flags and Python parameter regrouping;
• Markdown requiredness columns and default-source labels;
• component and buffer template precedence;
• minimal-template omission based on full default initialization;
• end-to-end generated example behavior with partial and omitted nested
  mappings.

[Copilot]

Pull request overview

This PR updates nml-tools’ requiredness model so that operational defaults can satisfy schema-declared required (including for supported derived-type defaults), and propagates that distinction through validation, namelist evaluation, Fortran/Python wrapper generation, and Markdown docs.

Changes:

• Introduces shared “declared required vs input required” analysis and uses it across validation/evaluation/codegen/docs.
• Adds supported mapping-valued defaults for derived scalars (default) and derived arrays (items.default broadcast), including constraint validation and deterministic missing-path checks.
• Updates generated Fortran, f2py, Python wrappers, templates, and documentation to reflect effective input requiredness and default precedence.

Reviewed changes

Copilot reviewed 19 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/nml_tools/validate.py	Adds PropertyRequirement + requirement analysis; validates derived object defaults and default coverage policy.
src/nml_tools/schema.py	Allows/validates derived-type object default mappings (keys/duplicates/unknowns).
src/nml_tools/_namelist_eval.py	Uses effective input-requiredness; applies derived object/item defaults as effective component defaults during evaluation.
src/nml_tools/codegen_fortran.py	Shifts requiredness checks and is_set/is_valid semantics to uncovered required components; updates runtime extent checks and default layering.
src/nml_tools/codegen_f2py.py	Regroups f2py args by effective requiredness and introduces presence flags for newly-optional values.
src/nml_tools/codegen_template.py	Uses effective requirement analysis to drive minimal/filled template emission and derived fallback defaults.
src/nml_tools/codegen_markdown.py	Documents both “Declared required” and “Input required”, including derived component default sources.
README.md	Documents the operational-default policy and derived default/requiredness behavior.
tests/test_validate.py	Adds validation/evaluation coverage for required defaults, derived object defaults, item defaults, and missing-path behavior.
tests/test_schema.py	Adds schema resolution tests for use-site vs inherited derived default mappings and default mapping validation.
tests/test_codegen_template.py	Tests template precedence for derived object/item defaults and minimal omission rules.
tests/test_codegen_markdown.py	Tests docs output for declared vs input required and default source labeling.
tests/test_codegen_fortran.py	Tests setter optionality, derived default application, and runtime extent policies in generated Fortran.
tests/test_codegen_f2py.py	Tests f2py argument regrouping and presence-flag emission for newly-optional values.
examples/04_derived_types/schema/run.yml	Updates example schema to use derived object and item broadcast defaults.
examples/04_derived_types/README.md	Updates example narrative and usage to reflect new default/requiredness behavior.
examples/04_derived_types/f2py/generated/f2py_config.f90	Regenerated f2py wrapper showing new presence flags (but currently has an optional-arg call-site issue).
examples/04_derived_types/fortran/generated/nml_run.f90	Regenerated Fortran showing derived default layering and updated is_set/is_valid checks.
examples/04_derived_types/docs/run.md	Regenerated docs with declared/input required columns and derived default source details.
examples/04_derived_types/src/nml_derived_types_example/config_wrappers.py	Regenerated Python wrapper signature/behavior for newly-optional derived array argument.
examples/04_derived_types/tests/test_derived_types_example.py	Extends example end-to-end tests for partial/omitted nested mappings enabled by defaults.

Comments suppressed due to low confidence (1)

src/nml_tools/codegen_f2py.py:519

• For derived fields, outer_has_flag is added when field.requires_input is false, and _derived_bridge_assignments correctly gates allocation/init inside if (has__field) then .... However set_call_arguments.append(f"{field.name}={maybe_name}") is unconditional, so the generated wrapper will still pass an unallocated/undefined maybe__* actual argument to this%set(...), making it present and potentially causing runtime errors and incorrect semantics (it can override defaults / reset prior values). The wrapper generation needs to omit the keyword argument entirely unless the corresponding presence flag is true.

                _derived_bridge_assignments(
                    field.name,
                    maybe_name,
                    leaves,
                    rank,
                    field.requires_input,
                    outer_has_flag,
                    dim_names,
                    init_allocates_array=field.runtime_sized_array,
                )
            )
            set_call_arguments.append(f"{field.name}={maybe_name}")
            continue

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MuellerSeb]

@copilot This is intentional Fortran optional-argument forwarding. maybe__ remains unallocated when the presence flag is false, and the corresponding this%set dummy is optional and non-allocatable. Fortran therefore considers that dummy not present despite the keyword appearing in the call. Optional derived fields and runtime-dimension bridges use the same mechanism, and the compiled example tests exercise it across the CI compiler matrix.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 24 changed files in this pull request and generated 1 comment.