Skip to content

Add Pydantic-based config validation and type correction#310

Open
MaStr wants to merge 9 commits intomainfrom
claude/pydantic-config-conversion-Qc7UW
Open

Add Pydantic-based config validation and type correction#310
MaStr wants to merge 9 commits intomainfrom
claude/pydantic-config-conversion-Qc7UW

Conversation

@MaStr
Copy link
Owner

@MaStr MaStr commented Mar 17, 2026 •
edited
Loading

Summary

Introduces comprehensive configuration validation using Pydantic models to eliminate scattered type conversion code throughout the codebase and fix Home Assistant addon issues where numeric values arrive as strings.

Key Changes

  • New Pydantic models (src/batcontrol/config_model.py):

    • BatcontrolConfig: Top-level configuration with validation for time_resolution_minutes (15 or 60) and loglevel
    • BatteryControlConfig, InverterConfig, UtilityConfig, MqttConfig, EvccConfig: Specialized config models for each subsystem
    • PvInstallationConfig, ConsumptionForecastConfig: Installation and forecast-specific models
    • validate_config() function: Entry point for validating raw config dicts and returning validated dicts

  • Type coercion at config load time:

    • String-to-int coercion for numeric fields (e.g., MQTT port, time resolution)
    • String-to-float coercion for decimal fields (e.g., VAT, fees, price differences)
    • Semicolon-separated string parsing for list fields (e.g., history_days="-7;-14;-21" → [-7, -14, -21])
    • Legacy field renaming support (max_charge_rate → max_grid_charge_rate)

  • Integration with config loading (src/batcontrol/setup.py):

    • load_config() now calls validate_config() to validate and coerce types before returning the config dict

  • Simplified downstream code:

    • Removed manual type conversions in core.py (time resolution validation)
    • Removed manual string-to-float conversions in dynamictariff.py
    • Removed manual list parsing in consumption.py
    • Removed legacy field handling in inverter.py

  • Comprehensive test coverage (tests/batcontrol/test_config_model.py):

    • 40+ unit tests covering all config models and edge cases
    • Tests for HA addon string coercion scenarios
    • Tests for backward compatibility (legacy field names)
    • Tests for validation errors and constraints

  • Integration tests (tests/batcontrol/test_config_load_integration.py):

    • Tests for load_config() with Pydantic validation
    • Tests for actual YAML file loading and validation

Notable Implementation Details

  • Uses Pydantic's extra='allow' to preserve unknown fields for forward compatibility
  • Implements custom validators for complex parsing (semicolon-separated lists, legacy field renaming)
  • All numeric fields support string input, automatically coerced to proper types
  • Validation happens once at config load time, eliminating runtime type checks throughout the codebase
  • Maintains backward compatibility with existing config files and legacy field names

https://claude.ai/code/session_015rEfxnRN99qtwpBmEZ3GpP

Closes #144

claude added 3 commits March 17, 2026 18:26
@claude
Add Pydantic config validation to eliminate type coercion bugs
ec324fb 
Introduces Pydantic models for all configuration sections that validate
and coerce types at load time, fixing HA addon string-to-int/float
issues (e.g. MQTT/EVCC port, time_resolution_minutes, vat/fees/markup).

- Add config_model.py with typed models for all config sections
- Integrate validate_config() into load_config() in setup.py
- Remove manual type coercion from core.py and dynamictariff.py
- Add 33 new tests covering model validation and type coercion
- Add pydantic>=2.0 dependency

https://claude.ai/code/session_015rEfxnRN99qtwpBmEZ3GpP
@claude
Enhance Pydantic config: history_days parsing, inverter alias, loglev…
ab51044 
…el validation

- Add field_validators for history_days/history_weights semicolon string
  parsing (e.g. "-7;-14;-21" -> [-7, -14, -21]) from HA addon
- Add model_validator for legacy max_charge_rate -> max_grid_charge_rate rename
- Add loglevel validator (must be debug/info/warning/error, normalized to lowercase)
- Remove manual type coercion from consumption.py and inverter.py
- Add 10 new tests (367 total passing)

https://claude.ai/code/session_015rEfxnRN99qtwpBmEZ3GpP
@claude
Add missing cache_ttl to InverterConfig, fix mqtt inverter passthrough
69de7c2 
- Add cache_ttl field to InverterConfig Pydantic model (used by MQTT inverter)
- Fix pre-existing bug: cache_ttl was not passed through in inverter
  factory's MQTT path, always falling back to default 120

https://claude.ai/code/session_015rEfxnRN99qtwpBmEZ3GpP
Copilot AI review requested due to automatic review settings March 17, 2026 18:50
@MaStr MaStr changed the title Add Pydantic-based config validation and type coercion Add Pydantic-based config validation and type correction Mar 17, 2026
@MaStr MaStr added enhancement New feature or request CodeQuality PRs fixing CodeQuality issues labels Mar 17, 2026
@MaStr MaStr added this to the 0.8.0 milestone Mar 17, 2026
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces centralized configuration validation/coercion via new Pydantic models, aiming to handle Home Assistant “numbers as strings” issues at config load time and remove scattered runtime conversions across Batcontrol.

Changes:

  • Added src/batcontrol/config_model.py with Pydantic models and a validate_config() entrypoint.
  • Integrated Pydantic validation into load_config() and removed several downstream coercions/parsers (core time resolution, tariff float casting, HA consumption list parsing, inverter legacy rename).
  • Added unit/integration tests covering validation, coercion, backward compatibility, and load_config() behavior; added pydantic dependency.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tests/batcontrol/test_core.py Updates time-resolution tests to reflect validation happening before Batcontrol init.
tests/batcontrol/test_config_model.py Adds unit tests for Pydantic config models and coercion/validation behavior.
tests/batcontrol/test_config_load_integration.py Adds integration tests for YAML load_config() + validation.
src/batcontrol/setup.py Calls validate_config() during config loading.
src/batcontrol/inverter/inverter.py Removes legacy field/default handling; adds cache_ttl wiring for MQTT inverter.
src/batcontrol/forecastconsumption/consumption.py Removes semicolon-list parsing and int coercion (now expected from validation).
src/batcontrol/dynamictariff/dynamictariff.py Removes float() casting; assumes validated numeric types.
src/batcontrol/core.py Removes time-resolution string coercion/validation; assumes validated config.
src/batcontrol/config_model.py New Pydantic models + validate_config() for centralized validation/coercion.
pyproject.toml Adds pydantic as a dependency.

claude and others added 2 commits March 17, 2026 18:59
@claude
Add Pydantic config migration docs for AI-assisted development
134c125 
Documents what was done, what remains, config model reference,
and HA addon coercion table.

https://claude.ai/code/session_015rEfxnRN99qtwpBmEZ3GpP
@MaStr @claude
Address PR review: exclude_none, required pvinstallations, cache_ttl …
9f82f47 
…default

Fixes all 5 review comments from Copilot:

1. Use model_dump(exclude_none=True) so None optional fields don't appear
   as keys in the output dict. Downstream code checks key presence
   (e.g. 'csv' in config, required_fields checks) and would break with
   None values present.

2. Make pvinstallations required (no default) - Pydantic now raises
   ValidationError if missing, instead of silently defaulting to [].

3. Give cache_ttl a real default (120) instead of Optional[None] to
   prevent None propagating into TTLCache.

4. Reorder load_config() to run validate_config() before the
   pvinstallations check, so Pydantic provides clear error messages.

5. tariff_zone_* None issue resolved by exclude_none=True.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings March 17, 2026 19:13
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a centralized, Pydantic v2–based configuration validation layer so config type coercion/validation happens once at load time (helping address HA addon “numbers as strings” issues) and downstream modules can rely on properly typed config values.

Changes:

  • Introduces src/batcontrol/config_model.py with Pydantic models plus validate_config() and wires it into load_config().
  • Removes scattered runtime type conversions in core.py, dynamictariff.py, forecastconsumption/consumption.py, and legacy mapping in inverter.py.
  • Adds extensive unit + integration tests for config validation/coercion and updates existing tests accordingly.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/batcontrol/config_model.py New Pydantic models + validate_config() entry point to coerce/validate config.
src/batcontrol/setup.py Calls validate_config() during load_config() to centralize validation.
src/batcontrol/core.py Removes time_resolution_minutes coercion/validation, relying on validated config.
src/batcontrol/dynamictariff/dynamictariff.py Removes float() casting for utility numeric fields, relying on validated config.
src/batcontrol/forecastconsumption/consumption.py Removes manual parsing for history_days/history_weights (now intended to be handled by validation).
src/batcontrol/inverter/inverter.py Removes legacy rename/default handling (moved to Pydantic) and passes cache_ttl for MQTT inverter.
tests/batcontrol/test_config_model.py New unit tests for models, coercions, constraints, and validate_config().
tests/batcontrol/test_config_load_integration.py New integration tests verifying load_config() + YAML loading + validation behavior.
tests/batcontrol/test_core.py Updates time-resolution tests to reflect validation occurring before Batcontrol.__init__().
pyproject.toml Adds pydantic>=2.0,<3.0 dependency.
docs/pydantic-config-migration.md Migration notes, design decisions, and follow-ups.

@MaStr @claude
Address PR review round 2: nested HA config parsing, validator mutation
ec85b81 
Fixes 2 valid review comments (3 others were already fixed):

1. Restore history_days/history_weights parsing in consumption factory
   for the nested homeassistant_api dict case. Pydantic only validates
   top-level ConsumptionForecastConfig fields; when values come from
   the nested homeassistant_api dict they bypass Pydantic validation
   and may still be semicolon-separated strings.

2. Copy input dict in InverterConfig model_validator before mutating
   via pop(). Prevents surprising side effects on callers who reuse
   the same dict.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MaStr MaStr requested a review from Copilot March 17, 2026 19:24
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds centralized configuration validation/coercion using Pydantic so config type fixes happen once at load time (addressing Home Assistant addon string-typed numeric values) and downstream modules can rely on normalized types.

Changes:

  • Introduces BatcontrolConfig and subsystem config models + validate_config() for type coercion and validation.
  • Integrates validation into load_config() and removes scattered manual coercions in core/tariff/consumption/inverter code paths.
  • Adds unit + integration tests and migration documentation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/batcontrol/config_model.py New Pydantic models + validate_config() to validate/coerce loaded config.
src/batcontrol/setup.py Calls validate_config() during config load.
src/batcontrol/core.py Removes local time resolution coercion/validation, assumes validated config.
src/batcontrol/dynamictariff/dynamictariff.py Removes float() casts, relies on validated float types.
src/batcontrol/forecastconsumption/consumption.py Simplifies list parsing; keeps fallback parsing for nested HA config.
src/batcontrol/inverter/inverter.py Removes legacy rename/default handling (now in Pydantic); passes cache_ttl.
tests/batcontrol/test_config_model.py New unit tests for models and coercion behavior.
tests/batcontrol/test_config_load_integration.py New integration tests for load_config() + YAML loading.
tests/batcontrol/test_core.py Updates time-resolution test to reflect validated-config expectations.
pyproject.toml Adds pydantic>=2.0,<3.0 runtime dependency.
docs/pydantic-config-migration.md Documents migration status, design decisions, and remaining follow-ups.

@MaStr @claude
Address PR review round 3: Optional vat/fees/markup, nested dict coer…
a045d63 
…cion, lint

Fixes 4 new review comments:

1. Make vat/fees/markup Optional[float]=None in UtilityConfig so
   downstream required_fields checks in dynamictariff.py can detect
   missing config (previously defaults of 0.0 masked misconfigurations).

2. Add float() coercion for cache_ttl_hours, multiplier, and
   annual_consumption in consumption factory for the nested
   homeassistant_api/csv dict paths that bypass Pydantic validation.

3. Remove unused import tempfile from test_config_load_integration.py.

4. Remove unused import BatteryControlExpertConfig from test_config_model.py.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MaStr MaStr requested a review from Copilot March 17, 2026 19:35
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds centralized configuration validation/coercion using Pydantic to address Home Assistant add-on string-typed options and remove scattered runtime type conversions across Batcontrol.

Changes:

  • Introduces src/batcontrol/config_model.py Pydantic v2 models + validate_config() to validate/coerce config at load time.
  • Integrates validation into setup.load_config() and simplifies downstream modules (core.py, dynamictariff.py, inverter.py, forecastconsumption/consumption.py) by removing redundant casts/renames.
  • Adds extensive unit + integration tests for config validation and YAML load behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/batcontrol/config_model.py New Pydantic models + validate_config() that returns a coerced dict (model_dump(exclude_none=True)).
src/batcontrol/setup.py Calls validate_config() during config load before further checks.
src/batcontrol/core.py Removes local time_resolution_minutes coercion/validation and relies on validated config.
src/batcontrol/dynamictariff/dynamictariff.py Removes float casts for utility pricing parameters assuming validated types.
src/batcontrol/inverter/inverter.py Removes legacy rename/default handling (moved to Pydantic) and adds cache_ttl passthrough for MQTT inverter config.
src/batcontrol/forecastconsumption/consumption.py Adjusts parsing/coercion for nested HA config cases and ensures numeric types for annual consumption.
tests/batcontrol/test_config_model.py New unit tests covering model defaults, coercions, validators, and validate_config() output shape.
tests/batcontrol/test_config_load_integration.py New integration tests validating YAML load + Pydantic coercion through load_config().
tests/batcontrol/test_core.py Updates time-resolution test expectations to reflect load-time validation behavior.
pyproject.toml Adds pydantic>=2.0,<3.0 dependency.
docs/pydantic-config-migration.md Adds migration/status documentation for the new validation layer.

@MaStr @claude
Fix docs: correct UtilityConfig types, provider list, and parsing des…
6903b32 
…cription

- vat/fees/markup are Optional[float]=None, not float=0.0
- Remove nonexistent 'entsoe' provider, list actual providers
- Clarify that semicolon parsing was kept for nested HA config case
- Note exclude_none=True in design decisions
- Remove nonexistent sungrow/huawei inverter types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@MaStr MaStr requested a review from Copilot March 17, 2026 20:18
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds centralized, Pydantic v2–based configuration validation/coercion at config load time to address Home Assistant “numbers as strings” issues and reduce scattered runtime type conversions across Batcontrol.

Changes:

  • Introduces BatcontrolConfig and subsystem config models in src/batcontrol/config_model.py, plus validate_config() returning a plain validated dict (exclude_none=True).
  • Integrates validation into setup.load_config() and removes/reduces downstream manual coercions (e.g., core.py, dynamictariff.py, inverter.py, forecastconsumption/consumption.py).
  • Adds extensive unit + integration tests covering coercion, defaults, validation errors, and YAML load behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
tests/batcontrol/test_core.py Updates time-resolution tests to reflect validation happening at load time.
tests/batcontrol/test_config_model.py Adds unit tests for new Pydantic config models and coercion/validation behavior.
tests/batcontrol/test_config_load_integration.py Adds integration tests for load_config() validating/coercing YAML configs.
src/batcontrol/setup.py Calls validate_config() after YAML load and before PV installation checks.
src/batcontrol/inverter/inverter.py Removes legacy/default handling now covered by Pydantic; passes cache_ttl for MQTT inverter.
src/batcontrol/forecastconsumption/consumption.py Adjusts parsing/coercion to handle nested HA config edge cases and numeric strings.
src/batcontrol/dynamictariff/dynamictariff.py Removes float casts assuming config has already been coerced by Pydantic.
src/batcontrol/core.py Removes manual time_resolution_minutes coercion/validation and uses validated value.
src/batcontrol/config_model.py New Pydantic models and validate_config() entry point.
pyproject.toml Adds pydantic>=2.0,<3.0 dependency.
docs/pydantic-config-migration.md Documents migration status, design decisions, and remaining follow-ups.

src/batcontrol/setup.py
@@ -66,9 +69,10 @@ def load_config(configfile:str) -> dict:

config = yaml.safe_load(config_str)

src/batcontrol/config_model.py
Comment on lines +213 to +214
timezone: str = 'Europe/Berlin'
time_resolution_minutes: int = 60
tests/batcontrol/test_core.py Outdated
@@ -290,7 +290,12 @@ def test_api_set_limit_applies_immediately_in_mode_8(


class TestTimeResolutionString:
@MaStr @claude
Address PR review round 4: empty YAML guard, rename misleading test c…
6307d1d 
…lass

- setup.py: raise RuntimeError when yaml.safe_load() returns None (empty
  or non-mapping YAML) before calling validate_config(), giving a clear
  error instead of a TypeError from **None unpacking
- test_core.py: rename TestTimeResolutionString -> TestTimeResolutionValidation
  and update docstring to reflect tests use int values (string coercion
  is covered in test_config_model.py)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@MaStr MaStr requested a review from Copilot March 17, 2026 20:29
Copilot AI reviewed Mar 17, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR centralizes configuration validation/coercion by introducing Pydantic v2 models and wiring them into config loading, reducing scattered runtime type conversions (notably for Home Assistant add-on stringly-typed options).

Changes:

  • Added BatcontrolConfig + subsystem config models and a validate_config() entry point to validate/coerce raw config dicts at load time.
  • Integrated validation into setup.load_config() and simplified downstream modules (core, inverter factory, dynamic tariff factory, consumption forecast factory) by removing redundant casts/legacy handling.
  • Added extensive unit + integration tests for config validation and YAML loading, plus a migration note doc.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/batcontrol/config_model.py New Pydantic models + validation entry point for centralized config coercion.
src/batcontrol/setup.py Calls validate_config() during config load; adds basic YAML shape check.
src/batcontrol/core.py Removes local time_resolution_minutes coercion/validation and relies on validated config.
src/batcontrol/inverter/inverter.py Drops legacy key mutation/defaulting in factory; passes through cache_ttl.
src/batcontrol/dynamictariff/dynamictariff.py Removes float() casts for already-coerced utility config values.
src/batcontrol/forecastconsumption/consumption.py Keeps parsing for nested HA config dicts; adjusts coercions for numeric/list fields.
tests/batcontrol/test_config_model.py New unit tests for model defaults, coercion, constraints, and backwards-compat rename.
tests/batcontrol/test_config_load_integration.py New integration tests covering YAML load + validation behavior.
tests/batcontrol/test_core.py Updates time resolution tests to reflect validation now happening in load_config().
pyproject.toml Adds pydantic>=2,<3 dependency.
docs/pydantic-config-migration.md Documents migration status, design decisions, and remaining follow-ups.

src/batcontrol/config_model.py
Comment on lines +222 to +231
battery_control: BatteryControlConfig = BatteryControlConfig()
battery_control_expert: Optional[BatteryControlExpertConfig] = None
inverter: InverterConfig = InverterConfig()
utility: UtilityConfig
mqtt: Optional[MqttConfig] = None
evcc: Optional[EvccConfig] = None
pvinstallations: List[PvInstallationConfig]
consumption_forecast: ConsumptionForecastConfig = (
ConsumptionForecastConfig()
)
src/batcontrol/forecastconsumption/consumption.py
Comment on lines +91 to +93
# Coerce numeric fields that may arrive as strings from nested HA config
cache_ttl_hours = float(ha_config.get('cache_ttl_hours', 48.0))
multiplier = float(ha_config.get('multiplier', 1.0))
src/batcontrol/core.py
Comment on lines +93 to 95
# in config_model.py (must be int, must be 15 or 60)
self.time_resolution = config.get('time_resolution_minutes', 60)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 0 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

CodeQuality PRs fixing CodeQuality issues enhancement New feature or request

Projects

None yet

Milestone

0.8.0

Development

Successfully merging this pull request may close these issues.

Move config DataModel to Pydantic

3 participants

@MaStr @claude