2 Pydantic Strictness in Third-Party APIs

[fderuiter]

Pydantic Strictness in Third-Party APIs
The project relies heavily on pydantic for modeling. While excellent for internal APIs, strictly validating payloads from external, proprietary enterprise systems (like iMednet) is highly fragile. Enterprise APIs frequently deploy silent, undocumented payload changes, such as adding new keys or returning null instead of empty strings. If the SDK models are strictly typed without fallback mechanisms, a minor upstream update will instantly break data pipelines relying on this library.

Actionable Solution: Ensure every Pydantic model is configured with model_config = ConfigDict(extra='ignore'). Utilize permissive typing (e.g., Optional[Any]) for any field that is not strictly required for routing or authentication to ensure the SDK does not crash simply because the API returned an unexpected telemetry field.

[fderuiter]

Phase 1: Establish the Resilient Base Model

Instead of repeating the model_config in every single model across the SDK, we will centralize this behavior in a single base class. Every model in the SDK will inherit from this class.

Action: Open or create src/imednet/models/base.py and define the foundational ImednetBaseModel.

# src/imednet/models/base.py
from pydantic import BaseModel, ConfigDict

class ImednetBaseModel(BaseModel):
    """
    Core base model for all iMednet API responses.
    
    Design Philosophy:
    - extra='ignore': Silently drops any new, undocumented fields the API introduces.
    - populate_by_name: Allows models to be instantiated using either pythonic snake_case 
      names or the original API camelCase names (via Field aliases).
    """
    model_config = ConfigDict(
        extra='ignore',
        populate_by_name=True,
        # Optional but recommended: strip whitespace from strings automatically
        str_strip_whitespace=True 
    )

Phase 2: Implement Permissive Typing on Domain Models

Now, we refactor the specific endpoint models. You must audit your models and differentiate between Routing/Core Identity Fields (which must be strict, because the SDK cannot function without them) and Informational Fields (which must be permissive).

• Rule 1: If a field is nullable or might suddenly disappear, type it as Optional[Type] = None.
• Rule 2: If the API might completely change a field's type (e.g., sending an integer 1 instead of a string "1" for a status), use Any or Union for maximum safety.

Action: Update your domain models (e.g., src/imednet/models/studies.py). Here is how you apply it to the Studies payload referenced in the iMednet API documentation.

# src/imednet/models/studies.py
from typing import Optional, Any
from pydantic import Field
from imednet.models.base import ImednetBaseModel

class StudyModel(ImednetBaseModel):
    # --- CORE IDENTITY FIELDS (Strict) ---
    # The SDK fundamentally breaks if the API stops returning study keys/IDs.
    # These do NOT have default values, meaning Pydantic will still raise an error if they are missing.
    study_key: str = Field(alias="studyKey")
    study_id: int = Field(alias="studyId")
    
    # --- INFORMATIONAL FIELDS (Permissive) ---
    # Everything else must be able to fail gracefully. 
    # Notice the explicit '= None' to handle missing keys entirely.
    sponsor_key: Optional[str] = Field(default=None, alias="sponsorKey")
    study_name: Optional[str] = Field(default=None, alias="studyName")
    
    # If the API might send `null`, an empty string `""`, or drop the key:
    study_description: Optional[str] = Field(default=None, alias="studyDescription")
    
    # Highly volatile fields (e.g., sometimes an int, sometimes a string, sometimes a complex object)
    # Using Any guarantees parsing will not crash the data pipeline.
    study_type: Optional[Any] = Field(default=None, alias="studyType")
    
    date_created: Optional[str] = Field(default=None, alias="dateCreated")
    date_modified: Optional[str] = Field(default=None, alias="dateModified")

Phase 3: Defensive Test Validation

We must prove that our models survive malicious or undocumented upstream changes.

Action: Create a targeted test file tests/unit/test_models_base_extra.py to lock in this behavior and prevent future regressions.

# tests/unit/test_models_base_extra.py
import pytest
from imednet.models.studies import StudyModel

def test_model_ignores_undocumented_fields():
    """Ensure that newly injected API keys do not crash the parser."""
    payload = {
        "studyKey": "PHARMADEMO",
        "studyId": 100,
        "undocumented_telemetry_tracker": "12345XYZ",
        "new_feature_flag": True
    }
    
    # Should parse successfully without ValidationErrors
    study = StudyModel.model_validate(payload)
    
    # Core fields are mapped
    assert study.study_key == "PHARMADEMO"
    
    # Undocumented fields are cleanly ignored, not stored on the object
    assert not hasattr(study, "undocumented_telemetry_tracker")

def test_model_survives_null_values():
    """Ensure that fields converting from string to null do not crash the parser."""
    payload = {
        "studyKey": "PHARMADEMO",
        "studyId": 100,
        "studyDescription": None, # Simulating the API returning null instead of ""
        "studyType": {"complex": "object_unexpected"} # Simulating type change
    }
    
    study = StudyModel.model_validate(payload)
    
    assert study.study_description is None
    assert study.study_type == {"complex": "object_unexpected"}

---

Phase 4: Execution Checklist & Definition of Done (DoD)

Execute this checklist to guarantee zero regressions.

Implementation Checklist

• ImednetBaseModel is created in src/imednet/models/base.py with ConfigDict(extra='ignore').
• ALL existing Pydantic models in src/imednet/models/ have been updated to inherit from ImednetBaseModel rather than directly from pydantic.BaseModel.
• A global search-and-replace has been performed to ensure no individual model is overriding the config with extra='forbid'.
• Non-essential fields (descriptions, dates, dynamic metadata) are typed with Optional[...] = None across all models.
• Volatile fields (metadata arrays, user roles, deeply nested unknown JSON) are typed with Optional[Any] = None or Optional[Dict[str, Any]] = None.

Validation

• Run poetry run pytest tests/unit/test_models_base_extra.py — Tests must pass.
• Run poetry run mypy src/imednet/models/ — Static types must pass.

Definition of Done (DoD)

1. Injecting 5 completely random, undocumented key-value pairs into any mock JSON API response in your test suite does not result in a single pydantic.ValidationError.
2. Changing studyName or siteName to null in a mock JSON response results in the model parsing successfully with a .study_name == None property, rather than throwing a ValidationError.