[KEP-7] Dataset Validation

[SajidAlamQB]

Context: Explore a Kedro-First Approach to Data Validation (#5390)
Spike: Prototype Dataset Validation Using Pandera (#5391)
Authors: @SajidAlamQB
KEP shepherd: @SajidAlamQB
Tech design recording: Here
Tech design notes: #5391 (comment)

---

What are we trying to do?

Introduce native dataset validation in Kedro by extending the existing parameter validation pattern from KEP-1.

Users annotate a node input with a Pandera DataFrameModel, and Kedro validates the DataFrame automatically at the I/O boundary, before the node function runs.

The same TypeExtractor discovers both:

• parameter type hints, such as Pydantic/dataclass models
• dataset type hints, such as Pandera models

The user-facing pattern is identical: type hints drive validation.

Example: Pandera

# schemas.py
import pandera as pa


class CompaniesSchema(pa.DataFrameModel):
    id: int = pa.Field(nullable=False, unique=True)
    company_rating: float = pa.Field(ge=0, le=1)

    class Config:
        strict = False

# nodes.py
import pandas as pd


def preprocess_companies(companies: CompaniesSchema) -> pd.DataFrame:
    # CompaniesSchema is validated automatically at load time.
    # No hooks, no catalog config, no decorators.
    companies["iata_approved"] = companies["iata_approved"].map(
        {"t": True, "f": False}
    )
    return companies

# pipeline.py
from kedro.pipeline import Pipeline, node


pipeline = Pipeline(
    [
        node(
            func=preprocess_companies,
            inputs="companies",
            outputs="preprocessed_companies",
            name="preprocess_companies",
        )
    ]
)

The DataFrame loaded from companies is validated against CompaniesSchema before reaching the node.

Validation failures raise DataValidationError, listing every failure at once via Pandera’s lazy=True.

---

What this is not

This is not:

• a breaking change to the Kedro API
• a replacement for hooks — hooks remain available for cases this does not cover
• a mandatory Pandera dependency — Pandera will be available through an optional extra: kedro[validation]
• a pluggable multi-backend validation framework — this KEP scopes to Pandera; pluggability is a future extension

---

How is it done today, and what are the limits of current practice?

Data validation in Kedro projects today is commonly implemented using hooks and third-party libraries.

The validation logic lives in a separate hook class registered in settings.py, often referencing schemas defined in another module.

# hooks.py
class DataValidationHook:
    @hook_impl
    def after_dataset_loaded(self, dataset_name, data):
        if dataset_name == "companies":
            CompaniesSchema.validate(data)

        if dataset_name == "shuttles":
            ShuttlesSchema.validate(data)

Limits

• Hidden control flow: node functions show no indication that validation runs.
• Discoverability: validation logic lives away from the code that depends on it.
• Maintenance burden: every new validated dataset requires editing the hook.
• No IDE support: there is no autocompletion or static analysis on validated columns.

---

What is new in this approach, and why will it be successful?

This proposes a dataset validation layer integrated with the existing Kedro Validation Framework from KEP-1.

Overview

Dataset validation reuses the validation infrastructure introduced for parameters and adds two new components:

• _ValidatingDataset, a proxy wrapper
• validate_dataframe(), the Pandera validation call

The same TypeExtractor discovers both parameter and dataset type hints:

• the parameter path filters for the params: prefix
• the dataset path filters against the params: prefix

---

Architecture

The dataset validation extension consists of two new components plus reuse of existing components.

KedroContext._get_catalog() (entry point)
        ↓
_apply_dataset_validation()
        ↓
┌─────────────────────┬─────────────────────┐
│ TypeExtractor       │ _ValidatingDataset  │
│ (existing, shared)  │ (new, proxy wrapper)│
│  • extract_dataset_ │  • intercepts load()│
│    schemas() NEW    │  • delegates rest   │
└─────────────────────┴─────────────────────┘
        ↓
   validate_dataframe()  ← calls schema.validate(df, lazy=True)

Core components

TypeExtractor

Location: type_extractor.py
Status: existing, extended

Shared discovery engine. Gains a new method, extract_dataset_schemas(), which filters for Pandera DataFrameModel annotations on non-params: inputs.

_ValidatingDataset

Location: dataset_validator.py
Status: new

Proxy that wraps a catalog dataset.

It:

• intercepts load()
• validates the DataFrame
• delegates everything else, including save(), versioning, and dataset-specific methods, to the wrapped dataset via __getattr__

validate_dataframe()

Location: dataset_validator.py
Status: new

Runs:

schema.validate(data, lazy=True)

and wraps failures in DataValidationError with dataset and schema context.

DataValidationError

Location: exceptions.py
Status: new

Lives alongside:

• ParameterValidationError
• ModelInstantiationError

_apply_dataset_validation()

Location: context.py
Status: new

Wires the validation layer into KedroContext._get_catalog().

---

Integration points

Context-level integration

Dataset validation integrates with KedroContext._get_catalog() via a new method: _apply_dataset_validation().

This runs after the catalog is built and parameters are added, but before after_catalog_created hooks fire.

def _get_catalog(self, ...) -> CatalogProtocol:
    catalog = catalog_class.from_config(...)              # existing

    parameters = self._get_parameters()                   # existing
    for param_name, param_value in parameters.items():
        catalog[param_name] = param_value                 # existing

    self._apply_dataset_validation(catalog)               # NEW

    _validate_transcoded_datasets(catalog)                # existing
    self._hook_manager.hook.after_catalog_created(...)    # existing

    return catalog

Matching catalog datasets are replaced with _ValidatingDataset wrappers.

The runner is unaware of this: catalog.load() works exactly as it does today.

---

Dataset validation flow

┌──────────────────────────┐
│ catalog.load("companies")│
└──────────┬───────────────┘
           │
           ▼
┌─────────────────────────┐    ┌─────────────────────────┐
│ _ValidatingDataset.load │───▶│ original.load()         │
│                         │    │ CSVDataset, etc.        │
└──────────┬──────────────┘    └─────────────────────────┘
           │ DataFrame
           ▼
┌─────────────────────────┐    ┌─────────────────────────┐
│ validate_dataframe()    │───▶│ DataValidationError     │
│ schema.validate(        │    │ lists all failures via  │
│   df, lazy=True         │    │ Pandera lazy mode       │
│ )                       │    │                         │
└──────────┬──────────────┘    └─────────────────────────┘
           │ valid
           ▼
┌─────────────────────────┐
│ Node function receives  │
│ guaranteed-valid data   │
└─────────────────────────┘

---

Key features and benefits

Strengths

• Extends existing infrastructure: around 50 lines added to TypeExtractor, around 40 to dataset_validator.py, and around 30 to context.py.
• Shared discovery with parameter validation: one mechanism discovers both parameter and dataset validation hints.
• Fully opt-in: no annotation means no validation and zero impact on existing pipelines.
• Graceful degradation: if Pandera is not installed, _is_pandera_model() returns False and the feature silently disappears without import errors.
• Composable: works alongside parameter validation, hooks, and custom datasets without conflict.
• Better error reporting: Pandera’s lazy=True reports all validation failures in a single error, not one at a time.

Drawbacks

• Core change: introduces additional surface area in Kedro core. This is mitigated by reusing the validation framework.
• Performance: adds validation cost on every load() for annotated datasets. This may be noticeable for very large DataFrames.

---

Usage

# install with optional validation extra
pip install kedro[validation]

# run your Kedro project with dataset validation
kedro run

---

Reference

• Prototype branch: prototype/validation-dataset-pandera
• Tech design notes: Prototype Dataset Validation Using Pandera #5391 (comment)

---

Who cares? If successful, what difference will it make?

This proposal would:

• Fail fast at the I/O boundary: data quality issues surface at load time with clear errors naming the dataset, schema, and every failed check.
• Make validation visible: validation is declared in the node function signature and discoverable through the same mechanism users already use for IDE support.
• Unify Kedro’s validation story: Kedro has one model — type hints drive validation, whether the input is a parameter or a dataset.
• Lower the barrier to validation: no more hook boilerplate; users only need to write schemas.
• Create a foundation for further validation work: opens the door to schema-driven catalog documentation and Kedro-Viz integration.

---

What are the risks?

Performance

Validation adds overhead on every load() for annotated datasets.

Pandera is fast on pandas DataFrames, but the cost may be noticeable on very large data. This is mitigated by the feature being fully opt-in and by Pandera supporting native validation on most backends.

Lazy backend behaviour

We need to verify per backend that validation does not force unnecessary materialisation.

An earlier draft of this prototype assumed Pandera forced .compute() on Dask. This was corrected during the tech design: Pandera supports Polars, Ibis, Spark, and Dask natively since approximately 0.16.

Only truly unsupported backends should be skipped.

Annotation style ambiguity

Pandera’s documented convention uses parameterised generics:

pd.DataFrame[CompaniesSchema]

The prototype uses the direct schema annotation:

CompaniesSchema

Both should be supported. The implementation must unwrap generics.

Compatibility issues reported in other frameworks, such as Dagster, need investigation.

Schema conflicts

If two pipelines annotate the same dataset with different schemas, the last one wins silently today.

Per-node schemas may be a legitimate use case, because different nodes may need different guarantees. This needs real-world examples from users.

Validation timing

Validation currently runs at catalog build time.

This may wrap datasets that are not part of the current run. We could defer to materialisation time to align with --pipeline scoping. This relates to PR #5443.

Maintenance

This adds surface area to Kedro core.

The risk is mitigated by reusing the existing validation framework.

---

How long will it take?

Around 2–3 weeks for the production implementation, assuming input and output validation are both in scope and annotation/backend questions are resolved during the initial PRs.

---

Appendix A: Proposed API changes

User-facing

# nodes.py — no API change, just a different type hint
def preprocess(companies: CompaniesSchema) -> pd.DataFrame:
    ...

Framework-facing

# kedro.validation.exceptions
class DataValidationError(Exception):
    ...

# kedro.validation.dataset_validator
from typing import Any


def validate_dataframe(dataset_name: str, data: Any, schema_class: type) -> Any:
    ...


class _ValidatingDataset:
    def __init__(self, wrapped_dataset, schema_class, dataset_name):
        ...

    def load(self) -> Any:
        ...

    def save(self, data: Any) -> None:
        ...

    def __getattr__(self, name: str) -> Any:
        ...

# kedro.validation.type_extractor
class TypeExtractor:
    def extract_dataset_schemas(self) -> dict[str, type]:
        ...  # NEW

# kedro.framework.context.context
class KedroContext:
    def _apply_dataset_validation(self, catalog: CatalogProtocol) -> None:
        ...  # NEW

Packaging

# pyproject.toml
[project.optional-dependencies]
validation = ["pandera>=0.16", "pydantic>=2"]

---

Appendix B: Optional design sketch

graph TD
    %% =============================
    %% SECTION 1: Startup (catalog build)
    %% =============================
    subgraph CB["Startup: Catalog Build"]
        A["KedroContext._get_catalog()"] --> B["Build catalog from YAML"]
        B --> C["Validate parameters (existing)"]
        C --> D["_apply_dataset_validation() NEW"]
        D --> E["TypeExtractor.extract_dataset_schemas()"]
        E --> F["Schema Map<br/>{ds_name: schema_class}"]
        F --> G["Wrap matching datasets<br/>with _ValidatingDataset"]
        G --> H["Fire after_catalog_created hook"]
    end

    %% =============================
    %% SECTION 2: Runtime (catalog.load)
    %% =============================
    subgraph RT["Runtime: Per Node Load"]
        I["catalog.load('companies')"] --> J["_ValidatingDataset.load()"]
        J --> K["original.load()<br/>CSV/Parquet/etc."]
        K --> L["validate_dataframe()<br/>schema.validate(df, lazy=True)"]
        L -->|valid| M["Node receives validated DataFrame"]
        L -->|invalid| N["DataValidationError<br/>(all failures listed)"]
    end

    %% =============================
    %% STYLES
    %% =============================
    style CB fill:#fafafa,stroke:#ccc,stroke-width:1px
    style RT fill:#fafafa,stroke:#ccc,stroke-width:1px

    style A fill:#e3f2fd,stroke:#90caf9,stroke-width:1px
    style B fill:#f3e5f5,stroke:#ce93d8,stroke-width:1px
    style C fill:#fff9c4,stroke:#fbc02d,stroke-width:1px
    style D fill:#fce4ec,stroke:#f48fb1,stroke-width:2px
    style E fill:#ffe0b2,stroke:#ffb74d,stroke-width:1px
    style F fill:#e8f5e9,stroke:#81c784,stroke-width:1px
    style G fill:#fce4ec,stroke:#f48fb1,stroke-width:2px
    style H fill:#fff3e0,stroke:#ffb74d,stroke-width:1px
    style I fill:#e3f2fd,stroke:#90caf9,stroke-width:1px
    style J fill:#fce4ec,stroke:#f48fb1,stroke-width:2px
    style K fill:#f3e5f5,stroke:#ce93d8,stroke-width:1px
    style L fill:#fff9c4,stroke:#fbc02d,stroke-width:1px
    style M fill:#66bb6a,color:#fff,stroke-width:1px
    style N fill:#ef5350,color:#fff,stroke-width:1px

Loading

---

Appendix C: Optional rejected designs

Hooks, current approach

Rejected because validation has hidden control flow.

Validation lives in a separate hook class registered in settings.py. Node functions show no indication that validation runs.

This provides full flexibility, but low discoverability.

Wrapper dataset configured in YAML

Rejected because the user would need to configure a validating dataset class in catalog.yml, with schemas referenced by string path.

This has several drawbacks:

• no IDE support
• no static analysis
• high framework coupling
• validation tied to configuration rather than code

Pandera’s @check_types decorator on node functions

Pandera natively supports input/output validation via @check_types on functions.

Rejected because it couples validation to the function itself. Validation runs every time the function is called, even in tests or notebooks where it may not be wanted.

Keeping validation at the framework/catalog layer means node functions remain plain Python.

Pluggable multi-backend validator interface

The parent issue (#5390) mentions extensibility for Great Expectations and other libraries.

Rejected for v1 to avoid premature abstraction.

The prototype validates that the abstraction point exists: _is_pandera_model and validate_dataframe could become part of a Validator protocol.

This can be revisited once the Pandera-only implementation ships.

---

Open questions

• Opt-out flag: should validation be opt-out as well as opt-in? Tech design consensus: defer for v1, but design so it can be added later.
• Schema conflicts: should Kedro warn or error? Should schemas be per-node or per-dataset? Needs real-world examples from users.
• Validation timing: should validation happen at catalog build time or materialisation time? This ties into PR Scope type_extractor to target pipeline instead of all registered pipelines for Validation #5443, which scopes TypeExtractor to the target pipeline.
• Annotation style: should both CompaniesSchema and pd.DataFrame[CompaniesSchema] be supported? We need to investigate Dagster-style compatibility issues.
• Catalog-in-hooks: does kedro-telemetry creating a new catalog inside hooks trigger validation multiple times?
• Pandera community engagement: when and how should we engage Niels, the Pandera creator? Options include a joint blog post, podcast, or coffee chat.

[SajidAlamQB]

Quick update from Neeraj (Pandera maintainer + Kedro user) on the annotation style question: he confirmed that pd.DataFrame[Schema] is the canonical Pandera form because the DataFrame class was extended to support it. We'll support both df: Schema and df: pd.DataFrame[Schema] in the implementation, but document the latter as the recommended style.

[deepyaman]

Willl df: Schema type check properly? Have you tested it with mypy/ty/tool of choice?

[SajidAlamQB]

Not really CompaniesSchema is a Pandera model class, not a DataFrame type. A type checker will treat df as if it's a CompaniesSchema instance, but the actual runtime value is a pd.DataFrame

The correct Pandera convention is df: pd.DataFrame[CompaniesSchema] Pandera extended the DataFrame class to accept the generic. Neeraj confirmed this.

So perhaps we document pd.DataFrame[Schema] as the recommended form. Support both at runtime.

[SajidAlamQB]

Worth flagging in the restructured KEP, the schema reference moves to catalog.yml, so node args can stay plain pd.DataFrame pd.DataFrame[Schema] annotations remain valid as IDE documentation, just no longer the validation driver.

[noklam]

Is there a way to disable this validation even when the model is defined? My use case is from the IDE I probably don't want to directly use this approach, but rather having some kind of API to validate dataset from the extension, and report it as a diagnosis. So I will need:

1. Disable this feature globally
2. An API that I can programatically validate datasets from the extension.

[SajidAlamQB]

Tech design consensus was to defer for v1 but design so it can be added. I'll propose this be a project level flag in settings.py (e.g. VALIDATION_ENABLED = False). The programatic api is a great use case I haven't thought of sounds good to me

[deepyaman]

Overall, I'm very excited to see native Pandera integration in Kedro. However, I do think there are some issues that need to be addressed.

On schema conflicts and schema definition location

If two pipelines annotate the same dataset with different schemas, the last one wins silently today.

I (still) think this is concerning.

I also think this is fundamentally confusing from a design perspective. If validation is tied to datasets, great, clearly define the schema on the dataset. https://pandera.readthedocs.io/en/latest/schema_inference.html#write-to-yaml provides a clear path to doing so with Kedro.

However, if validation is tied to nodes, the schema should be tied to the function argument, not the dataset. I think I gave this example in the tech design:

Suppose you have an ML training node, and the algorithm doesn't work with non-numeric columns. You can define a common schema for all possible inputs to this node that ensures all columns are numeric. Therefore, this schema is being used for multiple datasets.

Furthermore, (some of) these inputs may also be inputs to other nodes, which have different data validation checks, so the model of one schema per dataset doesn't hold here.

In short, I think it's OK to choose one path or the other to start here (schema tied to dataset feels more common, intuitively, but I'm not sure), but I could easily see the desire to support both at some point down the road. Furthermore, I definitely think we should make sure that, should we focus on associating one schema per dataset (and caching results on subsequent loads, etc.), then the schema is more closely tied to the catalog. If do go the route of defining validation config alongside the dataset, I think most of the _ValidatingDataset logic can be reused.

The other benefit of having the schema on the data catalog (as opposed to the nodes) is that you can potentially validate data used outside of the pipeline run flow.

P.S. As I was writing this, I think the cleanest way to support caching even if you go with the schema-on-node approach is to have a mapping from dataset/schema combination to result, not just dataset to result.

When should data validation happen?

The proposal focuses on data validation after load, but data validation before write is also a very common pattern. If schemas are defined alongside the dataset definitions, I think it's fair to assume that the load and save schemas are the same.

You may consider when to run validation (on save/load/both); hopefully, if you wrote the data, you don't need to revalidate on load (can use the cached result).

Supporting configuration options

Worth considering how you plan to support configuration options. If the answer for now is, use the environment variable, I guess that's OK?

User friendliness

If I understand correctly, if users inspect the catalog object, they will now see keys mapped to _ValidatingDataset instances—is that right?

At a minimum, I think the repr() for the _ValidatingDataset should clearly show the underlying dataset, because this is what matters to users.

But, more importantly—why not hooks? The DataValidationHook example in proposal is obviously bad, because it hardcodes the schemas to use in the hook implementation. However, there's no reason why the hook cannot look up/construct the appropriate schema for dataset keys. after_dataset_loaded and before_dataset_saved seem intuitive/perfect for validate on read/write, respectively.

The other benefit of data catalog hooks are that they're also referenced when executing outside of pipeline runs.

[SajidAlamQB]

Thank you @deepyaman this is the most important feedback the KEP got and it forced a real rethink. Your "schema-on-node vs schema-on-dataset" framing is correct, and the prototype was trying to do both. Pivoting to schema-on-dataset (see post update below) resolves several of your points at once:

Schema conflicts: gone by construction. One catalog entry, one validator. The "last wins silently" bug can't happen.

Schemas tied to catalog: yes having the catalog config is now the source of truth. The schema is right there with the dataset definition.

Validation outside pipeline runs: addressed via the new public validate_catalog_dataset(catalog, name) API. Anyone with a catalog object can validate datasets on demand. Returns a ValidationResult instead of raising — usable for IDE diagnostics, notebooks, CI.
_ValidatingDataset UX: proper repr that surfaces the wrapped dataset clearly is in v1 scope.
Save-side validation: explicitly in v1 scope, not a future item.

On the Pandera YAML schema route you suggested I want to support it, but not in v1. For now a Python-class-based Validator protocol is strictly more general. It supports custom checks, non-tabular validation, and any third-party backend. A YAML schema loader is straightforward to layer on top as it just produces a Python schema object that satisfies the same protocol. Scoping v1 to Python keeps the KEP smaller.

Your training-node example (one schema applied to many datasets) is the one case the new model handles differently. That becomes either an in-node check or a shared schema referenced from multiple catalog entries. Less elegant for that specific case, but the trade-off was worth it for resolving the silent-conflict bug. Happy to discuss if you think this isn't the right call.

Cache-by-(dataset, schema) is not needed in the new model since there's one schema per dataset.

[SajidAlamQB]

Thanks @deepyaman, @Adeikalam, and @noklam for the substantive comments. Reading them together, you've all pointed at the same gap from different angles:

• The prototype declares schemas on node function signatures, but applies them per dataset via a wrapper. That mismatch causes the "last schema wins silently" bug @deepyaman flagged.
• In real projects, putting schemas next to nodes (as @Adeikalam described) creates cross-pipeline dependencies and makes schemas hard to discover.
• The Pandera-only design doesn't accommodate non-tabular validation or alternative backends (GX, Pydantic, custom).
• There's no way to validate datasets outside a pipeline run for tooling like Kedro IDE extension.

I'm pausing the vote and pivoting the KEP to address all of this in one coherent design. The new shape:

Schema-on-dataset with a pluggable validator protocol

• Schemas live in a conventional src/<project>/schemas/ folder following the same pattern as parameters live in conf/<env>/parameters.yml
• Each catalog entry can optionally declare a validator: pointing at a schema class
• A minimal Validator protocol (validate(data) -> data) makes the backend pluggable. Pandera is the default and reference implementation, but GX, Pydantic, custom, and non-tabular validators all work
• One schema per dataset by construction so no silent conflicts possible
• Validates on load and save in v1
• Public validate_catalog_dataset() API for ad-hoc use (IDE extensions, notebooks, CI)
• Global opt-out flag
• The wrapper mechanism from the prototype carries over but only the discovery source changes (catalog config instead of node type hints).

I'll post the restructured KEP and reply individually to each of your comments to show how the new design maps to your specific points. Thank you everyone!

[SajidAlamQB]

Superseded by: #5602