Overview
The normalise step now produces data using Polars LazyFrames, but the parse step still uses the legacy implementation and cannot process this new output. To keep the pipeline consistent and efficient, the parse phase must be updated to use the Polars version. This will keep existing behaviour while modernising the underlying process.
Tech Approach
- Update digital_land/phase/parse.py so it delegates to the Polars parse implementation.
- Implement the full parse logic inside digital_land/phase_polars/transform/parse.py.
- Convert existing parse rules into Polars lazy transformations.
- Ensure the parse phase accepts the LazyFrame produced by the new normalise step.
- Keep the same public interface so no downstream code needs changing.
Useful references:
phase_polars/transform/normalise.py
phase_polars/transform/parse.py
phase/parse.py
Polars LazyFrame documentation
Acceptance Criteria/Tests
Resourcing & Dependencies
Depends on Normalise, no external teams.
Overview
The normalise step now produces data using Polars LazyFrames, but the parse step still uses the legacy implementation and cannot process this new output. To keep the pipeline consistent and efficient, the parse phase must be updated to use the Polars version. This will keep existing behaviour while modernising the underlying process.
Tech Approach
Useful references:
phase_polars/transform/normalise.py
phase_polars/transform/parse.py
phase/parse.py
Polars LazyFrame documentation
Acceptance Criteria/Tests
Resourcing & Dependencies
Depends on Normalise, no external teams.