Skip to content

Add column field log to task pipeline#554

Merged
pooleycodes merged 2 commits into
mainfrom
columns-task-pipeline
May 22, 2026
Merged

Add column field log to task pipeline#554
pooleycodes merged 2 commits into
mainfrom
columns-task-pipeline

Conversation

@pooleycodes
Copy link
Copy Markdown
Contributor

@pooleycodes pooleycodes commented May 19, 2026

What type of PR is this?

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Description

Extends TaskPipeline to detect and surface missing mandatory fields from a
column-field log as task rows.

A new column_field_path parameter (alongside mandatory_fields) can be passed
to TaskPipeline.run(). When provided, the pipeline reads the column-field CSV,
compares the field column against the supplied mandatory fields list, and emits
one column-field task row per missing field — using the same schema and SHA-256
reference hash as the existing log and issue sources.

Key behaviours:

  • mandatory_fields entries can be a plain string (field must be present) or a
    list of strings (any one of the alternatives is sufficient)
  • If column_field_path is given without mandatory_fields, a warning is logged
    and the check is skipped gracefully
  • If the path does not exist, a warning is logged and it is skipped
  • Fixes _transform_issues_to_tasks to guard against None organisation/endpoint
    (was passing None directly to pl.lit)
  • run() now consistently returns self.status rather than the bare enum value

Related Tickets & Documents

QA Instructions

Pass a column-field CSV (columns: dataset, resource, column, field) and a
mandatory_fields list to TaskPipeline().run(). Rows for any fields absent from
the CSV will appear in the output with task-source=column-field and details
containing {"field": "...", "issue_type": "missing-field"}.

Added/updated tests?

  • Yes — TestColumnFieldTasks in tests/unit/pipeline/test_task.py

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 15.62500% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.40%. Comparing base (c8265f6) to head (91a9366).
⚠️ Report is 51 commits behind head on main.

Files with missing lines Patch % Lines
digital_land/pipeline/task.py 15.62% 27 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #554      +/-   ##
==========================================
- Coverage   86.54%   86.40%   -0.15%     
==========================================
  Files          92       93       +1     
  Lines        6466     6641     +175     
==========================================
+ Hits         5596     5738     +142     
- Misses        870      903      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pooleycodes pooleycodes changed the title collumn issue log added to task pipeline Add column field log to task pipeline May 21, 2026
Copy link
Copy Markdown
Contributor

@gibahjoe gibahjoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pooleycodes pooleycodes merged commit 2cf0281 into main May 22, 2026
5 checks passed
@pooleycodes pooleycodes deleted the columns-task-pipeline branch May 22, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants