Skip to content

Data layer: normalisation stats, custom CSV name, LOCO country features #80

Merged
gabrieletijunaityte merged 2 commits into
developfrom
feature/cy-data-layer
Apr 16, 2026
Merged

Data layer: normalisation stats, custom CSV name, LOCO country features #80
gabrieletijunaityte merged 2 commits into
developfrom
feature/cy-data-layer

Conversation

@robknapen

Copy link
Copy Markdown
Collaborator

What does this PR do?

Data layer: normalisation stats, custom CSV name, LOCO country features, for CY use case

  • Normalisation stats in BaseDataModule: After splitting, the datamodule now computes per-feature and per-target mean/std on the training fold (_compute_tabular_normalisation_stats, _compute_target_normalisation_stats). Stats are exposed as abular_normalisation_stats and target_normalisation_stats for use by encoders/heads. Constant features are clamped to std=1 to avoid division by zero.
  • Custom CSV filename in BaseDataset: Added csv_name parameter so a dataset can load from a non-default CSV (e.g. a country-specific file) without subclassing. Falls back to model_ready_<dataset_name>.csv when not set.
  • LOCO-safe country features in YieldAfricaDataset: Added use_country_features flag (default True). When False, country one-hot columns are not injected. This is needed for leave-one-country-out evaluation where the held-out country's one-hot is always 0 in training and always 1 at test time — a guaranteed distribution shift that inflates error.
  • Vectorised file-path assignment: Replaced a slow row-by-row loop with a pd.concat + apply approach in BaseDataset._set_eo_paths.
  • Test fixtures now respect the --use-mock CLI flag instead of hardcoding mock=True.
  • Added test_yield_africa_dataset_no_country_features to verify no feat_country_* columns appear when the flag is off.

Changes are all CY dataset specific, and coded as opt-ins, so should not break existing code.

Before submitting

  • Did you make sure title is self-explanatory and the description concisely explains the PR?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you test your PR locally with pytest command?

@gabrieletijunaityte gabrieletijunaityte left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were great insights on some bugs!

@gabrieletijunaityte gabrieletijunaityte merged commit 9da1663 into develop Apr 16, 2026
4 checks passed
@robknapen robknapen deleted the feature/cy-data-layer branch April 16, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants