Skip to content

Prediction model training changes#111

Merged
gabrieletijunaityte merged 69 commits into
developfrom
train/prediction
May 28, 2026
Merged

Prediction model training changes#111
gabrieletijunaityte merged 69 commits into
developfrom
train/prediction

Conversation

@gabrieletijunaityte

@gabrieletijunaityte gabrieletijunaityte commented May 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

  • Introduce checkpoint dir to save model checkpoints with name that’s: their unique wandb run id + epoch number.
  • Introduce wandb script which connects go wandb API and creates a csv log of trianed models availabel on wandbs. This allows for controlling which models are relevant and automatic deletion of checkpoints that crashed/ were not relevant and deleted on wandb dashboard.
  • Define s2bms prediction model experiment configurations.
  • Add new s2bms data splits, which have union of available data for tessera and aef embeddings.
  • Changes to dataset class:
    • Allow aef to be loaded from numpy files.
    • S2 data can be normalised using division by 10000 (for SSL4EO models).

Fixes #<issue_number>

Before submitting

  • Did you make sure title is self-explanatory and the description concisely explains the PR?
  • Did you make sure your PR does only one thing, instead of bundling different changes together?
  • Did you list all the breaking changes introduced by this pull request?
  • Did you test your PR locally with pytest command?

@gabrieletijunaityte
Remove missing tessera tiles from training split
49fb4ed
@gabrieletijunaityte
Remove missing tessera tiles from training split
d6be8fe
@gabrieletijunaityte
Add s2bms prediction configs
c09c2e3
@gabrieletijunaityte
s2bms_experiment_1: aef 128
a3fa726
@gabrieletijunaityte
Remove missing tessera tiles from training split
b99e1d4
@gabrieletijunaityte
Remove missing tessera tiles from training split
4d01fc2
@gabrieletijunaityte
Add s2bms prediction configs
70134f2
@gabrieletijunaityte
s2bms_experiment_1: aef 128
da0f6a8
@gabrieletijunaityte
FIx data check for aef
0754317
@gabrieletijunaityte
For now deactivate tessera downloading
bbd57d2
@gabrieletijunaityte
Unified datasplit
4880073
@gabrieletijunaityte
Lower patience
99c2d56
@gabrieletijunaityte
LOg more frequently
4a4ed97
@gabrieletijunaityte
Allow "str" conversion in configurations
ae47cf9
@gabrieletijunaityte
Prediciton experiments: average encoder aef (1-2) and tessera (3-4) w…
945c5aa 
…ith lin or mlp pred-head
@gabrieletijunaityte
clean up
3c69908
@gabrieletijunaityte
Track wandb executed experiments
a15adcc
@gabrieletijunaityte
Add checkpoint dir configuration
1557577
@gabrieletijunaityte
Add checkpoint dir configuration
af4049a
@gabrieletijunaityte
Change where ckpt are saved and how they are logged to wandb and loca…
e913927 
…l csv log file
@gabrieletijunaityte
Longer training, larger lr
057fecd
@gabrieletijunaityte
garbage collection for oom
f715b40
@gabrieletijunaityte
Merge branch 'train/prediction' of github.com:WUR-AI/aether into trai…
71da776 
…n/prediction

# Conflicts:
#	configs/experiment/s2bms_experiment_1.yaml
#	src/data/base_dataset.py
@gabrieletijunaityte
restore object dict return
9ac6293
@gabrieletijunaityte
Remove missing tessera tiles from training split
00d6af9 
# Conflicts:
#	src/data/base_dataset.py
@gabrieletijunaityte
Remove missing tessera tiles from training split
3cefb8a
@gabrieletijunaityte
Add s2bms prediction configs
3eef6ae
@gabrieletijunaityte
s2bms_experiment_1: aef 128
d3cea69
@gabrieletijunaityte
FIx data check for aef
6e58eae
@gabrieletijunaityte
For now deactivate tessera downloading
6b438ba
@gabrieletijunaityte
Add s2bms prediction configs
ed43672
@gabrieletijunaityte
restore object dict return
6153d9b
@gabrieletijunaityte
Add dtype for avr geo_encoder
e1c955f
@gabrieletijunaityte
avr encoder with dtype float32
0f56726
@gabrieletijunaityte
Merge branch 'feature/fix_cnn' of github.com:WUR-AI/aether into train…
dd3ee9a 
…/prediction
@gabrieletijunaityte
Merge branch 'feature/batch_size_finder' of github.com:WUR-AI/aether …
3a40106 
…into train/prediction

# Conflicts:
#	src/data/base_datamodule.py
@gabrieletijunaityte
Merge branch 'feature/dtypes' of github.com:WUR-AI/aether into train/…
7885177 
…prediction
@gabrieletijunaityte
Visualisation of s2bms data split
5f5151c
@gabrieletijunaityte
Visualisation of s2bms data split
4ef8e48
@gabrieletijunaityte
spelling
34e543d
@gabrieletijunaityte
dtype param docstring
edf863a
@gabrieletijunaityte
Standard s2 normalisation + hooks
933ca79
@gabrieletijunaityte
Add cropping sanity check
dd7e0f5
@gabrieletijunaityte
Temp: comment out tessera missing data check: its handled through dat…
5b625a3 
…a splits
@gabrieletijunaityte
Add data_used into wandb projects, for cleaner sweep visualisation
de880ca
@gabrieletijunaityte
Print statements to see code latency
70845d2
@gabrieletijunaityte
Merge branch 'train/prediction' of github.com:WUR-AI/aether into trai…
2c364aa 
…n/prediction

# Conflicts:
#	notebooks/09-GT-aef-tessera-datasplit-filtering.ipynb
#	src/data/base_datamodule.py
#	src/data/base_dataset.py
#	src/train.py
#	src/utils/experiment_tracking.py
@gabrieletijunaityte
Prediction experiments
e15d515
@gabrieletijunaityte
Add splits for aef and tesera dataset union
bb9c3ca
@gabrieletijunaityte
Read aef from npy
454d42b
@gabrieletijunaityte
Log mse loss for the best model
8da67e4
@gabrieletijunaityte
Merge branch 'develop' of github.com:WUR-AI/aether into train/prediction
8a3390d
@gabrieletijunaityte
Add model config
b5fd5ca
@gabrieletijunaityte gabrieletijunaityte marked this pull request as ready for review May 27, 2026 09:33
vdplasthijs
vdplasthijs approved these changes May 27, 2026
View reviewed changes

@vdplasthijs vdplasthijs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really excellent work, the W&B cleaning looks very neat! I have two minor comments, have a look, but I'll approve it already so if those 2 things look good to you please go ahead and merge! Thanks!

Comment thread src/data/base_dataset.py Outdated
Comment thread src/data_preprocessing/aef_data_check.py
@gabrieletijunaityte gabrieletijunaityte merged commit e494d14 into develop May 28, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdplasthijs vdplasthijs vdplasthijs approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@gabrieletijunaityte @vdplasthijs