Add model_1702 comparison harness (6-class) - #49
Merged
Conversation
Replace fractions_skill_score as the training objective with a proper scoring rule: the MSE between observed and forecast neighborhood fractions (FSS's numerator without the hedgeable skill-score normalization), summed over 25/100/250 km tolerance scales. Zonal windows widen by 1/cos(lat) so tolerance stays isotropic in km, and domain edges use valid-cell normalization (no reflect or wrap; the longitude strip ends are not adjacent). No sigmoid discretization, which would break propriety. The 25 km scale reproduces the old 1-pixel label dilation, so front_dilation is removed from the dataset, evaluation, and configs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This reverts commit bdcb14e.
…-domain transfer Adds ModelConfig.pretrained_weights_path/freeze_layer_prefixes and train.py helpers to load a prior checkpoint's weights, reset input_normalization to the current run's domain stats, and optionally freeze layers by name prefix before compiling. Includes a schooner_train_full_pretrained_3d.yaml config wiring this up for a full-domain fine-tune from a CONUS-trained checkpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The full domain has no dryline labels because label coverage doesn't extend there, not because drylines don't occur, so leaving that class weighted would actively train the model toward "dryline never happens" and erode what the CONUS checkpoint already learned for it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The eval config was never updated for the brier_conus_3d_comparison 3D U-Net: it was missing volume_inputs (defaulting to the flattened 2D path), listed the wrong variable set/order, and pointed at the full-domain coordinates instead of the CONUS box the model trained on. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… add trough/tropical-trough/instability Forming (CF-F..OF-F) and dissipating (CF-D..OF-D) front codes now collapse into their parent front class instead of being dropped as background. Trough (TROF), tropical trough (TT), and instability axis (INST) become new independent classes 6, 7, and 8, bringing the total to 9 (was 6: background + CF/WF/SF/OF/DL). n_classes, class_weights, loss_class_weights, and front_types are updated across all training/eval/pipeline configs. The CONUS->full-domain warm-start config keeps skip_mismatch=True so the 6-class CONUS checkpoint's output layer is dropped and the 3 new classes train from scratch alongside the rest of the fine-tuned decoder. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lback callbacks.py had its own local 5-class FRONT_TYPE_CLASS_INDEX that was missed when the rest of the codebase (targets.py, evaluate.py, plot.py, configs) was expanded to 9 classes. Training-time visualization images and lite performance-diagram stats were silently excluding TROF/TT/INST. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Add trough/tropical-trough/instability-axis to training-viz callback
…U per batch model.predict() batches its forward passes internally but still accumulates every batch's output into one GPU-resident tensor before returning. On full-CONUS-domain training runs (large spatial extent x 200-timestep visualization subsample x 9 front classes), that accumulated buffer on top of training's already-resident GPU memory reliably OOMs (observed crash: OOM in ConcatV2 inside TestVisualizationCallback._predict at shape[200,320,960,9]). Loop over predict_on_batch and move each batch to CPU immediately instead, so only one batch's output is ever GPU-resident at a time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix GPU OOM in TestVisualizationCallback (stream predict to CPU per batch)
Runs the legacy AIES FrontFinder checkpoint (Keras 2.10 h5) through the 2.0 eval/plot machinery for a head-to-head against current models on identical 2019 test data: - loader: patches the stored model_config (TFOpLambda squeeze -> LevelSqueeze, standard inbound-node format), rebuilds under Keras 3, loads weights by name with full weight-consumption verification (8,745,968 params, 4 softmax heads) - legacy_formulas: verbatim numpy port of utils/variables.py@c59ac2b with docstring golden-value tests (derived.py diverges: Lv 2.501e6 vs 2.257e6) - normalization: embedded min-max table, parity-tested against model_1702_properties.txt - adapter: raw (batch, lat, lon, level, var) -> legacy normalization -> lon-major transpose -> sup1 head -> 6-to-9-class zero padding, matching evaluate.compute_stats' contract; ClassPaddingAdapter for 6-class baselines - store: side icechunk store generation (surface+1000/950/900/850, legacy units) from Arraylake ERA5 or ARCO fallback, resume-safe batched writes - run_eval: config-driven driver for both model kinds over full/land/ocean and the five office regions (with wrap-crossing longitude fix), writing evaluate-compatible stats_*.nc - figures/compare: batch 4-panel rendering and cross-model max-CSI summary with out-of-domain footnotes Zero core-module edits; verified end-to-end on the 2019-03-13 bomb cyclone (coherent CF/WF/DL structures from a real ARCO ERA5 timestep). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fronts.aies1702.case_study renders multi-panel model_1702 prediction maps in the paper's presentation: filled front-probability contours at 10% intervals (CF/WF/SF/OF cmaps from fronts.plot), state-border backgrounds via plot_background, yellow a)/b)/c) panel labels, and per-panel UTC captions. Inputs are derived on the fly from Arraylake or ARCO ERA5 with the legacy formulas and cached to NetCDF so repeat renders skip the fetch entirely. Ships configs/aies1702/case_study_xmas2023.yaml reproducing the 2023 Christmas storm panels (00Z/12Z 26-27 Dec 2023); verified end-to-end locally against the real checkpoint — panel structures match the paper's Fig. 14, including the d-panel occlusion remnants over the UP of Michigan and the Midwest called out in the text. Raw (uncalibrated) probabilities from ERA5 analysis vs the paper's calibrated GFS f000, noted in the figure title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…script The Arraylake earthmover-public ERA5 repo lacks the 950 and 900 hPa levels model_1702 requires, so both generate configs now point at the anonymous Google ARCO ERA5 zarr (no auth needed). scripts/aies1702_comparison.sbatch runs the whole 2019 comparison in one job: side-store generation (resume-safe), all five evaluations, figures, and the comparison summary. Resubmit-safe — completed eval stages are skipped via their final region's stats_derived marker file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The artifact under comparison is model_1702, so the package, tests, configs, SLURM script, model_kind string, keras serialization package, render-test env var (MODEL_1702_RENDER_TESTS), and all schooner paths now use model_1702. No functional changes; 83 tests pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
harness/model_1702-comparison was branched from feat/include-forming-dissipating-and-troughs, so every shared file it touched carried that branch's 9-class scheme (TROF/TT/INST plus forming/dissipating collapse) even though the actual model_1702 comparison harness itself never needed more than the original 6 classes. Reverts config.py, model.py, plot.py, targets.py, and the schooner_*.yaml training configs to their feat/2.0.0 state; keeps the callbacks.py predict_on_batch OOM fix and the evaluate.py predict_batches/accumulate_stats split plus numpy optimizations, which are unrelated to class count; and fixes the new model_1702/ package (adapter.py, eval configs, tests) to assume 6 classes throughout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resolves a conflict in evaluate.py by keeping both refactors: this branch's compute_stats/predict_batches/accumulate_stats split and feat/2.0.0's load_eval_arrays extraction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> # Conflicts: # src/fronts/evaluate.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
src/fronts/model_1702/,configs/model_1702/,scripts/model_1702_*.sbatch,tests/model_1702/) for evaluating the legacy AIES FrontFinder model_1702 head-to-head against 2.0 checkpoints.evaluate.py'spredict_batches/accumulate_statssplit, numpy vectorization optimizations in the TP/FP/TN/FN accumulation loop, and theTestVisualizationCallbackOOM fix (loop overpredict_on_batchinstead of onemodel.predict()call).feat/include-forming-dissipating-and-troughs, so it carried that branch's 9-class scheme (TROF/tropical-trough/instability-axis + forming/dissipating collapse) into every shared file it touched. That's been stripped back out —config.py,model.py,plot/plot.py,targets.py, their tests, and allschooner_*.yamltraining configs now matchfeat/2.0.0exactly (6 classes, no troughs, no forming/dissipating).Test plan
pixi run pytest -q— 403 passed, 48 skipped, 1 pre-existing unrelated failure (ModuleNotFoundError: No module named 'wandb', also fails on stockfeat/2.0.0in this env)git grepfor trough/forming/dissipating terms across the tree returns nothing outside of unrelated word matchesModelConfig.n_classes == 6and no config file has a >6-lengthclass_weights/front_typeslist🤖 Generated with Claude Code