Skip to content

Outputs filtering#128

Merged
Samuel-amap merged 152 commits intomainfrom
Outputs-filtering2
Apr 14, 2025
Merged

Outputs filtering#128
Samuel-amap merged 152 commits intomainfrom
Outputs-filtering2

Conversation

@Samuel-amap
Copy link
Copy Markdown
Collaborator

@Samuel-amap Samuel-amap commented Jan 22, 2025

List of mini-API changes :

  • Removed vars_not_propagated and init_fun on the ModelList side, and changed the status to be a Status object, instead of a TimeStepTable. This status can contain vectors.
  • Added outputs to run! in modellist mode, as a kwarg
  • This run! Returns a TimeStepTable{Status} with filtered outputs
  • Added an outputs function for sink conversion (similar to the one for GraphSimulation objects)
  • Removed/changed some of the run! functions (and renamed the leaf ones to run_node!/run_node_multiscale!).
  • Removed multi object parallelisation for now
  • Removed parallelisation check in multiscale mode
  • Changed default behaviour for outputs in multiscale (and handle the NamedTuple() case) and renamed outputs kwarg
  • Made the run! function return the outputs instead of a GraphSimulation object in multiscale mode
  • Renamed the outputs(graphsim, sink; ...) function (which stopped acting on the graphsim due to changes to run!) to conver_outputs(sim_outputs::Dict{String, O} where O, sink; ...) and same for the modellist. Avoids naming conflicts, better reflects current usage.
  • Changed 'mapping' kwarg for multiscalemodels to variable_mapping or something along those lines, the name conflicts with the overarching mapping

Other changes :

  • Added a test that combines a meteo, a modellist and a list of outputs and runs it in ModelList mode, then converts to multiscale. Detected a few API weaknesses this way, and validated filtering of the outputs in monoscale.
  • Added a ModelList save_results equivalent, and a pre_allocate_outputs equivalent
  • Fixed tests affected by the changes (adjustments for PBP and XPalm also done locally, but not in their PR yet)
  • Added a function that adjusts meteo timestep size to the status. This needs a little more work.
  • Added a function that takes a status and provides a flattened one (ie replaces vectors by their initial value) for actual simulation, and another that updates the value each timestep.
  • Added tests to detect regressions for BoundsError with a single model and several Weather timesteps #86, No outputs when simulating a mapping with one meteo timestep #105 and Multiscale : outputs not saved when dependency graph only has one depth level #111, which should be fixed by this PR.
  • Added a naive test that tries different meteo/modellist-mapping/outputs combos and expects either a successful run! or a dimensional mismatch.
  • PBP and XPalm tests updated to match API changes (with a related PR).
  • Improved weather handling in multiscale (there are still some quirks with the current way of doing things but there will likely be a new pass in the future with varying timesteps)
  • Reactivated downstream CI tests (and benchmark in PSE) now that XPalm release is done and the package is registered
  • Executor inconsistency (incorrectly overriding user-provided SequentialEx() value in some cases) will be fixed when parallelisation is reimplemented, since multi-object parallelisation is currently deactivated, that param is mostly unused.
  • Added a test to PBP for uncertainty propagation
  • Find a way to properly distinguish between public API, private but 'back pocket useful for tinkerers' API, and developer functions
  • Properly decide what to do regarding organ creation and the way models act on it or not in the same timestep, and document it
  • Changed parallelization in modellist mode slightly to deal with API changes and new output structure

Remaining task list :

  • Create a checklist of things to look out for for releases (benchmarks, specific documentation pages that have been added recently might be easily forgotten otherwise)
  • Check to see how downstream/Integration test break, and see how we want to do things for future releases.
  • Documentation update
  • API reference update, document the new internal functions
  • Export the new toy plant examples
  • Add the 'unstable' badge to the gh display

Not in this PR :

  • Rename outputs kwarg on GraphSimulation ? Might not be necessary since it's an internal object.
  • Implement check_dimensions for multiscale simulations ?
  • Provide more examples
  • Distinguish between Toy and Example
  • Some things that'd be nice to have for the documentation : proper graph visualisation, a nice example showcasing the dependency graph and the way it acts on the MTG
  • Delete some code (dataframe.jl ?) -> into its own PR
  • Changes to PBP documentation, some pages are outdated.
  • Change default output structure #123 Improve the performance of DataFrame conversion of outputs in multiscale mode (might go in its own PR). The XPalm benchmark has been set up, but might need to delete the old benchmark graph and fix up the path to the meteo data file before work can get started
  • Reintroduce multi-object parallelization

@Samuel-amap Samuel-amap changed the title Extra mapping with status vector test Outputs filtering Jan 23, 2025
@Samuel-amap
Copy link
Copy Markdown
Collaborator Author

Extra mapping with status vector test

…confusion between runners with different performance characteristics
…istics, but it seems like overkill anyway as CI already has 1.9 tests
@Samuel-amap Samuel-amap merged commit 7e352d4 into main Apr 14, 2025
19 of 20 checks passed
@Samuel-amap Samuel-amap linked an issue Apr 14, 2025 that may be closed by this pull request
@Samuel-amap Samuel-amap deleted the Outputs-filtering2 branch April 14, 2025 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants