resolves #75#76
Open
sahiljhawar wants to merge 8 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds new “fill” modes to the solar-wind multi-model reader (27‑day recurrence without truncation and a new 10‑year-average fill option), and speeds up OMNI high‑resolution monthly ingestion by parallelizing downloads when many files are needed.
Changes:
- Add
fill_averageoption and 10‑year average constants for filling future values. - Prevent end-time truncation when using recurrence or average-fill modes.
- Parallelize OMNI high-res download/process when more than 10 monthly files are required, with new tests for the parallel/sequential boundary.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
swvo/io/solar_wind/read_solar_wind_from_multiple_models.py |
Adds fill_average, adjusts truncation behavior, and applies recurrence/average filling. |
swvo/io/solar_wind/__init__.py |
Exposes AVERAGE_VALUES_TO_FILL constant from the package. |
tests/io/solar_wind/test_read_solar_wind_from_multiple_models.py |
Adds tests for non-truncation and average-fill behavior, plus mutual exclusivity. |
swvo/io/omni/omni_high_res.py |
Adds thread-based parallel download/process for large monthly ranges and factors out per-file processing. |
tests/io/omni/test_omni_high_res.py |
Adds tests ensuring parallelization triggers only when expected. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+180
to
+185
| for col, avg_value in AVERAGE_VALUES_TO_FILL.items(): | ||
| if col in df.columns: | ||
| future_mask = df.index > historical_data_cutoff_time | ||
| df.loc[future_mask, col] = avg_value | ||
| df.loc[future_mask, "model"] = "10_year_average_filled" | ||
| df.loc[future_mask, "file_name"] = "10_year_average_filled" |
| If True, fill missing values using 27-day recurrence from historical models (OMNI, ACE, SWIFT). | ||
| Defaults to False. | ||
| fill_average : bool, optional | ||
| If True, keep the final dataframe through the requested end time for average-based filling. |
Comment on lines
+226
to
+233
| monkeypatch.setattr(READ_SW_MODULE, "_read_from_model", lambda *args, **kwargs: data) | ||
| monkeypatch.setattr(READ_SW_MODULE, "_recursive_fill_27d_historical", lambda df, *_args: df) | ||
|
|
||
| result = read_solar_wind_from_multiple_models( | ||
| start_time=start_time, | ||
| end_time=end_time, | ||
| model_order=[SWOMNI(), SWSWIFTEnsemble()], | ||
| historical_data_cutoff_time=historical_data_cutoff_time, |
Collaborator
Author
|
TODO: @sahiljhawar: test when SWIFT is non-nan |
- Remove parameters "\`recurrence\`" and "\`rec_model_order\`" and delete "\`_recursive_fill_27d_historical\`" - Change average filling to only fill rows where all numeric columns are NaN and mark them as "10_year_average_filled" - Truncate interpolation using the final model index in "\`_interpolate_to_common_indices\`" and log SWIFT end times - Adjust logging calls (use "\`logging.info\`" for average fill)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.