Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Miscellaneous improvements for 2024-W14 #162

Merged
merged 35 commits into from Apr 9, 2024
Merged

Miscellaneous improvements for 2024-W14 #162

merged 35 commits into from Apr 9, 2024

Conversation

khaeru
Copy link
Member

@khaeru khaeru commented Apr 3, 2024

  • Migrate handing code for 4 sources of exogenous data from message_data.
    • Global Energy Assessment project.
      • Add fuzzed data for testing.
    • Global Fuel Economy Initiative.
      • Add fuzzed data for testing.
    • IEA Energy Efficiency Indicators.
    • SHAPE project.
      • Add fuzzed data for testing.
  • Improve docs of all of the above.
  • Add path_fallback(), generalized from a utility function used in message_data/MESSAGEix-Transport.

Housekeeping:

  • Adjust MANIFEST.in after Migrate legacy reporting #159.
  • Remove snapshot files also available from Zenodo; these were committed in Migrate legacy reporting #159.
  • Use nschloe/action-cached-lfs-checkout to reduce Git LFS bandwidth usage.
    • As we accumulate more Git LFS files (particularly, test data) in this repo, we are consuming the 300 GB monthly quota more and more quickly. One reason is that we currently run 18 jobs (6 upstream versions × 3 OSes), so even 100 MB of Git LFS files means a 1.8 GB hit towards that quota
    • This added, third-party action seems to be a response to others encountering the same issue. It appears the most developed, but is not 100% up to date; see some pending PRs https://github.com/nschloe/action-cached-lfs-checkout/pulls
    • We could mitigate by creating a fork or using someone else's.
  • Configure Codecov to allow coverage to drop by 0.5% > 0.46% = 90 lines. This drop will (currently) always appear on PRs, because the snapshot and legacy reporting tests are only run on the main branch and a schedule trigger.
  • Tidy a lot of nitpicks in docs.

How to review

Read the diff and note that the CI checks all pass.

PR checklist

  • Continuous integration checks all ✅
  • Add or expand tests; coverage checks both ✅
  • Add, expand, or update documentation.
  • Update doc/whatsnew.

@khaeru khaeru added the enh New features or functionality label Apr 3, 2024
@khaeru khaeru self-assigned this Apr 3, 2024
khaeru added a commit that referenced this pull request Apr 5, 2024
Copy link

codecov bot commented Apr 8, 2024

Codecov Report

Attention: Patch coverage is 99.61832% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 75.5%. Comparing base (ad7d356) to head (7dc5385).

Additional details and impacted files
@@           Coverage Diff           @@
##            main    #162     +/-   ##
=======================================
- Coverage   76.0%   75.5%   -0.5%     
=======================================
  Files         94     101      +7     
  Lines       6428    6675    +247     
=======================================
+ Hits        4890    5046    +156     
- Misses      1538    1629     +91     
Files Coverage Δ
message_ix_models/model/bare.py 100.0% <ø> (ø)
message_ix_models/model/build.py 100.0% <ø> (ø)
message_ix_models/model/disutility.py 100.0% <ø> (ø)
message_ix_models/model/emissions.py 98.0% <ø> (ø)
message_ix_models/model/macro.py 100.0% <ø> (ø)
message_ix_models/model/structure.py 100.0% <ø> (ø)
message_ix_models/project/shape/data.py 100.0% <100.0%> (ø)
message_ix_models/report/__init__.py 97.4% <100.0%> (+<0.1%) ⬆️
message_ix_models/report/compat.py 99.2% <ø> (ø)
message_ix_models/report/config.py 100.0% <ø> (ø)
... and 31 more

... and 4 files with indirect coverage changes

khaeru added 23 commits April 8, 2024 11:53
- From message_data.tools.gfei_fuel_economy.
- Converted to an ExoDataSource.
- From message_data.tools.gea.
- Simplified using ExoDataSource.
- Migrated from message_data.tools.iea_eei.
- Streamlined and improved using ExoDataSource.
- Add "Turkey" to .util.pycountry.
- Add standard .aggregate, .interpolate source kwargs to control
  transform().
- Add iamc_like_data_for_query() kwargs:
  - drop, to drop certain non-IAMC columns that may appear.
  - unique, to override default behaviour of requiring 4 dimensions
    contain unique values.
  - Other kwargs passed to pd.read_csv(); allow to override
    engine="pyarrow".
- Migrated from message_data.projects.shape.
- Improved using ExoDataSource.
- Also migrate docs, tests.
Always fetch these from Zenodo; avoid using LFS storage/bandwidth quota.
khaeru added a commit that referenced this pull request Apr 8, 2024
- Add to whatsnew.
- Split project docs to separate pages.
@khaeru khaeru force-pushed the enh/2024-W14 branch 2 times, most recently from 83a3b0e to 5d79051 Compare April 8, 2024 12:52
khaeru added a commit that referenced this pull request Apr 8, 2024
- Add to whatsnew.
- Split project docs to separate pages.
- Add configuration.
- Address ~270 of ~450 nitpick warnings in Sphinx build.
- Remove custom ReST aliases.
- Remove some nitpick_ignore_regex entries.
Copy link
Member

@glatterf42 glatterf42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for these manifold improvements :)

@khaeru khaeru merged commit 76f58d6 into main Apr 9, 2024
26 checks passed
@khaeru khaeru deleted the enh/2024-W14 branch April 9, 2024 07:25
khaeru added a commit that referenced this pull request Apr 10, 2024
Was incorrectly calculated in #162.
@khaeru khaeru mentioned this pull request Apr 19, 2024
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh New features or functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants