Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Clean up RDRS conversion module, create io module #115

Merged
merged 49 commits into from
Mar 30, 2023
Merged

Clean up RDRS conversion module, create io module #115

merged 49 commits into from
Mar 30, 2023

Conversation

Zeitsperre
Copy link
Collaborator

Pull Request Checklist:

  • This PR addresses an already opened issue (for bug fixes / features)
    • This PR fixes #xyz
  • Tests for the changes have been added (for bug fixes / features)
    • (If applicable) Documentation has been added / updated (for bug fixes / features)
  • HISTORY.rst has been updated (with summary of main changes)
    • Link to issue (:issue:number) and pull request (:pull:number) has been added

What kind of change does this PR introduce?

  • Splits setup code from rdrs modules into a template
  • Static typing
  • Refactored out several function-like blocks of code, apply existing functions where possible

Does this PR introduce a breaking change?

No (everything was already broken).

Other information:

@Zeitsperre Zeitsperre added the invalid This doesn't seem right label Mar 13, 2023
@Zeitsperre Zeitsperre self-assigned this Mar 13, 2023
@Zeitsperre
Copy link
Collaborator Author

@tlogan2000

I'm working my way through this, and it seems clear that we need an IO-handling module (similar things have been discussed for xscen). Do you have time to go over this with me? I'm going to try and pull out as much of the output-setup and file-writing code into their own functions and see whether I can integrate them into existing workflows and generalize them for multiple projects.

I also think it's time to create some development documentation to answer a few questions:

  • What corrections go where?
  • How do we gather files?
  • How do we line up jobs to be computed?

One approach that came to mind would be writing a wrapper for performing the setup for writing out files (gathering files, checking for tempfolder, etc.) so that users don't need to rewrite it all the time.

Ideally, I want to remove these monolithic conversion functions that have been rewritten 3–4 times now under various names.

@Zeitsperre Zeitsperre changed the title Clean up RDRS conversion module Clean up RDRS conversion module, create io module Mar 15, 2023
tlogan2000 and others added 15 commits March 16, 2023 11:12
@tlogan2000
add remarks attr
d4f5433
@tlogan2000
remove remarks fix
8b4b6f4
@tlogan2000
cleanup output rdrs
ec1709d
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
64bec4a 
for more information, see https://pre-commit.ci
@Zeitsperre
cleanup
e09697f
@Zeitsperre
smarter chunk handling
a8ab9c9
@Zeitsperre
rechunking adjustments
e595580
@Zeitsperre
typo
bab7de1
@tlogan2000
use main to allow dask.distibuted in pycharm
3fb90b0
@tlogan2000
avoid 'oveflow' into 2019; future proof to allow for incomplete years…
8aaaea3 
… via use of unique months in the dataset
@tlogan2000
adapt for rlon, rlat dims; allow correction of 2d lon COORDS not dims…
fce4c52 
…; use transpose dims var in sorting
@tlogan2000
add rdrs v2.1
a6f522a
@tlogan2000
add time dim check;
66a0e1a
@Zeitsperre
Merge branch 'fix_rdrs_integration' into fix_rdrs_integration_TL
6954fe2 
# Conflicts:
#	miranda/convert/eccc_rdrs.py
#	miranda/io/data/chunks.json
@Zeitsperre
cleanup
caa7b69
pre-commit-ci bot and others added 24 commits March 17, 2023 20:54
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
171b1f7 
for more information, see https://pre-commit.ci
@Zeitsperre
fix template
58df7fc
@Zeitsperre
no periods in rdrs-v21 project name
60d8a7f
@Zeitsperre
update CI
568baa3
@Zeitsperre
use strategy
c449282
@Zeitsperre
CI updates
4a32e8e
@tlogan2000
xr.set_options not xclim
824c5b1
@tlogan2000
get daily conversion working WIP
b76b80e
@tlogan2000
get daily conversion working WIP
691b1d0
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
74dbb34 
for more information, see https://pre-commit.ci
@tlogan2000
update_template
c375810
@Zeitsperre
support supplying a dictionary of facets
5887d4b
@Zeitsperre
WIP cleanup ECCC configurations
719c5d4
@Zeitsperre
cleanup gather_rdrs dict creation
0af3316
@tlogan2000
concat_zarr for rdrs
b6274c8
@tlogan2000
concat_zarr for rdrs
5d25d5e
@tlogan2000
Merge branch 'fix_rdrs_integration_TL' of github.com:Ouranosinc/miran…
351133e 
…da into fix_rdrs_integration_TL

Conflicts:
	templates/eccc_rdrs_processing.py
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
fbb08b8 
for more information, see https://pre-commit.ci
@tlogan2000
set rotated_pole as coord
11aa040
@tlogan2000
Merge remote-tracking branch 'origin/fix_rdrs_integration_TL' into fi…
2d93f7f 
…x_rdrs_integration_TL
@Zeitsperre
small adjustments
d4b3c63
@Zeitsperre
update HISTORY.rst
e1652b2
@Zeitsperre
Merge pull request #117 from Ouranosinc/fix_rdrs_integration_TL
b41dbc0 
Fix rdrs integration and CI updates
@Zeitsperre
Merge branch 'prepare_v040' into fix_rdrs_integration
18dfbf4
@Zeitsperre Zeitsperre marked this pull request as ready for review March 30, 2023 14:43
@Zeitsperre Zeitsperre merged commit e98d930 into prepare_v040 Mar 30, 2023
@Zeitsperre Zeitsperre deleted the fix_rdrs_integration branch March 30, 2023 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlogan2000 tlogan2000 Awaiting requested review from tlogan2000

Assignees

@Zeitsperre Zeitsperre

@tlogan2000 tlogan2000

Labels
invalid This doesn't seem right
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Zeitsperre @tlogan2000