Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add mfdataset post #238

Merged
merged 5 commits into from
Aug 17, 2023
Merged

Add mfdataset post #238

merged 5 commits into from
Aug 17, 2023

Conversation

dcherian
Copy link
Contributor

No description provided.

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

katiedagon
katiedagon reviewed Aug 15, 2023
View reviewed changes
posts/2023/mfdataset.ipynb Show resolved Hide resolved
posts/2023/mfdataset.ipynb Show resolved Hide resolved
posts/2023/mfdataset.ipynb Show resolved Hide resolved
posts/2023/mfdataset.ipynb Show resolved Hide resolved
@katiedagon
Copy link
Contributor

@dcherian this looks like a very useful blog post!

@dcherian dcherian marked this pull request as ready for review August 17, 2023 00:19
@dcherian dcherian requested a review from a team as a code owner August 17, 2023 00:19
@dcherian dcherian requested review from matt-long and erogluorhan and removed request for a team August 17, 2023 00:19
erogluorhan
erogluorhan approved these changes Aug 17, 2023
View reviewed changes
Copy link
Contributor

@erogluorhan erogluorhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great post; thanks for putting it together!

@dcherian dcherian merged commit dfa20fb into NCAR:main Aug 17, 2023
1 check passed
@dcherian dcherian deleted the mfdataset branch August 17, 2023 16:23
katiedagon
katiedagon reviewed Aug 17, 2023
View reviewed changes
posts/2023/mfdataset.ipynb
@@ -0,0 +1,5675 @@
{
Copy link
Contributor

@katiedagon katiedagon Aug 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the single file is more like ~2GB now? Or perhaps this is going to be updated with the new CESM2 LE file location


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah not sure what happened. Thanks!

posts/2023/mfdataset.ipynb
@@ -0,0 +1,5675 @@
{
Copy link
Contributor

@katiedagon katiedagon Aug 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above it mentions a chunk size of 16 lat and 32 lon, but the code here is chunking in time only?


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed now. thanks!

posts/2023/mfdataset.ipynb
@@ -0,0 +1,5675 @@
{
Copy link
Contributor

@katiedagon katiedagon Aug 17, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo "becasue"


Reply via ReviewNB

@dcherian
Copy link
Contributor Author

Ah the files have now been truncated to just 2000-2023 I'll rewrite to hit the "official" CESM2-LE timeseries dataset

@katiedagon
Copy link
Contributor

@dcherian I don't think these updates made it to the published version. I'm happy to work on that in the next week or so if that would be helpful.

@dcherian
Copy link
Contributor Author

dcherian commented Sep 9, 2023

Yes that would be great. I looked at it and it will need major updates: you'll have to concatenate along two dimensions: "member_id" and "time" with the timeseries files.

@katiedagon
Copy link
Contributor

Indeed, not having one timeseries file per ensemble member does make this more involved! Plus, the historical decades (1850-2014) and the SSP37 projection decades (2015-2100) are labeled differently in /glade/campaign/cgd/cesm/CESM2-LE/timeseries/atm/proc/tseries/day_1/PRECT/ such that sorted puts the historical decades first for each ensemble member, then listing SSP37. And there are other complications for the labeling of macro initializations...

Would it be easier to use the CESM2-LE catalog here as in previous blog posts? Though I'm not sure how updated that is, or whether that demonstrates the key messages of this blog post which are (I think) the various options of mf_dataset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@katiedagon katiedagon katiedagon left review comments

@erogluorhan erogluorhan erogluorhan approved these changes

@matt-long matt-long Awaiting requested review from matt-long matt-long is a code owner automatically assigned from NCAR/esds-core

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dcherian @katiedagon @erogluorhan