Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downsample data for climate project (hw03) #6

Open
facusapienza21 opened this issue Feb 19, 2023 · 2 comments
Open

Downsample data for climate project (hw03) #6

facusapienza21 opened this issue Feb 19, 2023 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@facusapienza21
Copy link
Member

Problem

Last Spring 2022 a lof of students were having problems running the climate dataset in the Hub. The problem was just a memory issue, which didn't really affect what the students where asked to do in the context of their homework. This homework was the most challenging for the students last semester, so it will be good to remove this difficulty.

Solution

I think we can just pre-process the dataset before sending them to the students and just downgrande the scale of the xarray objects. This will affect the spatial resolution of the plots, but for the purposes of the project I don't think it will have major impacts. Also it will be nice to have the code running faster.

What do you think @fperez ? Also, @isaacdsloan do you remember any other issue regarding this homework? I think the memory issue was the largest bottleneck in this project.

@facusapienza21 facusapienza21 added the bug Something isn't working label Feb 19, 2023
@facusapienza21 facusapienza21 self-assigned this Feb 19, 2023
@fperez
Copy link
Contributor

fperez commented Feb 21, 2023

Yup, agreed on downsampling to limit memory pressure - that problem is somewhat orthogonal to the key course content, so we don't need to burden them with it.

Please double-check that a clean solution runs without issue (while monitoring memory pressure) to be sure the fix works.

Thanks!!

@isaacdsloan
Copy link

@facusapienza21, love the solution, this will be a live saver for the students! Sounds like it will tackle the main issue of memory as well as run time for building the plots. For me, the biggest headache was the speed of building the plots. This made it difficult to try different things in a timely manner. Let me know if you need help testing out the speed of building the plots as well as checking for any potential memory issues when loading data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants