Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dask for memory usage #35

Merged
merged 27 commits into from
Jan 23, 2023
Merged

Dask for memory usage #35

merged 27 commits into from
Jan 23, 2023

Conversation

noahprime
Copy link
Collaborator

No description provided.

@noahprime
Using Dask implementation to run code in parrallel in chunks
d5da9a0
@noahprime
Updating notebooks
e3dd383
@noahprime
Updating Dask reqs
6c69e58
@noahprime
Adding progress bar
3704b3c
@noahprime
Ordering data dimensions
0de6fe1
@noahprime
Using xarray save_mfdatasets for writing out more efficiently
9219a81
@noahprime
Setting write mode to overwrite
b16f894
@noahprime
Reworking scripts to make use of Dask
11095ce
@noahprime
Forcing computation to take place at given time to avoid redundant co…
de76832 
…mputations later
@noahprime
Adding different option for saving output
96a6751
@noahprime
Updated size and shaped of chunks to be consistent with Dask reqs
646e1b6
@noahprime
Fixed indexing typo
b383a27
@noahprime
Specifying dtype for NetCDF output
be48441
@noahprime
Fixing improper transpose
34ca37a
@noahprime
Working through bugs
aec1e84
@noahprime
Bad reshape
1e45872
@noahprime
Bad repeat
7b314a8
@noahprime
Re-ordering variables
1d7ddee
@noahprime
Removing print statement
be21069
@noahprime
Continuing to debug data shape issues
8a28493
@noahprime
Removed print
aa75044
@noahprime
Bug fixes
4b5f776
@noahprime
explicit data type info for saving to netCDF
7cd9618
@noahprime
Adding encoding option for saving files
b1fdff4
@noahprime
Adding encoding option for saving files
946ad39
@noahprime
Placing persist
38d184c
@noahprime
Updating notebook
ea506e8
@noahprime noahprime merged commit 18cfeb2 into statistical-downscaling Jan 23, 2023
@noahprime
Copy link
Collaborator Author

Checks failing due to issues installing xesmf package via conda on github. Noted, but not an issue with the branch.

@noahprime noahprime deleted the dask_for_memory_usage branch January 23, 2023 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@noahprime