Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up data processing via use of 'cfp' #2

Open
GeorgeGayno-NOAA opened this issue Jan 3, 2020 · 0 comments
Open

Speed up data processing via use of 'cfp' #2

GeorgeGayno-NOAA opened this issue Jan 3, 2020 · 0 comments
Assignees
@GeorgeGayno-NOAA
Labels
enhancement New feature or request

Comments

@GeorgeGayno-NOAA
Copy link
Collaborator

The gldas_get_data.sh script processes its data serially by time period. This can be optimized by processing all time periods in parallel using 'cfp' on Dell and Cray and its equivalent on Hera.
@GeorgeGayno-NOAA GeorgeGayno-NOAA self-assigned this Jan 3, 2020
@GeorgeGayno-NOAA GeorgeGayno-NOAA added the enhancement New feature or request label Jan 3, 2020
GeorgeGayno-NOAA added a commit that referenced this issue Jan 3, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
e98a7b7 
Place data processing statments from 'gldas_get_data.sh' into
its own script - 'gldas_process_data.sh'.  This will facilitate
the use of 'cfp' for parallel processing.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 3, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
a613957 
Update "gldas_get_data.sh" to use 'cfp' on Dell.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 3, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
fe05c9c 
Update "gldas_get_data.sh" for Cray.  And use the correct
environment variable for machine.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 6, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
73cfdcd 
Update "gldas_get_data.sh" to process data in parallel on Hera.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 7, 2020
@GeorgeGayno-NOAA
feautre/cfp: This commit references #2.
ff1cd63 
Merge branch 'master' into feature/cfp
GeorgeGayno-NOAA added a commit that referenced this issue Jan 10, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
e970ebe 
Merge branch 'master' into feature/cfp
GeorgeGayno-NOAA added a commit that referenced this issue Jan 23, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
f246f55 
Merge branch 'master' into feature/cfp
GeorgeGayno-NOAA added a commit that referenced this issue Jan 30, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
8fad1ca 
Speed up 'copygb' processing in 'exgdas_gldas.sh.ecf' and
'gldas_forcing.sh' thru use of 'cfp'.  Only tested on
Dell.  Other machines still process serially.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 30, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
ded8627 
Merge branch 'master' into feature/cfp
GeorgeGayno-NOAA added a commit that referenced this issue Jan 31, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
e74478a 
Reduce the number of APRUN environment variables by
using APRUN_GLDAS_DATA_PROC in all scripts which use
cfp.
GeorgeGayno-NOAA added a commit that referenced this issue Jan 31, 2020
@GeorgeGayno-NOAA
feature/cfp: This commit references #2.
d1a2e82 
Merge branch 'master' into feature/cfp
@HelinWei-NOAA HelinWei-NOAA mentioned this issue Apr 18, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GeorgeGayno-NOAA GeorgeGayno-NOAA

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@GeorgeGayno-NOAA