GitHub Issue NOAA-EMC/GSI#313. Store data files for global sources in tar files. #338
Conversation
|
@DavidHuber-NOAA if you have time I'd welcome your review. |
MichaelLueken
requested review from
DavidHuber-NOAA,
MichaelLueken and
dtkleist
|
@MichaelLueken-NOAA I'm guessing David is out on leave or otherwise unable to review this. I'm on leave next week. If you want to look this over and merge it I'll be around today to handle any problems you might spot. Otherwise you could wait until David can do a review and/or I'm back. Your call and I'm good either way. |
|
@EdwardSafford-NOAA I'll go ahead and clone the work and make sure that everything compiles correctly. I'll then be able to merge it to the authoritative repository next Tuesday (after the current work that was submitted earlier this week has been merged). I'll let you know if anything jumps out. |
|
@EdwardSafford-NOAA @MichaelLueken-NOAA I am not officially back from leave, but I will take a quick look today. |
| @@ -260,7 +307,10 @@ for sat in ${big_satlist}; do | |||
| (( ii=ii+1 )) | |||
| done | |||
|
|
|||
| if [[ ! $MY_MACHINE = "jet" ]]; then | |||
| if [[ $MY_MACHINE = "hera" ]]; then | |||
| $SUB --account ${ACCOUNT} -n $ii -o ${logfile} -D . -J ${jobname} --time=4:00:00 \ | |||
There was a problem hiding this comment.
Why is --mem=0 required? Wouldn't all of the memory for the node be made available anyway? Or is it different for MPMD tasks?
There was a problem hiding this comment.
Apparently there was an upgrade to the batch processing software on hera somewhere around March 1, and this process began running out of memory. You're right all the memory for the node used to be available by default but after the upgrade it apparently isn't. I didn't go too far into the weeds but learned that, counterintuitively, with --mem=0 maximum memory for the node is made available.
… files. Completes issue NOAA-EMC#313
EdwardSafford-NOAA
force-pushed
the
master
branch
from
837f06a
to
77fbc4d
Compare
|
And thanks @DavidHuber-NOAA for taking a look at this on your day off. That was very kind of you. |
MichaelLueken
changed the title
|
Since there are no source code changes to this update and @DavidHuber-NOAA has approved the changes, I will now give final approval to these changes and merge them to the authoritative repository. |
GitHub Issue NOAA-EMC#313. Store data files for global sources in tar files.
… Completes issue #313
The change in storage is primarily to help alleviate issues on hera where file count (not size) has been an ongoing issue for developers. Additional changes were made to fix build issues on wcoss2.