hercules baseline fail because of permissions

[DeniseWorthen]

Description

I attempted to run the RT against baseline /work/noaa/epic/hercules/UFS-WM_RT/NEMSfv3gfs/develop-20231002 to verify one last time for PR #1894. All tests failed with "missing baseline". I believe this is due to permission issue on the baseline directory.

[jkbk2004]

drwxrwsr-x 166 jongkim epic 16384 Oct 2 07:12 develop-20231002 @DeniseWorthen can you try ?

[DeniseWorthen]

Now I can't run on orion either:

ls: cannot open directory /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/develop-20231002/: Permission denied
+ [[ 0 -lt 1 ]]
+ echo 'Baseline directory is empty:'
Baseline directory is empty:
+ echo '   /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/develop-20231002'
   /work/noaa/epic/UFS-WM_RT/NEMSfv3gfs/develop-20231002
+ exit 1
+ echo 'rt.sh finished'

[DeniseWorthen]

On Orion, I can ls the first of these directories, the second is permission denied

drwxr-x--- 134 zshrader noaa-hpc 16384 Sep 29 11:17 develop-20230928
drwxr-s--- 129 jongkim  epic     16384 Oct  2 07:11 develop-20231002

[zach1221]

On Orion, I can ls the first of these directories, the second is permission denied

drwxr-x--- 134 zshrader noaa-hpc 16384 Sep 29 11:17 develop-20230928
drwxr-s--- 129 jongkim  epic     16384 Oct  2 07:11 develop-20231002

so you can access develop-20230928 but not 20231002?

[zach1221]

Can you try again, on both Orion and Hercules @DeniseWorthen ?

[DeniseWorthen]

I was able to run the complete RT on hercules now and the first orion test has sucessfully completed so I think orion is also ok now.

[zach1221]

I was able to run the complete RT on hercules now and the first orion test has sucessfully completed so I think orion is also ok now.

Awesome! I'll keep a watchful eye on those permission going forward.

[natalie-perlin]

@jkbk2004 @zach1221
Permissions would be a recurrent issue, due to default permissions set to "---" for other users (non-epic group).
My suggestion is to set current and default "r-x" permissions for /work/noaa/epic/UFS-WM_RT/ and for most of the /work/noaa/epic/role-epic directories (contrib, spack-stack, landda, jenkins). This is done using setfacl utility.
Please let me know if this would be a helpful step to do, and if there are any particular directories that would benefit from that update.
At this point, I'm setting current and default permissions for the contrib and for spack-stack directories, recurrently, due to continuous issues with stacks (with no default "r-x" permissions set, any new directories and files would have "---" for the "other" users)