Migration to NERSC

[zonca]

We have now a CMB-S4 collaboration endpoint on Perlmutter running Globus 5. "NERSC Perlmutter cmbs4 Collab"

However this works differently than before, instead of sharing only the gsharing folder on the community file system, this directly accesses the cmbs4 home folder.

Opened a ticket asking for help.

[zonca]

Docs seems outdated: https://docs.nersc.gov/services/globus/

[rpwagner]

@zonca I'm at NERSC this week. Let me know if I can help by talking to someone while I'm there.

[zonca]

@rpwagner yes, I think it would be a good opportunity to get in contact with Lisa Gerhardt if you haven't met her already, and you understand better the configuration of Globus we need for the portal.

[zonca]

@jdborrill: today @rpwagner and I had a chat with Lisa Gerhardt and Nick Tyler.
there are some security restrictions that affect the data portal:

1. Guest collections that allow users to manage permissions can only be created by a user account, not a collaboration account. So I should create the guest collection with my account and I will be the only one that can change permissions on the folders. In case someone else gets in charge of the portal, they will have to point the guest collection to the data portal root and reconfigure again permissions, no need to move data. Should I create the guest collection? or do we want to put someone else in charge of permissions? For permission here we mean which Globus groups access which folder. Multiple people can manage membership to the Globus group (which is actually automated now).
2. UCSD allows files shared via HTTPS to be directly accessible without logging in to Globus. NERSC instead requires to authenticate to any Globus account before downloading the files. To go around this we could keep the public data at UCSD and only keep the protected data at NERSC. Alternatively, Lisa suggested we could ask NERSC to change policy on this, it will take some months but it could be changed.

[jdborrill]

Hi Andrea, Thanks for this. 1. I'm happy for you to be in charge of the guest collection, as you suggest. 2. Can the same portal support data at multiple sites (ie. NERSC and SDSC), or would we need two portals? Julian

…

On Thu, Oct 26, 2023 at 12:57 PM Andrea Zonca ***@***.***> wrote: @jdborrill <https://github.com/jdborrill>: today @rpwagner <https://github.com/rpwagner> and I had a chat with Lisa Gerhardt and Nick Tyler. there are some security restrictions that affect the data portal: 1. Guest collections that allow users to manage permissions can only be created by a user account, not a collaboration account. So I should create the guest collection with my account and I will be the only one that can change permissions on the folders. In case someone else gets in charge of the portal, they will have to point the guest collection to the data portal root and reconfigure again permissions, no need to move data. Should I create the guest collection? or do we want to put someone else in charge of permissions? For permission here we mean which Globus groups access which folder. Multiple people can manage membership to the Globus group (which is actually automated now). 2. UCSD allows files shared via HTTPS to be directly accessible without logging in to Globus. NERSC instead requires to authenticate to any Globus account before downloading the files. To go around this we could keep the public data at UCSD and only keep the protected data at NERSC. Alternatively, Lisa suggested we could ask NERSC to change policy on this, it will take some months but it could be changed. — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC4LSRB4HCGC4PXHGMUXGDYBK6A5AVCNFSM6AAAAAA6OU5BLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRHAYTAMJXHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zonca]

1 portal will point to the 2 different sites, public datasets will point to UCSD, private to NERSC. Users will hardly notice.

[jdborrill]

OK - sounds like a temporary plan while we try to get NERSC to change its policy. In the long term we want everything in one place so we don't need to move things when they become public. In the short term this would mean that the CMB-S4 stuff would be at NERSC and the PySM PanEx skies at UCSD, correct? J

…

On Thu, Oct 26, 2023 at 1:57 PM Andrea Zonca ***@***.***> wrote: 1 portal will point to the 2 different sites, public datasets will point to UCSD, private to NERSC. Users will hardly notice. — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC4LSSQMSOLEOEBCABB2PLYBLFERAVCNFSM6AAAAAA6OU5BLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRHA4DINBXGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zonca]

Everything correct. I'll get started and share updates here.

…

On Thu, Oct 26, 2023, 14:01 Julian Borrill ***@***.***> wrote: OK - sounds like a temporary plan while we try to get NERSC to change its policy. In the long term we want everything in one place so we don't need to move things when they become public. In the short term this would mean that the CMB-S4 stuff would be at NERSC and the PySM PanEx skies at UCSD, correct? J On Thu, Oct 26, 2023 at 1:57 PM Andrea Zonca ***@***.***> wrote: > 1 portal will point to the 2 different sites, public datasets will point to > UCSD, private to NERSC. > Users will hardly notice. > > — > Reply to this email directly, view it on GitHub > < #28 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAC4LSSQMSOLEOEBCABB2PLYBLFERAVCNFSM6AAAAAA6OU5BLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRHA4DINBXGY> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC5Q4SRWQDVDWXNVLVQ553YBLFUDAVCNFSM6AAAAAA6OU5BLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRHA4DSMRYHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zonca]

started to work on moving DC-0 LAT in #29

[zonca]

ok, it worked fine.
Now DC-0 LAT is served from NERSC, see all the links in https://data.cmb-s4.org/dc0-chlat-split01-025.html start with g-9f, everything else is still served through UCSD, links start in g-45.

I'll move the rest of DC-0 once we have the new maps, see #27.
Planck and Panexp skies remain at UCSD.

[zonca]

we can now move public data to NERSC, see #31