Set up compute and data resources to support the VO #4

backeb · 2021-05-10T07:32:14Z

Set up compute and data resources to support the VO

backeb · 2021-05-10T07:38:43Z

@gdonvito, @jorge-lip @jopina please start arranging compute and data resources to support the VO.
The VO (https://operations-portal.egi.eu/vo/update/serial/841) has been enabled in Perun, see #1
@enolfc is coordinating the SLA, see #2

backeb · 2021-05-17T08:12:28Z

@gdonvito, @jorge-lip, @jopina please respond :-)

Note that @enolfc suggested we can go ahead with allocating resources to this use case while they set up the SLA #2 (comment)

mariojmdavid · 2021-05-17T09:42:04Z

need to know the amount of VA to be provided by INCD for c-scale

backeb · 2021-05-17T09:56:49Z

@enolfc @sustr4, please assist @mariojmdavid with this.

Below is the information I have from the C-SCALE proposal

mariojmdavid · 2021-05-17T10:21:13Z

hi all, I found it in the proposal, and the number of VCPUs is different from those values
4500 VCPU days is wrong in the proposal since it is per month -> 180 VCPUs for the remaining period of the project
450 TB month - 18 TB for the remaining period of the project
we will setup the quotas accordingly
(this comment was edited to verify the numbers, there is an error in the google sheets and proposal that state this per day instead of per month)

mariojmdavid · 2021-05-17T11:05:55Z

need to know for the openstack mapping the following about the VO name
"any_one_of": ["^urn:mace:egi.eu:group:<VO_NAME>:role=vm_operator#aai.egi.eu$"]

enolfc · 2021-05-17T11:47:58Z

@mariojmdavid we need to test this as with Perun the entitlements change a bit from the defaults that we use for other VOs.

@backeb have you received my request to be member of the VO?

mariojmdavid · 2021-05-17T12:06:55Z

@enolfc what is the entitlement? I see that the name of the VO is aquamonitor.c-scale.eu
I will create the group, and can do an initial mapping based on that, and we will see later on

mariojmdavid · 2021-05-17T12:30:13Z

I have configures the usual EGI Checkin entitlement, but you can try the following

access: https://stratus.ncg.ingrid.pt/
Select in "Authenticate using": EGI Checkin
You will get a long URL which is not quite correct
https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

you will need to remove the duplicate part "?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/"
The is, you should have

https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

reload the page if needed

is a log standing issue in our Openstack deployment, that we are still trying to figure out, but after that you should be able to access the Dashboard

using the keystone CLI with federated identity and OpenID connect tokens should work as expected

backeb · 2021-05-17T13:38:40Z

@backeb have you received my request to be member of the VO?

@enolfc I don't think so - from which email address did you send it?

backeb · 2021-05-17T13:46:52Z

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

enolfc · 2021-05-17T13:49:40Z

you should be notified by Perun, not sure to which address though

mariojmdavid · 2021-05-17T16:54:25Z

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

can you confirm (or not) if it's because of my previous comment here? about the duplication in the URL?
if yes, check with
https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

mariojmdavid · 2021-05-17T16:55:53Z

if not put a date time when you tried so I can check the logs, or it can be because of the mapping/entitlement

backeb · 2021-05-17T17:00:09Z

you should be notified by Perun, not sure to which address though

I did get an email from perun, but same problem

backeb · 2021-05-19T13:12:19Z

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

can you confirm (or not) if it's because of my previous comment here? about the duplication in the URL?
if yes, check with
https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Hi @mariojmdavid, when I click on the above link (https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/) and log in using EGI SSO I still get the error message:
{"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

I did this now (19 May, 15h12 CET)

mariojmdavid · 2021-05-19T14:38:53Z

@backeb can you try again ? I had put the "role=vm_operator" now is =member, although some "higher role" should be defined at a certain moment

backeb · 2021-05-19T15:00:37Z

@backeb can you try again ? I had put the "role=vm_operator" now is =member, although some "higher role" should be defined at a certain moment

@mariojmdavid I get the same error and also tried in "Incognito" on Chrome

mariojmdavid · 2021-05-19T15:43:05Z

give me an approximate timestamp of this error
@enolfc need to know about the mapping string (entitlement) if need to modify because of Perun

backeb · 2021-05-19T16:19:58Z

give me an approximate timestamp of this error
@enolfc need to know about the mapping string (entitlement) if need to modify because of Perun

Time stamp is about the same time as this comment: #4 (comment)

enolfc · 2021-05-20T08:29:03Z

@mariojmdavid the current entitlement that users are getting is:
urn:mace:egi.eu:group:aquamonitor:members:role=member#aai.egi.eu

We many add more restrictive groups in Perun if needed

mariojmdavid · 2021-05-20T08:34:10Z

entitlement updated
@backeb can you try again?

enolfc · 2021-05-20T08:53:17Z

@mariojmdavid I confirm it works, I managed to get access to the VO.

backeb · 2021-05-20T15:20:10Z

Hi @gena,

@mariojmdavid @gdonvito @enolfc need to know how much storage resources they need to provide for Aquamonitor.

The list of data to be provided for Aquamonitor can be accessed here, I copy-pasted the list below:

Could you (@gena) give an indication of how much storage you would need?

Thanks

backeb · 2021-05-25T11:50:23Z

entitlement updated
@backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

gena · 2021-05-25T11:55:40Z

For Aqua Monitor, if we will try reproducing global analysis at 30m - something like 5TB should be enough. If the analysis will be limited to specific areas - probably 1TB would be ok.

Are there options to increase storage when needed? Or it is possible to do only by mounting additional disk and migrating datasets?

enolfc · 2021-05-26T11:10:13Z

entitlement updated
@backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

I would assume you can do the setup of the instance yourselves. If you need support we can guide you through it, but I'd prefer that we do not start VMs for users, as users need be responsible of their resources.

enolfc · 2021-05-26T11:15:19Z

For Aqua Monitor, if we will try reproducing global analysis at 30m - something like 5TB should be enough. If the analysis will be limited to specific areas - probably 1TB would be ok.

Are there options to increase storage when needed? Or it is possible to do only by mounting additional disk and migrating datasets?

@gena, the VMs can have volumes attached to them and these volumes are quite flexible for growing (shrinking not so easy). Additional disks can also be setup if needed.

backeb · 2021-05-26T11:54:35Z

entitlement updated
@backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/
Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

I would assume you can do the setup of the instance yourselves. If you need support we can guide you through it, but I'd prefer that we do not start VMs for users, as users need be responsible of their resources.

Thanks @enolfc. I assume the use case leads should set up the instances (I can do this for Deltares).

Once I have created the instance, do I just add the public keys of the developers to the VM so they can access it via SSH? Or do I also have to add them in PERUN? Could you provide me with some idea of the next steps?

enolfc · 2021-05-26T12:15:30Z

Once I have created the instance, do I just add the public keys of the developers to the VM so they can access it via SSH? Or do I also have to add them in PERUN? Could you provide me with some idea of the next steps?

Adding the ssh keys is good enough to get started. We could check how to automate this with PERUN if you think this is interesting (as with every automation it depends on how many times you need to do this)

backeb · 2021-05-26T13:26:21Z

Hi @gena and @avgils

I now have access to INCD’s OpenStack environment (https://stratus.ncg.ingrid.pt/dashboard/project/). To access the environment you need to

Have registered your credentials with EGI SSO (https://www.egi.eu/sso).
Follow this link https://stratus.ncg.ingrid.pt/dashboard/project/
Select EGI Checkin from the Authenticate using dropdown and click through the process
When you arrive at the url https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/ you will get a 401 error message. Remove the following string from the url: https://stratus.ncg.ingrid.pt/dashboard/auth/websso/?origin= and hit enter. You should then have access to the OpenStack environment.

@avgils on Monday let us sit together and create an instance.

mariojmdavid · 2021-05-28T09:43:43Z

hi all
you should do the instantiation of the VMs as you need, apologies but I will not do it :D
for the storage you should plan as much as you need, you can go up to 18TB which is the quota for the whole VO
and of course I will be here to help and answer questions as you have

backeb · 2021-05-31T08:47:02Z

hi all
you should do the instantiation of the VMs as you need, apologies but I will not do it :D
for the storage you should plan as much as you need, you can go up to 18TB which is the quota for the whole VO
and of course I will be here to help and answer questions as you have

Hi @mariojmdavid

I've set up an instance

I've associated two IP addresses to the instance (one public I think). When I try logging in I get the following error:

PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

avgils · 2021-05-31T08:59:25Z

Hi @mariojmdavid

I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa

Can you link my account so I can access the OpenStack environment?

Regards,

Anna

enolfc · 2021-05-31T09:51:19Z

PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

What image did you start? Should you try with the default user of the image (e.g. ubuntu)?

enolfc · 2021-05-31T09:55:14Z

Hi @mariojmdavid

I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa

Can you link my account so I can access the OpenStack environment?

Regards,

Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

backeb · 2021-05-31T11:16:20Z

PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

What image did you start? Should you try with the default user of the image (e.g. ubuntu)?

@enolfc we used: ubuntu-20.04-amd64

avgils · 2021-05-31T12:11:27Z

Hi @mariojmdavid
I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa
Can you link my account so I can access the OpenStack environment?
Regards,
Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

At the moment I get the following Error:

mariojmdavid · 2021-05-31T12:23:07Z

@backeb ssh ubuntu@194.210.120.109 ...

backeb · 2021-05-31T14:16:31Z

Thanks @mariojmdavid, we now have access via:
ssh -i "path/to/private_key.pem" ubuntu@194.210.120.109

backeb · 2021-05-31T14:17:26Z

Hi @mariojmdavid
I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa
Can you link my account so I can access the OpenStack environment?
Regards,
Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

At the moment I get the following Error:

@melanger could you please advise regarding this comment, please:
#4 (comment)

maricaantonacci · 2021-05-31T14:49:22Z

Dear @backeb , @enolfc
we have finalized the setup of the aquamonitor VO at our site (INFN-CLOUD-BARI). Openstack dashboard is accessible at cloud.recas.ba.infn.it, choose OpenID Connect Authentication and then click on aai.egi.eu/oidc/ link. If everything is working fine you should land in the project C-SCALE_aquamonitor.
Please let us know in case of problems
Cheers

mariojmdavid · 2021-06-16T08:34:20Z

hi all
I think this issue should be closed as both resource providers have setup the VO and resources
any further possible problems should be followed in other issues

backeb · 2021-06-16T12:31:02Z

Agreed @mariojmdavid. The final action here is on @backeb and @avgils to instantiate a VM on INFN-CLOUD-BARI's OpenStack dashboard.

backeb mentioned this issue May 10, 2021

Set up SLA to support the VO #2

Closed

backeb assigned gdonvito May 10, 2021

backeb assigned gena and unassigned gdonvito May 20, 2021

backeb mentioned this issue May 28, 2021

Associate service account with VO to access information from all sites #5

Closed

backeb mentioned this issue May 31, 2021

Access to INCD OpenStack #6

Closed

backeb closed this as completed Jun 16, 2021

sebastian-luna-valero mentioned this issue Jun 24, 2021

Arrange access to GRNET OpenStack c-scale-community/workflow-coastal-hydrowaq#5

Closed

backeb mentioned this issue Jun 30, 2021

Access to INFN-Cloud-BARI OpenStack #8

Closed

Set up compute and data resources to support the VO #4

Set up compute and data resources to support the VO #4

Comments

backeb commented May 10, 2021

backeb commented May 10, 2021

backeb commented May 17, 2021

mariojmdavid commented May 17, 2021

backeb commented May 17, 2021 • edited Loading

mariojmdavid commented May 17, 2021 • edited Loading

mariojmdavid commented May 17, 2021

enolfc commented May 17, 2021

mariojmdavid commented May 17, 2021

mariojmdavid commented May 17, 2021

backeb commented May 17, 2021 • edited Loading

backeb commented May 17, 2021

enolfc commented May 17, 2021

mariojmdavid commented May 17, 2021

mariojmdavid commented May 17, 2021

backeb commented May 17, 2021

backeb commented May 19, 2021

mariojmdavid commented May 19, 2021

backeb commented May 19, 2021 • edited Loading

mariojmdavid commented May 19, 2021

backeb commented May 19, 2021

enolfc commented May 20, 2021

mariojmdavid commented May 20, 2021

enolfc commented May 20, 2021

backeb commented May 20, 2021

backeb commented May 25, 2021

gena commented May 25, 2021

enolfc commented May 26, 2021

enolfc commented May 26, 2021

backeb commented May 26, 2021

enolfc commented May 26, 2021

backeb commented May 26, 2021

mariojmdavid commented May 28, 2021

backeb commented May 31, 2021 • edited Loading

avgils commented May 31, 2021

enolfc commented May 31, 2021

enolfc commented May 31, 2021

backeb commented May 31, 2021

avgils commented May 31, 2021

mariojmdavid commented May 31, 2021

backeb commented May 31, 2021

backeb commented May 31, 2021

maricaantonacci commented May 31, 2021

mariojmdavid commented Jun 16, 2021

backeb commented Jun 16, 2021

backeb commented May 17, 2021 •

edited

Loading

mariojmdavid commented May 17, 2021 •

edited

Loading

backeb commented May 17, 2021 •

edited

Loading

backeb commented May 19, 2021 •

edited

Loading

backeb commented May 31, 2021 •

edited

Loading