Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GAT2021 Followup Overhaul #2371

Merged
merged 175 commits into from
Jun 7, 2021
Merged

GAT2021 Followup Overhaul #2371

merged 175 commits into from
Jun 7, 2021

Conversation

hexylena
Copy link
Member

@hexylena hexylena commented Feb 15, 2021

GAT 2021 Followup

This PR contains a large number of bugfixes targeted at issues discovered during #2274

Fixes #2274

Admin Training: Automated

This branch includes numerous fixes for the admin training topic, collected from GAT2021:Online, but it also starts the project of automating GAT.

  • We now use diffs throughout the admin training series
  • These can be pulled out and added together, to build the full admin training playbooks
  • This is automated in https://github.com/hexylena/git-gat

There is alpha quality code for building the completely automated videos (slides to video, but instead tutorial to videos). This needs more testing before it's productive, we're aiming for GAT 2022 for that.

enable_quotas: true
allow_user_deletion: true
show_welcome_with_login: true
expose_user_name: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't add this, since it probably violates the GDPR (not that I'm advocating to add enable_beta_gdpr: true here).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I really wish we had a lawyer to comment on that but in retrospect it is a "Public User Name" (per the registration form), there should be a strong expectation that it's public. I was too conservative in the original implementation and I regret that.

topics/admin/tutorials/cvmfs/tutorial.md Outdated Show resolved Hide resolved
topics/admin/tutorials/pulsar/tutorial.md Outdated Show resolved Hide resolved
topics/admin/tutorials/pulsar/tutorial.md Outdated Show resolved Hide resolved
@hexylena

This comment has been minimized.

@hexylena hexylena changed the title GAT2021 Fixes GAT2021 Followup Overhaul Mar 26, 2021
@hexylena

This comment has been minimized.

@hexylena
Copy link
Member Author

hexylena commented Jun 3, 2021

tutorial playbooks tests
ansible-galaxy ✔️ ✔️
singularity ✔️ missing
tool management n/a missing
cvmfs ✔️ added + passing
data libs ✔️ ✔️
connect-to-compute ✔️ missing
job-destinations ✔️ missing
pulsar ✔️ ish. (we test that pulsar is running OK)

but I did something dumb with the galaxy systemd, it doesn't behave nicely if you just want to e.g. set the env, rather than also setting that it should be mules. That's what I get for using dictionaries instead of just a bazillion variables.

@hexylena
Copy link
Member Author

hexylena commented Jun 3, 2021

@natefoo @Slugger70 that's all I have energy for, hope y'all can get the ball across the line on this one.

  • @Slugger70 would you be up for re-recording singularity + pulsar (since we use vault now)?
  • @natefoo would you be up for recording the ansible-galaxy tuto?
  • @cat-bro since you've made a couple changes, do you want to re-record ephemeris? Is it worth doing?

(or any other mixture of people recording but yeah.)

@cat-bro
Copy link
Collaborator

cat-bro commented Jun 4, 2021

@hexylena I'll be rerecording the ephemeris stuff
A note on git-gat: I'm currently running it on one of the gat machines and I needed to update the cache before it was able to install packages. I'm not sure if this belongs in the playbook or the bootstrap.
A note on the ephemeris tutorial: I'm currently failing to install things on my gat machine with bioblend.ConnectionError: HTTPSConnectionPool(host='gat-2.oz.training.galaxyproject.eu', port=443): Max retries exceeded with url: /api/tools?in_panel=True&key=45fc909559337090b1969c4abcb80e9c (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))), 0 attempts left:. I don't think this came up in January. I'll have a go at troubleshooting this later on today.

@hexylena
Copy link
Member Author

hexylena commented Jun 4, 2021 via email

@cat-bro
Copy link
Collaborator

cat-bro commented Jun 4, 2021

Looking through the scrollback I can see that galaxy-wait did work perfectly during the run of .scripts/1-ansible-galaxy-run.sh but no longer does.

ubuntu@gat-2:~/git-gat$ galaxy-wait -g https://gat-2.oz.training.galaxyproject.eu -v
[00] Galaxy not up yet... HTTPSConnectionPool(host='gat-2.oz.training.galaxyproject.eu', port=443): Max retries exceeded with
[01] Galaxy not up yet... HTTPSConnectionPool(host='gat-2.oz.training.galaxyproject.eu', port=443): Max retries exceeded with

Galaxy is running and logs look fine

@hexylena
Copy link
Member Author

hexylena commented Jun 4, 2021

@cat-bro fixed, something was wrong with the certificates but can't tell what. The LE fake roots x1 and x2 were there (one had an unexpected filename, .pem.crt, both should have been .crt) so I'm suspicious that they didn't get deployed properly. I re-ran that task from bootstrapping and it works now.

@hexylena
Copy link
Member Author

hexylena commented Jun 4, 2021

I cannot explain it working initially, that's,, interesting.

@cat-bro
Copy link
Collaborator

cat-bro commented Jun 4, 2021

@hexylena thanks for fixing! scripts/1 was run, followed by scripts/2, followed by my installing ephemeris in a virtual environment. Whatever went wrong happened sometime after the end of scripts/1. That .pem.crt file is a fake one that I added to try to fix it.

@cat-bro
Copy link
Collaborator

cat-bro commented Jun 4, 2021

The problem seems to arise from installing ephemeris in a virtual environment when it is already installed globally. This time it's only broken within the virtual environment. The first time it broke globally as well, no idea why.

@hexylena
Copy link
Member Author

hexylena commented Jun 4, 2021

Ahhhh that could make sense. It was installed globally as part of the testing process.

Should we just install it globally? skip the venv step then?

@cat-bro
Copy link
Collaborator

cat-bro commented Jun 4, 2021

Would it be safer if git-gat had a venv for its requirements? If it's installed globally there's a chance that a student would create the ephemeris_venv and break it. The ephemeris_venv is also part of the data libraries tutorial

[updated] The machines already have global ephemeris because they have planemo installed as part of the bootstrap process, so it must have been there last time too but the venv was ok. I'll add a step to check whether ephemeris is already installed in the tut and skip the venv step.

@Slugger70
Copy link
Contributor

@natefoo @Slugger70 that's all I have energy for, hope y'all can get the ball across the line on this one.

  • @Slugger70 would you be up for re-recording singularity + pulsar (since we use vault now)?
  • @natefoo would you be up for recording the ansible-galaxy tuto?
  • @cat-bro since you've made a couple changes, do you want to re-record ephemeris? Is it worth doing?

(or any other mixture of people recording but yeah.)

Hey @hexylena - I will re-record singularity and pulsar as soon as this PR is merged.

@hexylena
Copy link
Member Author

hexylena commented Jun 7, 2021

@Slugger70 approved so I'm going to merge this. We'll put everything else in a new PR(s) but should be easier to contribute once this huge chunk is merged.

@hexylena hexylena merged commit 74bfd39 into main Jun 7, 2021
@hexylena hexylena deleted the gat-fixes branch June 7, 2021 08:12
@martenson
Copy link
Member

Nice job @hexylena 🥇

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Post-GAT2021 Improvements
9 participants