GAT-next Improvements

[hexylena]

• Fully automated videos Not yet :( 2023.
  • @mvdbeek suggests that the job_conf.xml file can be a job_conf.yaml, and the dependency resolver related stuff can be YAML, even within galaxy.yml, doesn't have to be a separate file. This would trigger re-recording of like 10 videos, we'll wait until helena's automated videos work.
• add cleanup cron job role
• move dependency resolvers and container resolvers to embedded
• move pulsar config to support the same.
• Create a dbservers group and putting postgres stuff in group_vars/dbservers.yml and connect over TCP, show modification of pg_hba.conf to support.
• Maybe move job_conf.xml and job_metrics_conf.xml to their own "adding additional configs" tutorial that gets run right after ansible-galaxy?
• Use "local" tools: Update job conf/pulsar tutorial for this amazing PR from @jmchilton that will save so many admins so much random pain. Allow specifying useful classes of tools for mapping in job conf YAML/XML galaxy#12258 looks like we just need <tool class="local" destination="local" />

These require nginx package changes:

• add mod_zip, thanks @mvdbeek @lldelisle https://docs.galaxyproject.org/en/latest/admin/nginx.html#creating-archives-with-mod-zip
  - [ ] maybe add https://docs.galaxyproject.org/en/latest/admin/nginx.html#receiving-files-with-nginx too? it's better for performance TUS!

Re-record next time

everything, tbh.

[natefoo]

Couple more things:

• Maybe should consider creating a dbservers group and putting postgres stuff in group_vars/dbservers.yml. It might be confusing up front but it will help people who later deploy on their own infra and have a separate DB server.
• Should we create galaxy_user_name and galaxy_db_name vars to avoid duplication in postrgresql_objects_* and database_connection?

[hexylena]

Should we create galaxy_user_name and galaxy_db_name vars to avoid duplication in postrgresql_objects_* and database_connection?

it's so early in the training, later it makes sense, but I worry about defining a variable in step like 1, and students thinking they have to define variables for everything (even if that's often best practice.)

[natefoo]

I worry about defining a variable in step like 1, and students thinking they have to define variables for everything (even if that's often best practice.)

There's definitely a balance. This mostly came up because I was recording the new video and realized how many times I pointed out having one canonical source of truth for everything, and then we don't do it in that case, heh.

[hexylena]

@natefoo and I discussed, we should aggressively cut down on the slide decks, at least on day 1. Some of them are just... not necessary, all the important/good information is in the tutorials after all.

slides	action	comments
DB	cut	we don't need to discuss sqlite or connection string formats, we have a table for that in the tutorial (or if we don't, we can), same with tuning parameters, this is documentation which isn't necessary for understanding "galaxy connects to a DB." and since we started including the good defaults, it's even less relevant. Install DB we mention and never touch again, not great.
systemd	merge	we should merge this into a tip box.
uwsgi	cut	:(
ephemeris	keep	they're good
cvmfs	keep	same
users groups quotes	keep	idk, it's on thin ice.
cluster computing	merge
DB querying	meh	these are now basically gxadmin advertising. I don't find them very useful, and it could be covered in a tuto. That said they're already an auto-video, so, meh, keep?
monitoring	idk	there is so little information there, it's basically a showcase. Is that useful? Is that the best format to talk about monitoring?

[hexylena]

Do we want to teach admins to use git this year @natefoo @Slugger70 @martenson? \

• if we do, teach them gitattributes for automatically decrypting vaults in git log -p output

  $ cat .gitattributes
  secret_group_vars/*.yml diff=ansible-vault merge=binary
  secret_host_vars/*.yml diff=ansible-vault merge=binary
  

[natefoo]

👍 to git, also wow that git log -p magic, I was not even aware of. Seems worth setting up?

[Slugger70]

I think that's a great idea. Especially the vault merging stuff which is always confusing..😁

…

On Tue, 30 Nov 2021, 04:43 Nate Coraor, ***@***.***> wrote: 👍 to git, also wow that git log -p magic, I was not even aware of. Seems worth setting up? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2583 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANI3XUC3I5WHIMZBK4SSLDUOO3UBANCNFSM457AQARA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[hexylena]

Update miniconda role with galaxy one

[hexylena]

To summarise for others passing through: testing.xml, my_rules.py and map_resources.py all have to go under files/galaxy/... rather than templates/galaxy/...

[hexylena]

Track minimum ansible version, note it in metadata of tutorials. Currently ansible-galaxy says >=2.7 which is hilariously outdated.

[natefoo]

I wish the linter could determine the minimum ansible version, because I always think of it as I'm updating the role and then forget by the time I go to update the meta.

[hexylena]

from @fubar2

With cvmfs reference data, doesn't singularity need /cvmfs added as a path in job_conf.xml.j2 ? :

<param id="singularity_volumes">$defaults,/cvmfs:ro</param>

It works then, but before I added that, mappers always failed - unable to access the /cvmfs reference data
It may be that I've broken something in removing lots of extra stuff - or nobody got around to testing with cvmfs reference data?

A more generic issue about singularity and cvmfs is noted at
https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#remounting-and-namespaces-containers, suggesting that -p is recommended for singularity so I also added:

<param id="singularity_run_extra_arguments"> -p </param>

Dunno if it makes any difference but...doesn't seem to be harmful.

Currently this is a bit of a sequencing issue, we cover singularity, then later cvmfs. I think that means cvmfs should come before singularity? maybe?

[hexylena]

@Slugger70 @natefoo and I had a short discussion ahead of GCC planning

• we'll move cvmfs ahead of singularity
• and enable singularity to use cvmfs containers?
  • No, we still can't do fall through dependency resolution 😭
• DBA ends up being gxadmin, I've got a collection of links but not sure how to turn that into a good tutorial

[mvdbeek]

• No, we still can't do fall through dependency resolution 😭

What's that about ? Do we have an issue for that ?

[hexylena]

@natefoo can provide some more details there but in general it seems everyone using singularity is manually choosing which tools/versions are going to a singularity destination

And for most smaller galaxies we need something more like "run in the right container, if available, and if not run in the default container and pull in the conda dependencies"

[mvdbeek]

And that's not how it works ? I had it configured that way back when I was admin for a small instance and I switched to singularity (lot's of resolver related things changed since then, so it may not work anymore ...).

The order should be

• resolve conda
• resolve container

If a container is resolved it'll take over, else conda is getting used.

[mvdbeek]

Oh, default container + conda dependencies, not just conda. I wasn't aware that's a thing. Is that going to build an image every time you run a tool, or is that mounting an existing conda env on disk and resolving dependencies in the image ?

[hexylena]

Sorry @mvdbeek I feel like we have this conversation once a year and every year I'm confused.

That sounds great if it's something we can use, that's going to work for local admins with their custom tools that may not have pre-mulled dependencies.

Or if it falls back to conda properly (even without the container) that's probably fine too. but I think in @natefoo and my discussion mounting was mentioned.

[natefoo]

It wouldn't build every time. I have an image I use for this here: https://github.com/natefoo/usegalaxy.org-legacy-environment

I believe what would be ideal is something like:

1. Resolve a matching container and run it in that if available
2. Resolve it in conda and run in a default container with conda mounted in if the deps are already installed in conda
3. Build/mull an image if the deps are available in conda but not already installed

[hexylena]

I am sufficiently emotionally done with this issue so, we're closing it. If we really want any of the open items in the OP, we can reopen.