GitHub Workflow?

[vsoch]

Hi! I'm looking to install lmod in a GitHub workflow for testing with another library, and I found your github workflow (test.yml) but it's not clear to me where lmod is installed (I see dependencies for it and then something called hermes). Do you have an example or best practices for installing lmod in a workflow? Thanks!

[wpoely86]

Actually, testing a proper install is not yet part of the pipeline. It's somewhere on my todo.

If you wanna be quick, a dnf/yum install Lmod or apt-get install Lmod should do the trick?

Else it's just ./configure && make && make install. For the (lua) deps, have a look at the workflow. We use luarocks to set up the needed bits.

[vsoch]

oh I didn't know it was released on package managers! GitHub actions supports ubuntu, so I'll give the second and try and get back to you.

[vsoch]

hmm no luck - do I need to add a mirror or something (I did do apt-get update first)

[vsoch]

I'll try it with lowercase L.

[wpoely86]

Let me double check cause I don't use Ubuntu. For RHEL flavours we have a RPM.

[wpoely86]

It should be there: https://packages.ubuntu.com/focal/lmod

[vsoch]

That seemed to do the trick - until it randomly exited with 1. Does lmod install do something that GitHub actions does not allow?

[wpoely86]

The version is pretty ancient however.

[vsoch]

hmm okay - I'll try a different approach, making a custom GitHub action that has it installed.

[wpoely86]

Weird, when I try apt-get install lmod in a fresh ubuntu container it works fine.

[vsoch]

GitHub actions workflows prevents the user from making specific changes to the runner, so likely one of those changes is not allowed. I'm not what, however. :/

[wpoely86]

If it's an option for you, pull in a centos 8 container in the action. dnf install Lmod works 😉

[vsoch]

I don't think GitHub workflows support centos (I just looked that up hoping to try the same thing!) 😆

[vsoch]

I'm going to give a shot on CircleCI!

[wpoely86]

no but you can run our flow in a container which is the same thing:

jobs:
  runtests:
    runs-on: ubuntu-latest
    container: centos:8
    steps:
    - name: install Lmod
      run: dnf -y install  Lmod

[vsoch]

AH! You're totally right!! I'll try that too <3

[vsoch]

Hmm, it didn't find a package for lmod or Lmod (both times)

[wpoely86]

It's in EPEL (forgot to mention that).

[boegel]

There's some inspiration in https://github.com/easybuilders/easybuild-framework/blob/develop/.github/workflows/unit_tests.yml for how to install Lmod in Ubuntu in GitHub Actions (we install it from source, because we want to control the version).
This uses a helper script that's available at https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/scripts/install_eb_dep.sh

[vsoch]

Woot! Thanks @boegel I'll take a look! It did work to install on CircleCI, and I suspect the dnf command would too (via EPEL). But I do ultimately want it working in GitHub actions to do a custom workflow to test container installs. It's a "Vanessa side project" so I'll probably test this out again after work hours / weekend, etc. Thanks to you both for help - I will post an update here when I've done a little more testing.

[ocaisa]

I wrote an Action recently for EESSI that includes Lmod (although mostly as a side effect): https://github.com/EESSI/github-action-eessi

[wpoely86]

That is supercool! We can use the EESSI stack so easily in a gh action?

[boegel]

Yes, see EESSI/software-layer#74

[vsoch]

hey everyone! So I think I'm close - I was able to get singularity and lmod installed from EPEL along with the software I'm testing. I'm having trouble with getting the expected bash files sourced, here -> https://github.com/singularityhub/singularity-hpc/pull/10/files#diff-d6aa2676ea17b3a10c82f4bb059444824abc364178f4b2af9f071d85e93dc405R68

No matter what I try, I always see the issue that "module" is not found. I suspect I need to make sure the GitHub bash profile has that line to source, but I haven't been able to figure out how to do that yet.

[vsoch]

okay another update! I got the above to work, and now the issue (as I expected) with running this all within a centos container is that we can't do any singularity run or exec, e.g.,:

Testing python
Module python/3.9.2-slim was created.
Testing load of python:3.9.2-slim
INFO:    Converting SIF file to temporary sandbox...
FATAL:   while extracting /tmp/shpc-test.c_94xdvn/python/3.9.2-slim/python-3.9.2-slim-sha256:85ed629e6ff79d0bf796339ea188c863048e9aedbf7f946171266671ee5c04ef.sif: root filesystem extraction failed: extract command failed: ERROR  : Failed to create user namespace: user namespace requires to set /proc/sys/kernel/unprivileged_userns_clone to 1
: exit status 1

Ideally everything would be installed and run on the native ubuntu image (without a container) and maybe we might be able to get exec/run to work. But that means the methods for installing lmod and singularity that I used (from yum) of course wouldn't work. So for a first pass, I have a working recipe tester, but I'm not happy with it because it just tests that the modules install and load, but we can't check the containers running. This also means that GitHub actions won't work (also in a container) so @boegel I think I'll need to look closer at the example you linked.

So not perfect, but closer! That's still progress :)

[vsoch]

Okay I got it working, with ubuntu (native) and installing LMOD from source (actually much easier than I expected!) and singularity exec's are working! Here is the finished recipe:

https://github.com/singularityhub/singularity-hpc/blob/main/.github/workflows/test.yml

Some of the logic in there (e.g., the artifact) resulted from weird functionality when using a container, I likely will redo those bits. Interestingly, it builds and runs SO much faster this way than using the centos container.

This is super exciting! I have the CI setup now so I can start adding actual container registry entries! 🥳 I really should just like, stop now for the night. 💤

[boegel]

@vsoch So, this issue can be closed?

[vsoch]

Yep thanks for the help!