Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance Conda easyblock: add support for using custom conda command (like mamba), and clean up after installation #2996

Merged
merged 7 commits into from Sep 15, 2023

Conversation

rsdmse
Copy link
Contributor

@rsdmse rsdmse commented Sep 7, 2023

  • Set DEFAULT_CONDA_CMD = 'conda' and allow for user-defined alternative (e.g. mamba)
  • Run conda clean -ya after env installation to clean up cache files

@ocaisa
Copy link
Member

ocaisa commented Sep 8, 2023

@boegelbot please test @ jsc-zen2 EB_ARGS="TRUST4-1.0.5.1.eb"

@boegelbot
Copy link

@ocaisa: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=2996 EB_ARGS="TRUST4-1.0.5.1.eb" EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_2996 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3332

Test results coming soon (I hope)...

- notification for comment with ID 1711708294 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@SebastianAchilles
Copy link
Member

@boegelbot please test @ jsc-zen2 EB_ARGS="TRUST4-1.0.5.1.eb"

@boegelbot
Copy link

@SebastianAchilles: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=2996 EB_ARGS="TRUST4-1.0.5.1.eb" EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_2996 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3334

Test results coming soon (I hope)...

- notification for comment with ID 1711861108 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS TRUST4-1.0.5.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2g1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/9b36c0c7095cf2daff5f132882e56101 for a full test report.

@ocaisa
Copy link
Member

ocaisa commented Sep 12, 2023

@boegelbot please test @ generoso EB_ARGS="TRUST4-1.0.5.1.eb"

@boegel boegel changed the title Add custom conda command and clean up enhance Conda easyblock: add support for using custom conda command (like mamba), and clean up after installation Sep 13, 2023
@ocaisa
Copy link
Member

ocaisa commented Sep 13, 2023

@boegelbot please test @ jsc-zen2 EB_ARGS="TRUST4-1.0.5.1.eb"

@boegelbot
Copy link

@ocaisa: Request for testing this PR well received on jsczen2l1.int.jsc-zen2.easybuild-test.cluster

PR test command 'EB_PR=2996 EB_ARGS="TRUST4-1.0.5.1.eb" EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --mem-per-cpu=4000M --job-name test_PR_2996 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen2.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 3365

Test results coming soon (I hope)...

- notification for comment with ID 1717641672 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS TRUST4-1.0.5.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
jsczen2c1.int.jsc-zen2.easybuild-test.cluster - Linux Rocky Linux 8.5, x86_64, AMD EPYC 7742 64-Core Processor (zen2), Python 3.6.8
See https://gist.github.com/boegelbot/b8f1b5e41f37759934c326068a337cbd for a full test report.

@ocaisa
Copy link
Member

ocaisa commented Sep 13, 2023

@boegelbot please test @ generoso EB_ARGS="TRUST4-1.0.5.1.eb"

@boegelbot
Copy link

@ocaisa: Request for testing this PR well received on login1

PR test command 'EB_PR=2996 EB_ARGS="TRUST4-1.0.5.1.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_2996 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11706

Test results coming soon (I hope)...

- notification for comment with ID 1717662302 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS TRUST4-1.0.5.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/6d4e0f676e1062926567bcbdecf40696 for a full test report.

Copy link
Member

@ocaisa ocaisa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rsdmse I think I agree with @branfosj that rather than set conda_cmd on the basis of a config variable we should check the environment. Search for get_software_root in this repo to get some examples of usage. For example:

from easybuild.tools.modules import get_software_root
...
if get_software_root('mamba'):
    conda_cmd = 'mamba'
elif get_software_root('conda'):
    conda_cmd = 'conda'
else
    raise EasyBuildError("Neither mamba nor conda available as a dependency!")

@rsdmse
Copy link
Contributor Author

rsdmse commented Sep 13, 2023

@ocaisa Thanks for the suggestion and code snippet. I've taken into account of the variants that I'm aware of: anaconda, miniconda, mamba, micromamba. If the number of variants keep expanding and go out of hand we may have to revert to allowing a user-defined command.

easybuild/easyblocks/generic/conda.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/conda.py Show resolved Hide resolved
@ocaisa
Copy link
Member

ocaisa commented Sep 15, 2023

@boegelbot please test @ generoso EB_ARGS="TRUST4-1.0.5.1.eb"

@boegelbot
Copy link

@ocaisa: Request for testing this PR well received on login1

PR test command 'EB_PR=2996 EB_ARGS="TRUST4-1.0.5.1.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_2996 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 11718

Test results coming soon (I hope)...

- notification for comment with ID 1721565503 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS TRUST4-1.0.5.1.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns2 - Linux Rocky Linux 8.5, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/4ab72b317ca2785b373e8f050e82089e for a full test report.

@ocaisa ocaisa merged commit 7e89d90 into easybuilders:develop Sep 15, 2023
47 checks passed
@ocaisa
Copy link
Member

ocaisa commented Sep 15, 2023

Going in, thanks for the patience @rsdmse !

@rsdmse
Copy link
Contributor Author

rsdmse commented Sep 15, 2023

@ocaisa Thanks for all the suggestions!

@boegel boegel added this to the next release (4.8.2?) milestone Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants