feat: added support for micromamba frontend #2013

charmoniumQ · 2022-12-14T22:11:53Z

Description

Updating #1889

QC

The PR contains a test case for the changes or the changes are already covered by an existing test case.
The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

johanneskoester

Very nice, thanks a lot! Just a few things below.
And may I ask you to format with black?

johanneskoester · 2022-12-16T09:29:03Z

snakemake/deployment/conda.py

@@ -465,8 +465,12 @@ def create(self, dryrun=False):
            )
            env_archive = self.archive_file
            try:
+                # Create empty env
+                if self.frontend == "micromamba":
+                    (Path(env_path) / "conda-meta").mkdir(parents=True, exist_ok=True)


Can you add a comment here why this is needed? And if you consider this a bug in micromamba, add a todo for removal once the bug is fixed.

sonarcloud · 2022-12-17T21:11:04Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

sonarcloud · 2023-01-23T15:14:50Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

johanneskoester · 2023-01-23T15:16:06Z

@charmoniumQ can you please format the code with black? I would do it myself, but you configured your fork such that I don't have permissions to push changes to it.

charmoniumQ · 2023-01-23T19:20:34Z

This should be a draft PR, because it doesn't completely work yet. Specifically mamba info --json has slightly different fields from conda info --json. I will run black on the code before I submit it for review.

johanneskoester · 2023-01-24T08:20:05Z

I see. Thanks!

johanneskoester · 2023-02-17T13:03:21Z

@charmoniumQ any chance to finalize this soon? Or shall I try to find some time to help out?

charmoniumQ · 2023-02-17T17:06:13Z

Unfortunately, I haven't had a lot of time recently. I may have more time in two weeks. I would appreciate if you helped out.

Specify how conda info is read with micromamba Work around micromamba not being able to create environment in existing directory

Austin-s-h · 2023-04-05T14:40:19Z

This should be a draft PR, because it doesn't completely work yet. Specifically mamba info --json has slightly different fields from conda info --json. I will run black on the code before I submit it for review.

Also running into this issue when I tried to use micromamba in my CI as the conda provider.

I tried a workaround by setting an alias of conda=mamba, but Snakemake was still complaining the command
conda info --json that the conda executable wasn't found and the pipeline was crashed.

maxim-h · 2023-04-21T11:14:47Z

@johanneskoester
I've been making some small additions to this PR in the last couple of weeks and got it to a somewhat working condition, at least based on a couple of very basic tests.
https://github.com/maxim-h/snakemake/tree/micromamba-2
@charmoniumQ seems a little too busy to review those at the moment and merge with his tree.
I'm not sure whether it's best to open a new PR (there are already 2 open regarding the same thing: #1889 and #2013), or if you could just merge them with this PR.
Let me know if it's useful at all and how to proceed.

Basic working version with micromamba

charmoniumQ · 2023-04-21T15:26:08Z

Somehow I missed the notification. Sorry to keep you waiting @maxim-h

Now this PR reflects the tip of Maxim-H's work.

charmoniumQ · 2023-04-21T17:34:20Z

I can't get Nose to work in Python 3.10, so I changed the docs/project_info/contributing.rst to refer to pytest instead.

But, with pytest, the tests pass on my machine except for some unrelated service/credential issues:

tests/test_google_lifesciences.py::test_github_issue1396
tests/test_google_lifesciences.py::test_github_issue1460
tests/test_slurm.py::test_slurm_mpi
tests/test_slurm.py::test_slurm_group_job
tests/test_slurm.py::test_slurm_complex
tests/test_tibanna.py::test_tibanna

sonarcloud · 2023-04-21T17:41:54Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

No Coverage information
0.0% Duplication

maxim-h · 2023-04-21T18:47:39Z

@charmoniumQ awesome! No worries, we all have a day job after all)
I also noticed a problem with nose, hence the #2211. But your changes look good as well.
@johanneskoester feel free to ignore the #2211 if you merge this, since df92405 also introduces other useful changes.

sparktx-ariel-hippen · 2023-08-14T13:51:29Z

Hi, what is the status of this feature? I only have micromamba, not conda, installed on my machine and when I include the --use-conda parameter on my workflow, I get the following error:

Traceback (most recent call last):
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/__init__.py", line 790, in snakemake
    success = workflow.execute(
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/workflow.py", line 1078, in execute
    dag.create_conda_envs(
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/dag.py", line 350, in create_conda_envs
    env.create(dryrun)
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 393, in create
    pin_file = self.pin_file
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/common/__init__.py", line 217, in __get__
    value = self.method(instance)
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 103, in pin_file
    f".{self.conda.platform}.pin.txt"
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/common/__init__.py", line 217, in __get__
    value = self.method(instance)
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 96, in conda
    return Conda(
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/deployment/conda.py", line 653, in __init__
    shell.check_output(self._get_cmd("conda info --json"), text=True)
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/site-packages/snakemake/shell.py", line 61, in check_output
    return sp.check_output(cmd, shell=True, executable=executable, **kwargs)
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/ubuntu/micromamba/envs/sm803/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'conda info --json' returned non-zero exit status 127.

charmoniumQ · 2023-08-16T16:55:25Z

This feature seems to work for me and passes tests packaged with the source code.

I think we're just waiting on @johanneskoester to merge or give feedback.

maxim-h · 2023-08-17T09:00:12Z

@sparktx-ariel-hippen not sure what the issue is in your case.

A couple things to check:

This PR is not merged, so make sure that you install snakemake from https://github.com/charmoniumQ/snakemake/tree/micromamba-2 if you want to use it.
I don't think snakemake can automatically determine whether you're using micromamba. In addition to --use-conda make sure to set --conda-frontend micromamba

baraaorabi · 2024-05-06T19:31:00Z

Any update on this? Really appreciate the work you're doing 🙏🏻

maxim-h · 2024-05-13T13:11:54Z

I've had moderate success dogfooding the changes made here.
This feature is definitely not yet complete, although under the right circumstances it does works.

Some of the issues I've observed are:

Not recognizing conda base path sometimes (have to provide --conda-base-path manually)
Not being able to use an existing conda environment, only a yaml file definition. Also doesn't happen in all cases, but I don't know what exactly triggers it.
An issue with the api change in of Pulp requires manually restricting it's version to <2.8. This was already fixed in main, but these changes are now drifting away more and more from it.

And most glaring issue in my opinion is that I didn't manage to use this branch without using the testing environment (from test-environment.yml). My guess is that testing environment includes among other things conda and mamba, so in some cases it's able to fall back on those. This, of course, kinda goes against the whole point of switching to micromamba. And none of the tests can really show that these changes work on their own (they don't, at least not fully) unless they are ran outside of the testing environment with only micromamba present.

Unfortunately my python skills aren't good enough to push this all the way home. There are some things in the Workflow and the shell class that initialize Conda without specifying the frontend, but I still can't tell if that's the issue.

Adapt activate shell command to micromamba

sonarcloud · 2024-05-13T16:37:34Z

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.4% Duplication on New Code

See analysis details on SonarCloud

fgvieira and others added 4 commits December 14, 2022 16:10

Added support for micromamba frontend

76784bd

Fixed typo

f241acb

Fixed creating empty env

d55cf16

Apply --yes flag for micromamba

e895ea9

charmoniumQ requested a review from johanneskoester as a code owner December 14, 2022 22:11

charmoniumQ changed the title ~~Micromamba 2~~ feat: added support for micromamba frontend Dec 14, 2022

charmoniumQ closed this Dec 15, 2022

charmoniumQ reopened this Dec 15, 2022

charmoniumQ changed the title ~~feat: added support for micromamba frontend~~ feat: added support for micromamba frontend Dec 15, 2022

johanneskoester requested changes Dec 16, 2022

View reviewed changes

Test old method of mkdir

59e8f59

charmoniumQ mentioned this pull request Dec 17, 2022

Add support for micromamba #1038

Open

johanneskoester approved these changes Dec 19, 2022

View reviewed changes

charmoniumQ marked this pull request as draft December 19, 2022 19:18

pvandyken linked an issue Dec 22, 2022 that may be closed by this pull request

Add support for micromamba #1038

Open

Merge branch 'main' into micromamba-2

8fd437f

johanneskoester marked this pull request as ready for review January 23, 2023 15:16

charmoniumQ marked this pull request as draft January 24, 2023 18:12

maxim-h added 2 commits April 5, 2023 14:34

Basic working version with micromamba

5c7a845

Specify how conda info is read with micromamba Work around micromamba not being able to create environment in existing directory

Additional tests for micromamba

0df3f75

formatting

ce8beb2

maxim-h and others added 2 commits April 21, 2023 13:33

Small refactoring and sanity check

9b6ef6a

Merge pull request #1 from maxim-h/micromamba-2

49cdb63

Basic working version with micromamba

Remove dependence on Conda when frontend is micromamba

df92405

charmoniumQ marked this pull request as ready for review April 21, 2023 17:37

Blackened source

67988fb

Adapt activate shell command to micromamba

4d5e62e

maxim-h mentioned this pull request May 13, 2024

Adapt activate shell command to micromamba charmoniumQ/snakemake#2

Merged

2 tasks

Merge branch 'main' into micromamba-2

4715bdb

This was referenced May 13, 2024

feat: added support for micromamba frontend #1889

Draft

Add "micromamba" as an option for --conda-frontend #2322

Open

charmoniumQ added 2 commits May 13, 2024 11:36

Merge branch 'micromamba-2' into micromamba-2

a87efac

Merge pull request #2 from maxim-h/micromamba-2

41b6518

Adapt activate shell command to micromamba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: added support for micromamba frontend #2013

feat: added support for micromamba frontend #2013

charmoniumQ commented Dec 14, 2022

johanneskoester left a comment •

edited

johanneskoester Dec 16, 2022

sonarcloud bot commented Dec 17, 2022

sonarcloud bot commented Jan 23, 2023

johanneskoester commented Jan 23, 2023

charmoniumQ commented Jan 23, 2023

johanneskoester commented Jan 24, 2023

johanneskoester commented Feb 17, 2023

charmoniumQ commented Feb 17, 2023

Austin-s-h commented Apr 5, 2023

maxim-h commented Apr 21, 2023 •

edited

charmoniumQ commented Apr 21, 2023

charmoniumQ commented Apr 21, 2023

sonarcloud bot commented Apr 21, 2023

maxim-h commented Apr 21, 2023 •

edited

sparktx-ariel-hippen commented Aug 14, 2023

charmoniumQ commented Aug 16, 2023

maxim-h commented Aug 17, 2023

baraaorabi commented May 6, 2024

maxim-h commented May 13, 2024 •

edited

sonarcloud bot commented May 13, 2024

feat: added support for micromamba frontend #2013

Are you sure you want to change the base?

feat: added support for micromamba frontend #2013

Conversation

charmoniumQ commented Dec 14, 2022

Description

QC

johanneskoester left a comment • edited

Choose a reason for hiding this comment

johanneskoester Dec 16, 2022

Choose a reason for hiding this comment

sonarcloud bot commented Dec 17, 2022

sonarcloud bot commented Jan 23, 2023

johanneskoester commented Jan 23, 2023

charmoniumQ commented Jan 23, 2023

johanneskoester commented Jan 24, 2023

johanneskoester commented Feb 17, 2023

charmoniumQ commented Feb 17, 2023

Austin-s-h commented Apr 5, 2023

maxim-h commented Apr 21, 2023 • edited

charmoniumQ commented Apr 21, 2023

charmoniumQ commented Apr 21, 2023

sonarcloud bot commented Apr 21, 2023

maxim-h commented Apr 21, 2023 • edited

sparktx-ariel-hippen commented Aug 14, 2023

charmoniumQ commented Aug 16, 2023

maxim-h commented Aug 17, 2023

baraaorabi commented May 6, 2024

maxim-h commented May 13, 2024 • edited

sonarcloud bot commented May 13, 2024

Quality Gate passed

johanneskoester left a comment •

edited

maxim-h commented Apr 21, 2023 •

edited

maxim-h commented Apr 21, 2023 •

edited

maxim-h commented May 13, 2024 •

edited