Conda environment being treated as disabled, and not appending environment to PATH.

[saikonen]

Encountered this issue with Conda packages that install binaries, where the binary was successfully installed, but during step execution it is not found. The underlying reason is that in some cases conda_decorator does not successfully append the environmment to PATH.

Example flows for reproducing the issue:

test case 1: FAILS

from metaflow import FlowSpec, step, conda_base,conda
import sys
import os

libraries = {
    "dbt-postgres": ">=1.7.0",
}
@conda_base(python="3.11", packages=libraries)
class CondaPathTest(FlowSpec):

    @step
    def start(self):
        print(f"path is: {os.environ.get('PATH')}")
        import dbt  # test that the library imports successfully
        print(dbt)
        from metaflow.util import which

        # Test that the binary is also found.
        # Only true if the environment/bin was added to PATH.
        if not which("dbt"):
            raise Exception("did not find dbt")

        print(f"python version: {sys.version_info}")
        self.next(self.end)

    @step
    def end(self):
        print(f"python version: {sys.version_info}")

if __name__ == "__main__":
    CondaPathTest()

test case 2: SUCCEEDS

from metaflow import FlowSpec, step, conda
import sys
import os

libraries = {
    "dbt-postgres": ">=1.7.0",
}
class CondaPathTest(FlowSpec):

    @conda(python="3.11", packages=libraries)
    @step
    def start(self):
        print(f"path is: {os.environ.get('PATH')}")
        import dbt  # test that the library imports successfully
        print(dbt)
        from metaflow.util import which

        # Test that the binary is also found.
        # Only true if the environment/bin was added to PATH.
        if not which("dbt"):
            raise Exception("did not find dbt")

        print(f"python version: {sys.version_info}")
        self.next(self.end)

    @step
    def end(self):
        print(f"python version: {sys.version_info}")

if __name__ == "__main__":
    CondaPathTest()

Notes

In the failing flow, the failure happens with which, where it fails to locate the installed binary. The library is still able to be imported, because the python interpreter is correctly set to the conda environment one.

The underlying cause for all of this is that unless we explicitly specify a conda decorator for a step, the environment is treated as disabled, and will fail all env setup processes.