You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Encountered this issue with Conda packages that install binaries, where the binary was successfully installed, but during step execution it is not found. The underlying reason is that in some cases conda_decorator does not successfully append the environmment to PATH.
Example flows for reproducing the issue:
test case 1: FAILS
frommetaflowimportFlowSpec, step, conda_base,condaimportsysimportoslibraries= {
"dbt-postgres": ">=1.7.0",
}
@conda_base(python="3.11", packages=libraries)classCondaPathTest(FlowSpec):
@stepdefstart(self):
print(f"path is: {os.environ.get('PATH')}")
importdbt# test that the library imports successfullyprint(dbt)
frommetaflow.utilimportwhich# Test that the binary is also found.# Only true if the environment/bin was added to PATH.ifnotwhich("dbt"):
raiseException("did not find dbt")
print(f"python version: {sys.version_info}")
self.next(self.end)
@stepdefend(self):
print(f"python version: {sys.version_info}")
if__name__=="__main__":
CondaPathTest()
test case 2: SUCCEEDS
frommetaflowimportFlowSpec, step, condaimportsysimportoslibraries= {
"dbt-postgres": ">=1.7.0",
}
classCondaPathTest(FlowSpec):
@conda(python="3.11", packages=libraries)@stepdefstart(self):
print(f"path is: {os.environ.get('PATH')}")
importdbt# test that the library imports successfullyprint(dbt)
frommetaflow.utilimportwhich# Test that the binary is also found.# Only true if the environment/bin was added to PATH.ifnotwhich("dbt"):
raiseException("did not find dbt")
print(f"python version: {sys.version_info}")
self.next(self.end)
@stepdefend(self):
print(f"python version: {sys.version_info}")
if__name__=="__main__":
CondaPathTest()
Notes
In the failing flow, the failure happens with which, where it fails to locate the installed binary. The library is still able to be imported, because the python interpreter is correctly set to the conda environment one.
The underlying cause for all of this is that unless we explicitly specify a conda decorator for a step, the environment is treated as disabled, and will fail all env setup processes.
The text was updated successfully, but these errors were encountered:
Encountered this issue with Conda packages that install binaries, where the binary was successfully installed, but during step execution it is not found. The underlying reason is that in some cases
conda_decorator
does not successfully append the environmment toPATH
.Example flows for reproducing the issue:
test case 1: FAILS
test case 2: SUCCEEDS
Notes
In the failing flow, the failure happens with
which
, where it fails to locate the installed binary. The library is still able to be imported, because the python interpreter is correctly set to the conda environment one.The underlying cause for all of this is that unless we explicitly specify a conda decorator for a step, the environment is treated as
disabled
, and will fail all env setup processes.The text was updated successfully, but these errors were encountered: