Demo workflow not working on CSF3 with latest matflow

[gcapes]

HI @aplowman
I've reproduced an error that @YuchenZZheng is having using MatFlow, version 0.3.0a128 on CSF. I've installed matflow in a python venv, Yuchen has installed it following these instructions https://docs.matflow.io/stable/installation.html#matflow-cli

We're running this workflow: matflow demo-workflow go tension_DAMASK_Al, but get the same error for tension_DAMASK_Ti.
The error is in the first task, which then looks to cause the rest to fail. Looking in artifacts/submissions/... at the SGE error file
TypeError: Command.__init__() got an unexpected keyword argument 'rules'

Not sure where to go from here really - is this a problem with the task schema?

[aplowman]

This is because the same version of MatFlow needs to be installed in the "software" environment that is used for each task. In your case, there is a mismatch. If you are using the shared MatFlow conda environment matflow_full_env, this has not yet been updated (as discussed here). So, you can either update the shared conda environment, or you could help @YuchenZZheng set up their own conda environment with the current matflow version. You would need to update MatFlow's environment file so it points to the new conda environment.

[gcapes]

Currently trying to get the action to run (LightForm-group/non-repo-issues#25), in order to build the environment file for Yuchen to use.

[gcapes]

I've been unpacking the environment file on CSF3 for a long time now (into my own user space), so just making a note in case I need to return to it another time.
I've installed matflow-new using pip, and was going to look into editing a copy of the environment file once this has unpacked.
This is likely duplication of work yuchen is doing, but will be useful practice for updating a shared installation in the future

[gcapes]

Restarted it and it went quicker. I noticed when modifying a duplicate of the csf3 env file that previously each env e.g. damask_parse_env defdap_env have different paths, but I only get the one path for the matflow_full_env conda environment I've unpacked. Does the full env include all the others, which were previously done separately?

[gcapes]

I am currently running the demo workflow, and get this warning

$ matflow show
/mnt/iusers01/support/mbexegc2/yuchen-zheng/.venv/lib/python3.11/site-packages/paramiko/pkey.py:100: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be removed
from this module in 48.0.0.
  "cipher": algorithms.TripleDES,
/mnt/iusers01/support/mbexegc2/yuchen-zheng/.venv/lib/python3.11/site-packages/paramiko/transport.py:259: CryptographyDeprecationWarning: TripleDES has been moved to cryptography.hazmat.decrepit.ciphers.algorithms.TripleDES and will be 
removed from this module in 48.0.0.
  "class": algorithms.TripleDES,

MatFlow, version 0.3.0a129

[gcapes]

The demo workflow ran without errors :)
Looks like I'll be able to update matflow on CSF3 following discussion

[gcapes]

In summary, this wasn't a MatFlow error at all - it was just that the environment wasn't up-to-date.