Test demo workflow simulate_yield_surface_2D

[gcapes]

[gcapes]

$ matflow demo-workflow go simulate_yield_surface_2D
Traceback (most recent call last):
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/bin/matflow", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/demo/cli.py", line 168, in make_and_submit_demo_workflow
    out = app.make_and_submit_demo_workflow(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/app.py", line 333, in <lambda>
    return lambda *args, **kwargs: func(*args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/app.py", line 1692, in _make_and_submit_demo_workflow
    wk = self.make_demo_workflow(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/app.py", line 333, in <lambda>
    return lambda *args, **kwargs: func(*args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/app.py", line 1599, in _make_demo_workflow
    wk = self.Workflow.from_file(
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 1001, in from_file
    return cls.from_template(
           ^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 675, in from_template
    with wk.batch_update(is_workflow_creation=True):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/apps/apps/binapps/anaconda3/2024.10/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 1666, in batch_update
    self._store._pending.commit_all()
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/pending.py", line 160, in commit_all
    getattr(self, meth)()
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/pending.py", line 261, in commit_element_sets
    self.store._append_element_sets(task_id, es_js)
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/zarr.py", line 568, in _append_element_sets
    task_idx = task_idx = self._get_task_id_to_idx_map()[task_id]
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/base.py", line 1593, in _get_task_id_to_idx_map
    return {i.id_: i.index for i in self.get_tasks()}
                                    ^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/base.py", line 1652, in get_tasks
    tasks = self._get_persistent_tasks(range(self._get_num_persistent_tasks()))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 43, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/zarr.py", line 1003, in _get_persistent_tasks
    elem_IDs_arr_dat = self._get_tasks_arr().get_coordinate_selection(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/zarr/core.py", line 1154, in get_coordinate_selection
    out = self._get_selection(indexer=indexer, out=out, fields=fields)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/zarr/core.py", line 1341, in _get_selection
    self._chunk_getitems(
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/zarr/core.py", line 2186, in _chunk_getitems
    self._process_chunk(
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/zarr/core.py", line 2099, in _process_chunk
    chunk = self._decode_chunk(cdata)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows/.venv/lib/python3.12/site-packages/zarr/core.py", line 2380, in _decode_chunk
    chunk = chunk.reshape(expected_shape or self._chunks, order=self._order)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: cannot reshape array of size 5 into shape (1,)

[gcapes]

Now using MatFlow, version 0.3.0a145hpcFlow, version 0.2.0a205

(.venv) [mbexegc2@login1 [csf3] test-demo-workflows-latesthpcmatflow]$ vim simulate_yield_surface_2D_2025-04-01_101059/artifacts/submissions/0/app_std/1399.txt 
(.venv) [mbexegc2@login1 [csf3] test-demo-workflows-latesthpcmatflow]$ cat $_
Traceback (most recent call last):
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/utils.py", line 1178, in redirect_std_to_file
    yield
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 4043, in execute_run
    self.set_EAR_end(
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 2761, in set_EAR_end
    for EAR_dep_ID in run.get_dependent_EARs(as_objects=False):
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/actions.py", line 609, in get_dependent_EARs
    for elem in task.elements[:]
                ~~~~~~~~~~~~~^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/task.py", line 3454, in __getitem__
    elements = self.task.workflow.get_task_elements(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/workflow.py", line 2570, in get_task_elements
    for te in self._store.get_task_elements(task.insert_ID, idx_lst)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/base.py", line 2573, in get_task_elements
    EARs_dcts = remap(
                ^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/core/utils.py", line 802, in remap
    return reshape(mapping_func(x), y)
                   ^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/base.py", line 2575, in <lambda>
    lambda ears: [ear.to_dict() for ear in self.get_EARs(ears)],
                                           ^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/base.py", line 2381, in get_EARs
    EARs = self._get_persistent_EARs(id_pers) if id_pers else {}
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/zarr.py", line 1725, in _get_persistent_EARs
    EAR_arr_dat = _zarr_get_coord_selection(arr, sel, self.logger)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/zarr.py", line 112, in _zarr_get_coord_selection
    return _inner(arr, selection)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/decorator.py", line 235, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/reretry/api.py", line 161, in retry_decorator
    return retry_call(
           ^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/reretry/api.py", line 218, in retry_call
    return func(
           ^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/reretry/api.py", line 31, in __retry_internal
    return f()
           ^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/log.py", line 81, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/hpcflow/sdk/persistence/zarr.py", line 110, in _inner
    return arr.get_coordinate_selection(selection)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/core.py", line 1156, in get_coordinate_selection
    out = self._get_selection(indexer=indexer, out=out, fields=fields)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/core.py", line 1343, in _get_selection
    self._chunk_getitems(
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/core.py", line 2177, in _chunk_getitems
    cdatas = self.chunk_store.getitems(ckeys, contexts=contexts)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/_storage/store.py", line 161, in getitems
    return {k: self[k] for k in keys if k in self}
               ~~~~^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/storage.py", line 1113, in __getitem__
    return self._fromfile(filepath)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/.venv/lib/python3.12/site-packages/zarr/storage.py", line 1087, in _fromfile
    with open(fn, "rb") as f:
         ^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/simulate_yield_surface_2D_2025-04-01_101059/metadata/runs/1/572'

[aplowman]

@gcapes are there any other errors (in js_std or app_std)? Could you try running this again with the latest versions?

I can't easily test this unfortunately, as i don't have a Matlab licence at the moment.

[gcapes]

Using latest versions, I get this:

$ cat simulate_yield_surface_2D_2025-04-29_103438/artifacts/submissions/0/js_std/0/js_0.sh.o5761936 
python: can't open file '/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/simulate_yield_surface_2D_2025-04-29_103438/artifacts/submissions/0/tmp/generate_microstructure_seeds_from_random_act_0.py': [Errno 2] No such file or directory

This looks like the same error as in #296 (comment)

[aplowman]

Can you check you have "<<script_path>>" and not <<script_name>>, as per #354?

And no other output in js_std or app_std?

Can you simplify by just running the offending task: matflow demo-workflow go simulate_yield_surface_2D --tasks 0, zip it up and send it to me?

[gcapes]

I now have "<<script_path>>", and that error has disappeared, but it still doesn't run properly.

simulate_yield_surface_2D_2025-04-29_140830.zip

[aplowman]

I think this is a problem with where hpcflow is installed from, as in #296 (comment) (you need to be using the fix/shell hpcflow branch, given your configuration file).

[gcapes]

I now get this

$ cat simulate_yield_surface_2D_2025-04-29_155401/artifacts/submissions/0/js_std/1/js_1.sh.o5763196.1 
/net/scratch2/mbexegc2/test-demo-workflows-latesthpcmatflow/simulate_yield_surface_2D_2025-04-29_155401/artifacts/submissions/0/commands/1.sh: line 9: -softwareopengl: command not found

I think I need to check my environment given I'd had lots of changes recently trying to get MTEX to compile. Learning - I really should have version controlled the environments file :)

[aplowman]

What's in /artifacts/submissions/0/commands/1.sh?

[gcapes]

Yep, I need to fix my environment as suspected:

$ cat simulate_yield_surface_2D_2025-04-29_155401/artifacts/submissions/0/commands/1.sh 
. "$MATFLOW_JS_FUNCS_PATH"

module load apps/binapps/matlab/R2019a
module load apps/binapps/matlab/third-party-toolboxes/mtex/5.3
#export MATLAB=/opt/apps/apps/binapps/matlab/R2019a/bin/matlab
#export MTEX_DIR=/opt/apps/apps/binapps/matlab/third-party-toolboxes/mtex/5.3
for dir in $(find ${MTEX_DIR} -type d | grep -v -e "@" -e "private"); do MATLABPATH="${dir};${MATLABPATH}"; done
export MATLABPATH=${MATLABPATH}
${MATLAB} -softwareopengl -singleCompThread -batch "addpath('$MATFLOW_RUN_SCRIPT_DIR'); $MATFLOW_RUN_SCRIPT_NAME_NO_EXT js_${MATFLOW_JS_IDX}_block_${MATFLOW_BLOCK_IDX}_act_${MATFLOW_BLOCK_ACT_IDX}_inputs.json js_${MATFLOW_JS_IDX}_block_${MATFLOW_BLOCK_IDX}_act_${MATFLOW_BLOCK_ACT_IDX}_outputs.h5"