[initialisation step] No more initialisation (and thus calculation launch) seems to work in mia! #261

servoz · 2022-03-25T10:19:54Z

Minimum process to reproduce :

Nipype:
- Open a project with at least one anat in the DataBrowser
- Drag and drop the Smooth process into PipelineManager editor (nipype > interfaces > spm)
- Right click on the smooth_1 brick and choose Export all unconnected plugs
- Hit the inputs brick then on the filter button of the in_files parameter
- Select a data
- Run the pipeline
- The calculation does not take place:
- No popup window to indicate that there is an issue
- Pipeline "NoName" has not been correctly run is observed in the status bar

We observe in stdout:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/casa/host/src/capsul/master/capsul/process/process.py", line 731, in run_from_commandline
    result = ce.study_config.run(process, configuration_dict=configuration)
  File "/casa/host/src/capsul/master/capsul/study_config/study_config.py", line 429, in run
    configuration_dict=configuration_dict)
  File "/casa/host/src/capsul/master/capsul/study_config/run.py", line 165, in run_process
    returncode = process_instance._run_process()
  File "/casa/host/src/capsul/master/capsul/process/process.py", line 1864, in _run_process
    raise ValueError('output_directory is not set but is mandatory '
ValueError: output_directory is not set but is mandatory to run a NipypeProcess

Mia_processes
- Open a project with at least one anat in the DataBrowser
- Drag and drop the Smooth process into PipelineManager editor (mia_processes > bricks > preprocess > spm)
- Right click on the smooth_1 brick and choose Export all unconnected plugs
- Hit the inputs brick then on the filter button of the in_files parameter
- Select a data
- Run the pipeline
- The calculation does not take place:
  - A window pop up
  - Pipeline "NoName" was not initlialised succesfully is observed in the status bar
  - No warning / exception message is observed in the stdout

The text was updated successfully, but these errors were encountered:

servoz · 2022-03-25T15:49:15Z

Initial investigation seems to indicate that the issue is related to the workflow object and the part of code in the pipeline_manager_tab module around L1599.
The job.param_dict in self.workflow.jobs seems to not have the output file(s) generated by the process...
I think you coded that part @LStruber, didn't you? Do you have any idea?
I'm still investigating...

servoz · 2022-03-25T16:33:33Z

In capsul.pipeline.pipeline_workflow module, workflow_from_pipeline function, build_job funtion() L353 I observe that value is <undefined> (for the smoothed_files parameter in the Smooth from mia_processes example). It seems param_dict is populated only with no undefined parameter ?
Where is the completion done (a priori it has not yet been done at this time)?

servoz · 2022-03-25T17:21:10Z

For Smooth from nipype, it is not the same issue. Indeed, with nipype, the outputs are automatically determined when the input data is/are defined, so there is no completion problem like for the Smooth process from mia_processes. Here it is the output_directory that is not set.

Many changes have been made in the init_pipeline() method (populse_mia.user_interface.pipeline_manager.PipelineManagerTab.init_pipeline()). This ticket has allowed me to update myself a bit on what has been changed ... :-0)))

I'm starting to understand the problems.

Before these changes the completion was done in the list_outputs method of the mia_processes bricks. A priori, we don't use this method anymore (problem 1 with the Smooth brick from mia_processes).
For the problem 2 (with the Smooth brick of nipype), before the output_directory parameter was completed by passing in populse_mia.user_interface.pipeline_manager.process_mia.MIAProcessCompletionEngine.complete_nipype_common(), a priori, we do not pass any more by this method.

So, a priori, mia can't do any calculation on the nipype and mia_processes, maybe also capsul? That's almost all the processes that don't work anymore.
I'm willing to take on fixing this ticket, but I may not be the one who knows the changes best, so I will be may not be the fastest.

denisri · 2022-03-25T18:43:19Z

Does this mean that completion is not played any longer ?
It was doing it before, so something has been broken somewhere. I have not looked at the code since the init step has been merged with the run step...

servoz · 2022-03-26T07:34:44Z

Yes, it seems so.
If I remember correctly, the calculations worked after gathering the initialisation and execution. I think this would come from the workflow work?

LStruber · 2022-03-28T05:35:15Z

The initialization has indeed been changed to use workflow object from capsul and to be merged into the run step, but it had been tested (with Smooth and SpatialPreprocessing bricks from mia_processes) and it was working after this work... I don't know what changed after that, I'll check a bit on what's going in here...

servoz · 2022-03-28T05:47:38Z

Oh, in that case it would be better to test the previous commits and see when it stops working. I'm working on the #191 ticket, I hope I didn't do anything wrong while cleaning the modules...

LStruber · 2022-03-28T06:01:55Z

I tried to run the smooth process on my machine (with same procedure as @servoz ), and it's working smoothly (if I may say so...). Here is my current status:

lucas@timc-spm-13p:~/populse_dev/populse_mia$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   .idea/misc.xml
        modified:   .idea/populse_mia.iml
        modified:   .idea/workspace.xml
        modified:   properties/config.yml
        modified:   properties/saved_projects.yml

no changes added to commit (use "git add" and/or "git commit -a")

lucas@timc-spm-13p:~/casa_distro/brainvisa-opensource-master/src/capsul/master$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .idea/
        capsul.egg-info/

nothing added to commit but untracked files present (use "git add" to track)

lucas@timc-spm-13p:~/casa_distro/brainvisa-opensource-master/src/soma/soma-workflow/master$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

lucas@timc-spm-13p:~/casa_distro/brainvisa-opensource-master/src/soma/soma-base/master$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

LStruber · 2022-03-28T06:04:56Z

lucas@timc-spm-13p:~/populse_dev/mia_processes$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .idea/

nothing added to commit but untracked files present (use "git add" to track)

LStruber · 2022-03-28T06:10:26Z

Whoops, I may have looked at the wrong path ! I just launched a bv_maker, my capsul package was not up-to-date. I'll retest at the end of the bv_maker:

Configuration file: None
$ svn cleanup /casa/host/src
$ svn propset svn:externals --file /casa/host/src/bv_maker.externals /casa/host/src
property 'svn:externals' set on '.'
$ svn commit -m '' /casa/host/src
$ svn update /casa/host/src
Updating '.':
At revision 1.


Updating git repository /casa/host/src/aims/aims-free/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/aims/aims-gpl/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/anatomist/anatomist-free/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/anatomist/anatomist-gpl/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/axon/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/brainvisa-share/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/brainvisa-spm/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/capsul/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
From https://github.com/populse/capsul
 - [deleted]           (none)     -> origin/gui_capsul_config
 - [deleted]           (none)     -> origin/spm_edit_config
remote: Enumerating objects: 135, done.
remote: Counting objects: 100% (132/132), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 99 (delta 79), reused 72 (delta 54), pack-reused 0
Unpacking objects: 100% (99/99), done.
   fa3f04c1..8c90e5a0  master              -> origin/master
 * [new branch]        new_config          -> origin/new_config
   c1a96d98..3e684e4d  pydantic_controller -> origin/pydantic_controller
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Updating fa3f04c1..8c90e5a0
Fast-forward
 capsul/engine/module/matlab.py           | 53 ++++++++++++++++++++++++++++++++---------------------
 capsul/engine/module/spm.py              | 23 +++++++++++++++++------
 capsul/qt_gui/widgets/settings_editor.py |  5 ++++-
 3 files changed, 53 insertions(+), 28 deletions(-)
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit


Updating git repository /casa/host/src/cortical_surface/brainvisa_freesurfer/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.


Updating git repository /casa/host/src/cortical_surface/cortical_surface-gpl/master
$ git fetch --tags --prune origin '+refs/heads/*:refs/remotes/origin/*'
$ git update-ref --no-deref refs/bv_head refs/remotes/origin/master
$ git -c remote.origin.skipDefaultUpdate remote update
$ git merge --ff-only refs/bv_head
Already up to date.

LStruber · 2022-03-28T06:17:15Z

The smooth process is still working here after the bv_maker (and all went good). I tried both (from mia_processes and directly from nypipe)

LStruber · 2022-03-28T06:25:05Z

Hmmm sorry for the multiple comments, but populse_mia was not up to date either... I don't know why git reported an "up-to-date" status in the mia folder.
I was 5 commits late on mia, and now I pulled these commits, I can't launch mia anymore which close immediately with following exception:

An issue is detected in the Mia's configuration:
import_configs() got an unexpected keyword argument 'cont_on_error'
Please check the settings in File > Mia Preferences > Pipeline ...
python-BaseException
Traceback (most recent call last):
  File "/opt/pycharm-community-2021.3.2/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm-community-2021.3.2/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/lucas/populse_dev/populse_mia/python/populse_mia/main.py", line 1280, in <module>
    main()
  File "/home/lucas/populse_dev/populse_mia/python/populse_mia/main.py", line 799, in main
    c = config.get_capsul_config()
  File "/home/lucas/populse_dev/populse_mia/python/populse_mia/software_properties.py", line 411, in get_capsul_config
    self.capsul_engine.settings.export_config_dict(
AttributeError: 'Settings' object has no attribute 'export_config_dict'

Process finished with exit code 1

Do I need to update/remove my configuration cause of recent commits ?

denisri · 2022-03-28T08:44:56Z

@LStruber, it seems that capsul is not up-to-date for you either. It's strange that bv_maker hasn't updated several projects. Do you have a single clone of the repository, managed by bv_maker ?

denisri · 2022-03-28T08:46:51Z

(and are you actually on the master branch in your capsul sources ?)

servoz · 2022-03-28T08:47:45Z

I was not on casa distro.
I just tested in casa_distro with everything up to date and I observe exactly the same things as on my first post.

File "/home/lucas/populse_dev/populse_mia/python/populse_mia/software_properties.py", line 411, in get_capsul_config
   self.capsul_engine.settings.export_config_dict(
AttributeError: 'Settings' object has no attribute 'export_config_dict'

is strange. ... export_config_dict is indeed a method of Settings objects ...

Are you sure everything is up to date? (especially capsul)

servoz · 2022-03-28T08:56:55Z

@LStruber, it seems that capsul is not up-to-date for you either. It's strange that bv_maker hasn't updated several projects. Do you have a single clone of the repository, managed by bv_maker ?

@lucas, if you have a capsul clone in the same directory as populse_mia, this one takes precedence over the capsul of casa_distro

LStruber · 2022-03-28T08:57:52Z

Indeed. forgot to delete it... sorry

denisri · 2022-03-28T08:58:41Z

Anyway I confirm that initialization is not workin any longer. Same as you @servoz . I'll have a look too, it could be one of my modifs which made it break (?)...

LStruber · 2022-03-28T09:03:26Z

At least what my wanderings told us is that it was working with mia 5 commits behind and capsul also not up to date

servoz · 2022-03-28T09:05:11Z

There are several priorities to manage.
I prefer to stay on the ticket, if possible, (the configuration should work, especially since I merged with master, sorry I shouldn't have!!!).
The easiest way would be to look at which commit it's not working anymore and trace it back to the issue?

servoz · 2022-03-28T09:07:01Z

Edit *I prefer to stay on the #185 ticket,

denisri · 2022-03-28T09:20:54Z

It might be related to #185 actually: I suspect that the completion config in capsul is not correctly setup as it should, thus Mia completion is not called. I'm looking at it.

denisri · 2022-03-28T10:33:46Z

I think I get it: a part of internal notification in capsul engine/studyconfig sync was broken (one erased the notification setup by the other), so some internal factories were not updated, and thus were not looking for the MIA completion additions. I'm fixing that right now.

(populse/populse_mia#261)

denisri · 2022-03-28T10:49:42Z

Things are working now for me. Can you check this on your side please ?

servoz · 2022-03-28T10:52:47Z

@denisri, Great it works on my side too !!!
We close the ticket (I love it when a ticket is closed!)?

denisri · 2022-03-28T11:52:17Z

If it's OK for you, yes...

LStruber · 2022-03-28T11:54:50Z

For what it's worth, it's ok for me. (I didn't test when it wasn't working)

servoz · 2022-03-28T12:04:02Z

gloup, gloup ...

servoz added the bug Something isn't working label Mar 25, 2022

denisri added a commit to populse/capsul that referenced this issue Mar 28, 2022

add notifiers, without erasing existing ones, for attributes config

1255f6a

(populse/populse_mia#261)

servoz closed this as completed Mar 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[initialisation step] No more initialisation (and thus calculation launch) seems to work in mia! #261

[initialisation step] No more initialisation (and thus calculation launch) seems to work in mia! #261

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022 •

edited

denisri commented Mar 25, 2022

servoz commented Mar 26, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

servoz commented Mar 28, 2022

servoz commented Mar 28, 2022

LStruber commented Mar 28, 2022

denisri commented Mar 28, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

servoz commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

servoz commented Mar 28, 2022

denisri commented Mar 28, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

[initialisation step] No more initialisation (and thus calculation launch) seems to work in mia! #261

[initialisation step] No more initialisation (and thus calculation launch) seems to work in mia! #261

Comments

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022

servoz commented Mar 25, 2022 • edited

denisri commented Mar 25, 2022

servoz commented Mar 26, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

LStruber commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

servoz commented Mar 28, 2022

servoz commented Mar 28, 2022

LStruber commented Mar 28, 2022

denisri commented Mar 28, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

servoz commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

denisri commented Mar 28, 2022

servoz commented Mar 28, 2022

denisri commented Mar 28, 2022

LStruber commented Mar 28, 2022

servoz commented Mar 28, 2022

servoz commented Mar 25, 2022 •

edited