add cellprofiler v4.2.1 tool #1163

sunyi000 · 2021-09-15T11:58:14Z

main module of cellprofiler 4.2.1 as galaxy tool

sunyi000 · 2021-09-15T13:36:03Z

@bgruening I still get all tests passed on my local machine with clean setup, but failed during CI.

the only difference I see is that the ci workflow uses py3.7 during test and combined_test..
since cellprofiler only works with 3.8 and above (https://github.com/CellProfiler/CellProfiler/blob/master/setup.py#L96) I just want to try upgrading the python version in ci.yaml but it seems that circleci cached the old version somewhere.
could you help me clear the cache, or do I simply just need to rename the cache key in ci.yml ?

just want to see if changing python version in ci.yml makes any difference

thanks very much

bgruening · 2021-09-15T16:23:10Z

Please run your tests locally with planemo test --docker. Are they failing when you do that?

sunyi000 · 2021-09-15T18:47:43Z

Please run your tests locally with planemo test --docker. Are they failing when you do that?

it succeeded. all 4 tests passed with planemo test --docker cellprofiler.xml

tools/cellprofiler_v4/cellprofiler.xml

Co-authored-by: Cristóbal Gallardo <gallardoalbac@gmail.com>

tools/cellprofiler_v4/cellprofiler.xml

Co-authored-by: Cristóbal Gallardo <gallardoalbac@gmail.com>

gallardoalba · 2021-09-22T16:25:35Z

When I run it in local, I get this error:

But it works after downgrading the NumPy version:

Then I guess that it is necessary to modify the conda recipe.

sunyi000 · 2021-09-22T16:44:11Z

When I run it in local, I get this error:

But it works after downgrading the NumPy version:

Then I guess that it is necessary to modify the conda recipe.

ah.. I got numpy 1.19 in my environment..I will go change the recipe then...
thanks

sunyi000 · 2021-09-22T17:23:16Z

I just updated the recipe, and created a PR 30645
@gallardoalba Thank you.

gallardoalba · 2021-09-22T18:18:46Z

I just updated the recipe, and created a PR 30645
@gallardoalba Thank you.

You can ask here for a review https://gitter.im/bioconda/Lobby

simonbray · 2021-09-23T08:11:57Z

Could you rename the @CP_VERSION@ token to @TOOL_VERSION@? This will allow automated updates of the cellprofiler tool. :)

… with cp3

tools/cellprofiler_v4/cellprofiler.xml

gallardoalba · 2021-09-23T09:09:52Z

It seems to be a similar error @sunyi000 scikit-image/scikit-image#4664. I think that you will need to create a Docker image in order to solve it.

sunyi000 · 2021-09-23T09:25:25Z

thanks for the link.. I will create a docker
but I'm still confused why this is working in a local env...

gallardoalba · 2021-09-23T09:39:38Z

thanks for the link.. I will create a docker
but I'm still confused why this is working in a local env...

In that case, the reason is that in local don't exist such problems with folder permissions, but in Galaxy, due to security reasons, there are restrictions. Regards.

Co-authored-by: Simon Bray <32272674+simonbray@users.noreply.github.com>

mvdbeek · 2021-10-29T11:02:55Z

In that case, the reason is that in local don't exist such problems with folder permissions, but in Galaxy, due to security reasons, there are restrictions. Regards.

To clarify this, the tool is writing to $HOME, as you can see from the traceback:

Traceback (most recent call last):
  File "/tmp/tmpzruufokr/galaxy-dev/test/functional/test_toolbox.py", line 98, in test_tool
    self.do_it(tool_version=tool_version, test_index=test_index)
  File "/tmp/tmpzruufokr/galaxy-dev/test/functional/test_toolbox.py", line 35, in do_it
    verify_tool(tool_id, self.galaxy_interactor, resource_parameters=resource_parameters, test_index=test_index, tool_version=tool_version, register_job_data=register_job_data)
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1038, in verify_tool
    raise e
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1034, in verify_tool
    job_stdio = _verify_outputs(testdef, test_history, jobs, tool_id, data_list, data_collection_list, galaxy_interactor, quiet=quiet)
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1208, in _verify_outputs
    raise JobOutputsError(found_exceptions, job_stdio)
galaxy.tool_util.verify.interactor.JobOutputsError: Job in error state.. tool_id: cp_cellprofiler4, exit_code: 1, stderr: Traceback (most recent call last):
  File "/usr/local/bin/cellprofiler", line 7, in <module>
    from cellprofiler.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/cellprofiler/__main__.py", line 24, in <module>
    from cellprofiler_core.object import ObjectSet
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/__init__.py", line 1, in <module>
    from ._object_set import ObjectSet
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/_object_set.py", line 1, in <module>
    from ._objects import Objects
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/_objects.py", line 9, in <module>
    from ..utilities.core.object import downsample_labels
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/utilities/core/object.py", line 3, in <module>
    import skimage.color
  File "/usr/local/lib/python3.8/site-packages/skimage/__init__.py", line 135, in <module>
    from .data import data_dir
  File "/usr/local/lib/python3.8/site-packages/skimage/data/__init__.py", line 270, in <module>
    _init_pooch()
  File "/usr/local/lib/python3.8/site-packages/skimage/data/__init__.py", line 255, in _init_pooch
    os.makedirs(data_dir, exist_ok=True)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/runner/.cache'

The home directory is often not writable on HPC systems, and never when running in docker. This is fixed by using a profile version >= 18.01 in 884f9e6

What this does is provide a home directory for the job that is within the job working directory, and so is mounted R/W in docker. All new tools should use a profile version that shouldn't be older than about a year or two (see here for the recommendations for submitting tools to the IUC: https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#tool-profile).

gallardoalba · 2021-10-29T11:05:04Z

In that case, the reason is that in local don't exist such problems with folder permissions, but in Galaxy, due to security reasons, there are restrictions. Regards.

To clarify this, the tool is writing to $HOME, as you can see from the traceback:

Traceback (most recent call last):
  File "/tmp/tmpzruufokr/galaxy-dev/test/functional/test_toolbox.py", line 98, in test_tool
    self.do_it(tool_version=tool_version, test_index=test_index)
  File "/tmp/tmpzruufokr/galaxy-dev/test/functional/test_toolbox.py", line 35, in do_it
    verify_tool(tool_id, self.galaxy_interactor, resource_parameters=resource_parameters, test_index=test_index, tool_version=tool_version, register_job_data=register_job_data)
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1038, in verify_tool
    raise e
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1034, in verify_tool
    job_stdio = _verify_outputs(testdef, test_history, jobs, tool_id, data_list, data_collection_list, galaxy_interactor, quiet=quiet)
  File "/tmp/tmpzruufokr/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 1208, in _verify_outputs
    raise JobOutputsError(found_exceptions, job_stdio)
galaxy.tool_util.verify.interactor.JobOutputsError: Job in error state.. tool_id: cp_cellprofiler4, exit_code: 1, stderr: Traceback (most recent call last):
  File "/usr/local/bin/cellprofiler", line 7, in <module>
    from cellprofiler.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/cellprofiler/__main__.py", line 24, in <module>
    from cellprofiler_core.object import ObjectSet
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/__init__.py", line 1, in <module>
    from ._object_set import ObjectSet
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/_object_set.py", line 1, in <module>
    from ._objects import Objects
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/object/_objects.py", line 9, in <module>
    from ..utilities.core.object import downsample_labels
  File "/usr/local/lib/python3.8/site-packages/cellprofiler_core/utilities/core/object.py", line 3, in <module>
    import skimage.color
  File "/usr/local/lib/python3.8/site-packages/skimage/__init__.py", line 135, in <module>
    from .data import data_dir
  File "/usr/local/lib/python3.8/site-packages/skimage/data/__init__.py", line 270, in <module>
    _init_pooch()
  File "/usr/local/lib/python3.8/site-packages/skimage/data/__init__.py", line 255, in _init_pooch
    os.makedirs(data_dir, exist_ok=True)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/runner/.cache'

The home directory is often not writable on HPC systems, and never when running in docker. This is fixed by using a profile version >= 18.01 in 884f9e6

What this does is provide a home directory for the job that is within the job working directory, and so is mounted R/W in docker. All new tools should use a profile version that shouldn't be older than about a year or two (see here for the recommendations for submitting tools to the IUC: https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#tool-profile).

Great to know, thanks!

sunyi000 · 2021-10-29T11:08:20Z

thank you very much for the clarification and all the helps @mvdbeek @gallardoalba

jkh1 · 2022-04-25T14:32:20Z

I was looking for it in the toolshed and realized it wasn't there yet. It looks like all checks are fine. Is there any other thing that's holding up the merge?

gallardoalba · 2022-04-25T18:19:45Z

I was looking for it in the toolshed and realized it wasn't there yet. It looks like all checks are fine. Is there any other thing that's holding up the merge?

Done!

jkh1 · 2022-04-26T05:07:22Z

Thanks a lot.

sunyi000 added 5 commits September 15, 2021 11:56

cp4.2.1

025a619

add shed file

bfa5118

added images.tar

9bd0819

update python3.8 in ci.yaml

4d10b70

update ci.yaml python3.8

1ba01b9

gallardoalba reviewed Sep 22, 2021

View reviewed changes

tools/cellprofiler_v4/cellprofiler.xml Outdated Show resolved Hide resolved

remove single quote around $detailed_output

5424e0b

Co-authored-by: Cristóbal Gallardo <gallardoalbac@gmail.com>

gallardoalba reviewed Sep 22, 2021

View reviewed changes

tools/cellprofiler_v4/cellprofiler.xml Show resolved Hide resolved

sunyi000 and others added 2 commits September 22, 2021 16:11

revert back to py3.7 in ci.yaml

401189f

export mplconfdir

01c478d

Co-authored-by: Cristóbal Gallardo <gallardoalbac@gmail.com>

rename cp_version to tool_version. modified tool id to avoid conflict…

daa19b2

… with cp3

simonbray reviewed Sep 23, 2021

View reviewed changes

tools/cellprofiler_v4/cellprofiler.xml Outdated Show resolved Hide resolved

added profile to <tool>

884f9e6

Co-authored-by: Simon Bray <32272674+simonbray@users.noreply.github.com>

gallardoalba approved these changes Apr 25, 2022

View reviewed changes

gallardoalba merged commit 4cb71c4 into bgruening:master Apr 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add cellprofiler v4.2.1 tool #1163

add cellprofiler v4.2.1 tool #1163

sunyi000 commented Sep 15, 2021

sunyi000 commented Sep 15, 2021 •

edited

bgruening commented Sep 15, 2021

sunyi000 commented Sep 15, 2021

gallardoalba commented Sep 22, 2021 •

edited

sunyi000 commented Sep 22, 2021

sunyi000 commented Sep 22, 2021 •

edited

gallardoalba commented Sep 22, 2021

simonbray commented Sep 23, 2021

gallardoalba commented Sep 23, 2021 •

edited

sunyi000 commented Sep 23, 2021

gallardoalba commented Sep 23, 2021

mvdbeek commented Oct 29, 2021 •

edited

gallardoalba commented Oct 29, 2021

sunyi000 commented Oct 29, 2021 •

edited

jkh1 commented Apr 25, 2022

gallardoalba commented Apr 25, 2022

jkh1 commented Apr 26, 2022

add cellprofiler v4.2.1 tool #1163

add cellprofiler v4.2.1 tool #1163

Conversation

sunyi000 commented Sep 15, 2021

sunyi000 commented Sep 15, 2021 • edited

bgruening commented Sep 15, 2021

sunyi000 commented Sep 15, 2021

gallardoalba commented Sep 22, 2021 • edited

sunyi000 commented Sep 22, 2021

sunyi000 commented Sep 22, 2021 • edited

gallardoalba commented Sep 22, 2021

simonbray commented Sep 23, 2021

gallardoalba commented Sep 23, 2021 • edited

sunyi000 commented Sep 23, 2021

gallardoalba commented Sep 23, 2021

mvdbeek commented Oct 29, 2021 • edited

gallardoalba commented Oct 29, 2021

sunyi000 commented Oct 29, 2021 • edited

jkh1 commented Apr 25, 2022

gallardoalba commented Apr 25, 2022

jkh1 commented Apr 26, 2022

sunyi000 commented Sep 15, 2021 •

edited

gallardoalba commented Sep 22, 2021 •

edited

sunyi000 commented Sep 22, 2021 •

edited

gallardoalba commented Sep 23, 2021 •

edited

mvdbeek commented Oct 29, 2021 •

edited

sunyi000 commented Oct 29, 2021 •

edited