images from ROIs all ZT planes #137

will-moore · 2017-12-12T16:39:00Z

See https://trello.com/c/my7BGCEb/7-crop-iviewer-and-images-from-rois

This supports cropping a multi-Z or multi-T image to create a new image with the same number of Z or T planes.
This was supported before, but needed a multi-shape ROI with shapes on all the Z/T planes to be included.
Now, if an ROI has no Z set on its shape(s) then we use all Z planes. Same for T.
To test:

Using an image with multiple Z and T, create several rectangles, some with Z and T set, some with no Z or T set. See all combinations in screenshot.
Run the Images from ROIs on the image.
If a shape has no Z set, we create new image with all Z planes, same for T.
For screenshot below, we created 1 single-plane image, 1 single-Z timelapse, 1 single-T z-stack and 1 image of same Z and T size as original.

jburel · 2017-12-12T20:55:31Z

travis failure: see #136 (comment)

We could include the change in the devspace and ignore travis failure for now
but that might put us in a precarious position

joshmoore · 2017-12-19T06:42:53Z

Alternatively, we say it's time to open next_infra against develop.

jburel · 2017-12-19T10:43:12Z

@joshmoore probably

joshmoore · 2017-12-21T07:39:30Z

Since it's just the one PR and it's a fast-forward, I've pushed next_infra onto develop:

$ git show-branch origin/{develop,next_infra,workflows}
! [origin/develop] Merge pull request #135 from will-moore/label_rotate_fix
 ! [origin/next_infra] Merge pull request #136 from jburel/test-in-travis
  ! [origin/workflows] Merge pull request #135 from will-moore/label_rotate_fix
---
 -  [origin/next_infra] Merge pull request #136 from jburel/test-in-travis
 +  [origin/next_infra^2] Update name of test-omero
 +  [origin/next_infra^2^] Update to use test-omero
 +  [origin/next_infra^2~2] Parse inputs
 +  [origin/next_infra^2~3] Flake8 scripts
 +  [origin/next_infra^2~4] Fix typo
 +  [origin/next_infra^2~5] Add doc
 +  [origin/next_infra^2~6] Add line
 +  [origin/next_infra^2~7] Mark the movie test as fail
 +  [origin/next_infra^2~8] Enable check
 +  [origin/next_infra^2~9] Minor edit
 +  [origin/next_infra^2~10] Unify extension
 +  [origin/next_infra^2~11] Print error
 +  [origin/next_infra^2~12] Organize check
 +  [origin/next_infra^2~13] Change range
 +  [origin/next_infra^2~14] Fix script
 +  [origin/next_infra^2~15] Change image dimension
 +  [origin/next_infra^2~16] no longer install pillow and numpy via pip
 +  [origin/next_infra^2~17] Add export test
 +  [origin/next_infra^2~18] Add movie dependency
 +  [origin/next_infra^2~19] Add test
 +  [origin/next_infra^2~20] Install dependencies
 +  [origin/next_infra^2~21] Add more tests
 +  [origin/next_infra^2~22] Add extra line
 +  [origin/next_infra^2~23] Add new support for more versions
 +  [origin/next_infra^2~24] Exclude .DS_Store
 +  [origin/next_infra^2~25] Add support for multiple versions
 +  [origin/next_infra^2~26] adjustment
 +  [origin/next_infra^2~27] Review setup
 +  [origin/next_infra^2~28] Run pytest
 +  [origin/next_infra^2~29] Change permissions
 +  [origin/next_infra^2~30] Do not check readme
 +  [origin/next_infra^2~31] Run scripts in travis
--- [origin/develop] Merge pull request #135 from will-moore/label_rotate_fix

Closing and re-opening to launch travis.

joshmoore · 2017-12-21T08:02:14Z

test/integration/test_util_scripts.py::TestUtilScripts::test_images_from_rois FAILED [ 81%]
...
  File "/opt/omero/server/OMERO.server/lib/python/omero/gateway/__init__.py", line 3750, in createImageFromNumpySeq

    raise exc

ValueError: invalid value for argument 1 in operation `getTile'

joshmoore · 2018-01-16T09:08:18Z

@will-moore : did you want to try to fix this?

will-moore · 2018-01-16T12:33:33Z

@joshmoore That should fix the test.

In testing ome/omero-scripts#137 this was originally passing with Make_Image_Stack was False but failing if True. Now we test with both and check the created image has the expected sizeZ

joshmoore · 2018-01-16T13:11:54Z

./omero/util_scripts/Images_From_ROIs.py:234:17: N806 variable 'theZ' in function should be lowercase
./omero/util_scripts/Images_From_ROIs.py:235:17: N806 variable 'theT' in function should be lowercase

this was originally passing with Make_Image_Stack was False but failing if True. Now we test with both and check the created image has the expected sizeZ

will-moore · 2018-01-16T13:54:17Z

@joshmoore How do I run these tests locally?
I get some strange result with

$ ./setup.py test -t test/integration/test_util_scripts.py 
Version: ImageMagick 6.9.3-6 Q16 x86_64 2016-02-28 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2016 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules 
Delegates (built-in): bzlib freetype jng jpeg ltdl lzma png tiff xml zlib
Usage: import [options ...] [ file ]

I cloned...

git clone --recurse-submodules git://github.com/openmicroscopy/omero-test-infra .omero

But I don't see anything obvious there. Do I have to use docker to run tests?

joshmoore · 2018-01-16T13:55:01Z

From https://github.com/ome/scripts/blob/develop/.travis.yml#L15:

.omero/scripts-docker

will-moore · 2018-01-16T15:07:11Z

.omero/scripts-docker seems to do a lot of installing and flake8 checking (and failing) but I'm not seeing any tests run:

.omero/scripts-docker
...
./omero/export_scripts/Export_to_other_omero.py:157:9: E116 unexpected indentation (comment)
./omero/export_scripts/Export_to_other_omero.py:173:22: W292 no newline at end of file
flake8.main.application   MainProcess   1873 INFO     Found a total of 359 violations and reported 161

joshmoore · 2018-01-16T15:18:32Z

You mean locally? On travis, I see this:

test/integration/test_util_scripts.py::TestUtilScripts::test_images_from_rois[True] PASSED [ 78%]
test/integration/test_util_scripts.py::TestUtilScripts::test_images_from_rois[False] PASSED [ 82%]

under https://travis-ci.org/ome/scripts/builds/329477205#L671

i.e. 👍

will-moore · 2018-01-16T15:24:54Z

Yes, I want to be able to run the tests locally, 1 test at a time, while writing & debugging (before committing).

joshmoore · 2018-01-16T15:29:31Z

Hmmm.... a couple of things:

Mea culpa but I certainly was thinking more about the travis use-case than the developer use case. Happy for us to look into adding dev support, but it might take some back and forth.
Likely the first goal of that would be to add an argument or environment variable to choose a single test, which doesn't sound too hard.
It looks like though that you're not making it to the tests locally since flake8 is failing. Perhaps this is due to the state of your branch?? (Travis merges the branch into the mainline which may fix existing flake8 issues)
If that still doesn't help, then we could look into making the flaking optional, but that shouldn't be a first resort.

will-moore · 2018-01-16T15:47:19Z

Not sure if "mainline" is develop or test_infra, so I tried both, but no joy...

$ git fetch origin
$ git merge origin/develop
Already up-to-date.
$ git merge origin/next_infra
Already up-to-date.
$ .omero/scripts-docker
...
./omero/export_scripts/Export_to_other_omero.py:173:22: W292 no newline at end of file

Now I don't see the summary message from above

flake8.main.application   MainProcess   1873 INFO     Found a total of 359 violations and reported 161

but otherwise looks the same.

joshmoore · 2018-01-16T15:49:11Z

Can you try fixing that file and/or commenting out flake8 under .omero/py-setup?

will-moore · 2018-01-16T15:55:41Z

Ah, sorry - I've got a couple of untracked local files that are getting included. I'll move them...

will-moore · 2018-01-16T16:03:29Z

Passing now:
==================== 27 passed, 1 xfailed in 180.86 seconds ====================

joshmoore · 2018-01-16T16:03:47Z

Nice!

jburel · 2018-01-17T10:44:25Z

Not for that PR but we should add to the description of the new image
the ID of the source image. This is added only when the "stack option is on"

The stack option will need to be reviewed since we end up with a one channel image
not really consistent with the rest of the script.
I will create card for the various points

The changes made in this PR works as expected. The PR re-introduced some print statement
we should agree on the approach, useful for debugging but can be annoying in prod

Ready for 5.4.2

will-moore · 2018-01-17T11:05:39Z

Print statements are a simple way of logging to the stdout file that is shown to users at the "Info" button and we use them in all our scripts.
The Make Stack option was the original behaviour of this script since it applies to cryo-EM workflows where you typically compile many similar-shaped Rect ROIs on a single-plane image into a Z-stack.
We could add more explanation in the description, but it's already quite lengthy. Currently it includes "If you choose to 'make an image stack' from all the ROIs, the script will create a single new Z-stack image with a single plane from each ROI."

joshmoore · 2018-01-17T11:19:55Z

... useful for debugging but can be annoying in prod

Had never considered it before, but it'd be fairly straight-forward to have omero.scripts.client setting a debugging level so that all scripts could begin using logging and have the clients allow users to choose their verbosity.

joshmoore · 2018-01-18T12:36:38Z

@jburel will open a card for the remaining issues. Merging for inclusion in 5.4.2

If theZ or theT not set, use all Z or T planes

bed6a6f

jburel mentioned this pull request Dec 12, 2017

Test in travis #136

Merged

jburel changed the base branch from develop to workflows December 12, 2017 20:55

joshmoore closed this Dec 21, 2017

joshmoore reopened this Dec 21, 2017

Merge 'origin/develop' into HEAD

78dbeea

joshmoore changed the base branch from workflows to develop January 16, 2018 10:52

Handle 'Make_Image_Stack':True for Images_From_ROIs with theZ:None

326f82e

will-moore mentioned this pull request Jan 16, 2018

improve_test_Images_from_ROIs ome/openmicroscopy#5617

Closed

will-moore added 3 commits January 16, 2018 13:14

Rename theZ and theT to the_z and the_t

bc93053

improve_test_Images_from_ROIs

1b7926b

this was originally passing with Make_Image_Stack was False but failing if True. Now we test with both and check the created image has the expected sizeZ

flake8 & copy/paste error fixes

59da99e

rename image_stack for flake8

b1d28a6

will-moore changed the title ~~If theZ or theT not set, use all Z or T planes~~ images from ROIs all ZT planes Jan 16, 2018

jburel added this to the 5.4.2 milestone Jan 17, 2018

joshmoore merged commit 2b80718 into ome:develop Jan 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

images from ROIs all ZT planes #137

images from ROIs all ZT planes #137

will-moore commented Dec 12, 2017 •

edited by jburel

Loading

jburel commented Dec 12, 2017 •

edited

Loading

joshmoore commented Dec 19, 2017

jburel commented Dec 19, 2017

joshmoore commented Dec 21, 2017

joshmoore commented Dec 21, 2017

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018 •

edited

Loading

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

jburel commented Jan 17, 2018

will-moore commented Jan 17, 2018

joshmoore commented Jan 17, 2018

joshmoore commented Jan 18, 2018

images from ROIs all ZT planes #137

images from ROIs all ZT planes #137

Conversation

will-moore commented Dec 12, 2017 • edited by jburel Loading

jburel commented Dec 12, 2017 • edited Loading

joshmoore commented Dec 19, 2017

jburel commented Dec 19, 2017

joshmoore commented Dec 21, 2017

joshmoore commented Dec 21, 2017

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018 • edited Loading

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

will-moore commented Jan 16, 2018

will-moore commented Jan 16, 2018

joshmoore commented Jan 16, 2018

jburel commented Jan 17, 2018

will-moore commented Jan 17, 2018

joshmoore commented Jan 17, 2018

joshmoore commented Jan 18, 2018

will-moore commented Dec 12, 2017 •

edited by jburel

Loading

jburel commented Dec 12, 2017 •

edited

Loading

joshmoore commented Jan 16, 2018 •

edited

Loading