Batch download original files 12137 #2220

will-moore · 2014-03-28T01:04:14Z

This addresses https://trac.openmicroscopy.org.uk/ome/ticket/12137

This PR is on-top of #2219 which should be merged first (or I guess it could be closed if we merge this).

To test, select multiple images in web and try to download from toolbar in right panel.
Try some multi-file images and see how long it takes!
You are given the file count and size of files (before being zipped) in the download option.

sbesson · 2014-03-28T09:20:17Z

components/tools/OmeroWeb/omeroweb/webclient/controller/container.py

+            if 'image' in objDict:
+                params = omero.sys.ParametersI()
+                params.map = {}
+                params.map["ids"] = rlist([rlong(a.getId()) for a in objDict['image']])


As you are using omero.sys.ParametersI(), the map should be automatically initialized and you should be able to use

params.addIds([a.getId() for a in objDict['image']])

But how are these then mapped to the key "ids"?

https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/blitz/src/omero/sys/ParametersI.java#L474
addIds() maps the input to ome.parameters.Parameters.IDS which is defined as ids in https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/model/src/ome/parameters/Parameters.java#L59

will-moore · 2014-03-31T13:48:57Z

The re-working of the file count and size message fixes the issues that @manics raised in #2219. Only remaining question is what to name the zip when downloading multiple images. @imunro Any suggestions?

imunro · 2014-03-31T14:43:34Z

Would it be easy to prompt the user for a name?

imunro · 2014-04-01T10:21:56Z

Tested with some of our .sdt files. Behaved as expected (including correct handling of multi-image filesets) . Downloaded files are identical to originals. Looks good. Nice work.

joshmoore · 2014-04-07T15:49:44Z

@will-moore:

I assume the description is wrong when it says "not ready for merging" right?
and this must be merged along with Original File download 12136 #2219, correct?

will-moore · 2014-04-07T20:35:12Z

@joshmoore This PR is on top of #2219, but both need testing again, just to confirm both single image and multiple images are downloading OK, and there are no bugs when including images that have Pixels instead of Filesets etc.

manics · 2014-04-09T10:26:27Z

See #2219 (comment)

manics · 2014-04-10T09:22:13Z

#2219 (comment)

will-moore · 2014-04-14T23:27:14Z

Main bug reported on #2219 was that OMERO 4 images that had file Archived on import did not allow these files to be 'batch downloaded' when multiple images are selected at once. This is now fixed.

I also added the size of the archived / fileset files when a single image is selected (previously the download menu only showed the count of files.

In addition, I added a couple of new methods to the Blitz Gateway.
This now renders several methods redundant, namely

image.countImportedImageFiles()
image.countFilesetFiles()
image.countArchivedFiles()

However, I won't remove these for 5.0.x since that would be a breaking API change.
But they could be removed from 5.1

will-moore · 2014-04-14T23:30:44Z

@chris-allan @cneves @knabar BlitzGateway API additions above (and proposed removals when rebasing this to develop).

joshmoore · 2014-04-15T06:26:28Z

(NB: this PR will need origin/dev_5_0 merged in)

jburel · 2014-04-15T08:28:03Z

@will-moore: do you have a DB that we can test against since we need a mix of 4 and 5 data.

will-moore · 2014-04-15T08:29:24Z

See the various images on gretzky that are linked from the discussion at #2219

jburel · 2014-04-15T08:35:41Z

@will-moore: thanks missed it

…ginal_files_12137 Conflicts: components/tools/OmeroPy/src/omero/gateway/__init__.py

jburel · 2014-04-16T08:16:27Z

The download options are a bit strange.
If click on an image I have OME-TFF, JPEG, if I have multi-selection, no option only the "0 archived". (for example I click on https://gretzky.openmicroscopy.org.uk/omero/webclient/?show=image-8920%7Cimage-8919%7Cimage-8918)
It will be preferable to have a menu like "original, OME-TIFF, JPEG" certain actions being greyed out depending on the context.

I just noticed that the point was raised by @manics on gh-2219
This should be addressed

jburel · 2014-04-16T08:21:33Z

https://gretzky.openmicroscopy.org.uk/omero/webclient/?show=image-9428%7Cimage-9426%7Cimage-9425%7Cimage-9424%7Cimage-9422
5 images selected: 8 archived files. From a usability point of view, counting the log file is misleading. The user won't consider the "log" as an image+ it is not displayed in the browser.

jburel · 2014-04-16T08:42:04Z

The download of mixed archived (version 4) and image imported in 5 is working.
Issue with download dialog:

pops up on the other side of the screen i.e. click on right-hand side, dialog pops up on the left
Download is completed. But the wheel is still spinning

Still some work to be done before the functionality can be merged in and having 2 PR makes it difficult to follow e.g comments

jburel · 2014-04-16T10:27:03Z

scoping session to schedule with @gusferguson when he gets back

will-moore · 2014-04-16T23:28:30Z

This PR was only due to support download of original files. If we want to also support jpeg, OME-TIFF etc, that should be handled in a different PR.

I don't think we should distinguish between log files and other files, since we don't have any logic in the clients to know the difference, and we don't know what users expect for every different format. All we can do is state the facts.

We don't have any way (as far as I know) for the web page to manage the download of files, to know when the download starts or stops. All I can do is show some placeholder prior to requesting the download. Since the download request takes some time (grabbing all the files to a temp folder and zipping them up) I show a spinner gif, but have no control over it after that.

jburel · 2014-04-17T07:36:20Z

There is a major change in the UI depending on action. That needs to be addressed, downloading as OME-TIFF does not have to be implemented in that PR but we need to keep some consistency in the UI.
We need to review where we are before going ahead and we now have 3 clients (CLI, web, insight) with "download" functionalities (note that the wording needs to be unified too since we have export/download etc) and they are not always following the same approach/displaying the same info.

chris-allan · 2014-04-17T07:48:41Z

Just to add my 2 cents on download and export I definitely agree wholeheartedly with @jburel. Especially with respect to thinking seriously about the implications of this PR in a bit wider scope.

The state of features surrounding download and export functionality has been pretty chaotic of late. In the last few versions we've moved it around in the user interface at least twice and changed the semantics at least once. Our user facing documentation (http://help.openmicroscopy.org/scripts.html) is now even out of date.

We'd certainly have to forgive an end user at this point for being pretty frustrated at the level of flip-flopping and their confusion as to what functionality is where and how to use it. We add to that sense of chaos at our peril.

/cc @gusferguson

will-moore · 2014-04-22T12:31:08Z

OK - so do I wait for @gusferguson to review this or do I go ahead and add support for JPEG to the multi-image download menu?

jburel · 2014-04-22T12:33:37Z

@will-moore: wait for @gusferguson to do a review of what we have already in-place and come up with proposal before going ahead. We can then adjust the various clients.

jburel · 2014-04-24T11:04:08Z

@gusferguson is currently reviewing the script menu and the download options and terminology.

jburel · 2014-05-01T09:28:25Z

Closing this PR for now as discussed during yesterday meeting.

will-moore added 3 commits March 27, 2014 14:36

Use countImportedImageFiles for download. Fixes #12136

e678b04

Container.py getFilesetFileInfo() - see #12137

9deeba9

Batch Annotate panel toolbar with download

8e3a240

sbesson reviewed Mar 28, 2014
View reviewed changes

will-moore added 2 commits March 28, 2014 11:23

Remove duplicate files from download info

7a84889

Remove duplicate files from download zip

208bbfb

jburel added the dev_5_0 label Mar 28, 2014

will-moore added 2 commits March 28, 2014 15:54

Use HttpStreamingResponse to fix deprecation warning

2ddadcf

Show placeholder popup while preparing download

434d72b

will-moore mentioned this pull request Mar 31, 2014

Original File download 12136 #2219

Merged

Fix show_link button for batch annotate panel

16f40a8

will-moore added 2 commits April 1, 2014 00:31

Basic form for download 'zip name'

2447d2e

Fix archived downloads for single image

ec7b81b

Handle zero or 1 archived file for multiple images

9b90737

will-moore added 7 commits April 14, 2014 14:30

getArhivedFilesInfo() for OMERO 4 images

891488f

Add archivedFilesInfo to filesetInfo

81f7868

Add BlitzGateway getFilesetFilesInfo() and getArchivedFilesInfo()

c4af619

Blitz count*Files() uses get*FilesInfo()

31c9d98

web uses conn.getFilesetFilesInfo() methods

be522cf

Blitz getImportedFilesInfo() new method

2c4cd28

Web uses image.getImportedFilesInfo()

70149ef

Don't use image.countImportedImageFiles()

3f7a4d7

Merge remote-tracking branch 'origin/dev_5_0' into batch_download_ori…

fa38f89

…ginal_files_12137 Conflicts: components/tools/OmeroPy/src/omero/gateway/__init__.py

sbesson mentioned this pull request Apr 15, 2014

CLI download improvements #2294

Merged

joshmoore added the ON_HOLD label Apr 24, 2014

jburel closed this May 1, 2014

will-moore mentioned this pull request May 2, 2014

Remove cut link 11882 (rebased onto dev_5_0) #2310

Merged

This was referenced Jun 23, 2014

Batch download originals jpegs #2671

Closed

Batch download originals jpegs #2672

Merged

atarkowska mentioned this pull request Jul 7, 2014

Batch download originals jpegs (rebased onto develop) #2763

Merged

will-moore deleted the batch_download_original_files_12137 branch February 18, 2019 04:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch download original files 12137 #2220

Batch download original files 12137 #2220

will-moore commented Mar 28, 2014

sbesson Mar 28, 2014

will-moore Mar 28, 2014

sbesson Mar 28, 2014

will-moore commented Mar 31, 2014

imunro commented Mar 31, 2014

imunro commented Apr 1, 2014

joshmoore commented Apr 7, 2014

will-moore commented Apr 7, 2014

manics commented Apr 9, 2014

manics commented Apr 10, 2014

will-moore commented Apr 14, 2014

will-moore commented Apr 14, 2014

joshmoore commented Apr 15, 2014

jburel commented Apr 15, 2014

will-moore commented Apr 15, 2014

jburel commented Apr 15, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

will-moore commented Apr 16, 2014

jburel commented Apr 17, 2014

chris-allan commented Apr 17, 2014

will-moore commented Apr 22, 2014

jburel commented Apr 22, 2014

jburel commented Apr 24, 2014

jburel commented May 1, 2014

Batch download original files 12137 #2220

Batch download original files 12137 #2220

Conversation

will-moore commented Mar 28, 2014

sbesson Mar 28, 2014

Choose a reason for hiding this comment

will-moore Mar 28, 2014

Choose a reason for hiding this comment

sbesson Mar 28, 2014

Choose a reason for hiding this comment

will-moore commented Mar 31, 2014

imunro commented Mar 31, 2014

imunro commented Apr 1, 2014

joshmoore commented Apr 7, 2014

will-moore commented Apr 7, 2014

manics commented Apr 9, 2014

manics commented Apr 10, 2014

will-moore commented Apr 14, 2014

will-moore commented Apr 14, 2014

joshmoore commented Apr 15, 2014

jburel commented Apr 15, 2014

will-moore commented Apr 15, 2014

jburel commented Apr 15, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

jburel commented Apr 16, 2014

will-moore commented Apr 16, 2014

jburel commented Apr 17, 2014

chris-allan commented Apr 17, 2014

will-moore commented Apr 22, 2014

jburel commented Apr 22, 2014

jburel commented Apr 24, 2014

jburel commented May 1, 2014