Import notification #1876

jburel · 2013-12-05T15:53:20Z

Review handling of event returned during the import process
see https://trac.openmicroscopy.org.uk/ome/ticket/11146

To test:

Select a collection of files e.g. dicom folder
Click import.
Make sure that all successful imports indicate "Complete" on the second progress bar.

pwalczysko · 2013-12-06T13:39:40Z

Works as expected on Mac and Windows. I am not able to repeat the original bug with older builds of Insight, but I understand that this is a bit elusive one.

jburel · 2013-12-08T21:36:24Z

A good test is to import using a slow connection e.g. from home

pwalczysko · 2013-12-09T10:11:54Z

I just got an issue which might be connected to this PR (although not directly testing it).
On omero4-demo server, the develop server, import public/images/HCS/BBC/bbbc017 (~90 GB plate).
Using Insight OMERO.insight-5.0.0-beta2-RC2-335-a99bd54-ice33-b513-mac (this should be an Insight which was built with all the commits of this PR in).
Got Java Heap Space problem (which is possibly fine), but see below for the reaction of IMporter -

Importer did not stop importing, although all imports were either successful or failed
Importer makes impression like if it is still doing something (the progress bars are "refreshing", but always the same values are being displayed = this is a bogus progress
it seems like any java heap space errror will end up with such unexpected importer state


java.lang.OutOfMemoryError: Java heap space
    at IceInternal.BasicStream.readString(BasicStream.java:1338)
    at Ice.ContextHelper.read(ContextHelper.java:45)
    at omero.model.Details.__read(Details.java:502)
    at IceInternal.BasicStream.readObject(BasicStream.java:1590)
    at IceInternal.BasicStream.readPendingObjects(BasicStream.java:1739)
    at omero.cmd._HandleDelM.getRequest(_HandleDelM.java:168)
    at omero.cmd.HandlePrxHelper.getRequest(HandlePrxHelper.java:243)
    at omero.cmd.HandlePrxHelper.getRequest(HandlePrxHelper.java:217)
    at ome.formats.importer.ImportLibrary$ImportCallback.loadLogFile(ImportLibrary.java:515)
    at ome.formats.importer.ImportLibrary$ImportCallback.<init>(ImportLibrary.java:510)
    at ome.formats.importer.ImportLibrary.createCallback(ImportLibrary.java:489)
    at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageFile(OMEROGateway.java:6368)
    at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:236)
    at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1286)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:77)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:53)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:102)
    at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
    at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
    at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
    at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
    at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
    at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
    at java.lang.Thread.run(Thread.java:680)

    at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageFile(OMEROGateway.java:6376)
    at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:236)
    at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1286)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:77)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:53)
    at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:102)
    at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
    at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
    at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
    at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
    at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
    at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
    at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at IceInternal.BasicStream.readString(BasicStream.java:1338)
    at Ice.ContextHelper.read(ContextHelper.java:45)
    at omero.model.Details.__read(Details.java:502)
    at IceInternal.BasicStream.readObject(BasicStream.java:1590)
    at IceInternal.BasicStream.readPendingObjects(BasicStream.java:1739)
    at omero.cmd._HandleDelM.getRequest(_HandleDelM.java:168)
    at omero.cmd.HandlePrxHelper.getRequest(HandlePrxHelper.java:243)
    at omero.cmd.HandlePrxHelper.getRequest(HandlePrxHelper.java:217)
    at ome.formats.importer.ImportLibrary$ImportCallback.loadLogFile(ImportLibrary.java:515)
    at ome.formats.importer.ImportLibrary$ImportCallback.<init>(ImportLibrary.java:510)
    at ome.formats.importer.ImportLibrary.createCallback(ImportLibrary.java:489)
    at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageFile(OMEROGateway.java:6368)
    ... 12 more

jburel · 2013-12-09T10:14:11Z

Out of Memory: not related to the PR.

ximenesuk · 2013-12-09T10:28:04Z

Tested over 3G connection with 3 folders of images, 4 dicom, 65 png, 8 dv, each folder as a single import queue. All notifications looks good showing Complete.

joshmoore · 2013-12-09T13:58:55Z

non-whitespaced diff is:

@@ -635,6 +631,9 @@ public class StatusLabel
         if (event == null) return;
         cancellable = false;
         if (event instanceof ImportEvent.IMPORT_DONE) {
+            step = 6;
+            processingBar.setValue(step);
+            processingBar.setString(STEPS.get(step));
             pixels = (Set<PixelsData>) PojoMapper.asDataObjects(
                     ((ImportEvent.IMPORT_DONE) event).pixels);
             firePropertyChange(IMPORT_DONE_PROPERTY, null, this);
@@ -719,10 +718,6 @@ public class StatusLabel
             step = 5;
             processingBar.setValue(step);
             processingBar.setString(STEPS.get(step));
-       } else if (event instanceof ImportEvent.OBJECTS_RETURNED) {
-           step = 6;
-           processingBar.setValue(step);
-           processingBar.setString(STEPS.get(step));

Once other formatting issues are done, I think this is good to go. (@jburel: there was a further issue of the "Scanning" state in ticket 10740 if you'd also like to look at it here).

jburel · 2013-12-10T08:53:23Z

@pwalczysko: @ximenesuk will add a new event "prep upload" so we can notify the user that something is happening. see https://trac.openmicroscopy.org.uk/ome/ticket/10740 cc @joshmoore

ximenesuk · 2013-12-11T09:41:12Z

@bpindelski You should also be able to see the new notification in the cli import output. So can you please test the cli as well as Insight?

bpindelski · 2013-12-11T10:50:54Z

Tested Insight (Mac, Windows) and CLI. The new event is indeed reported to the user. Changes look OK. Good to merge.

pwalczysko · 2013-12-11T14:15:35Z

Tested INCELL2000 (9 GB) which I was testing before. Now imported locally (omero-merge-develop). = with this PR
Scanning 41 minutes
Preparing Upload 1 minute
Upload 13 minutes
Total import 66 minutes

See movie on team/pwalczysko/fs reimport of INcell 9 Gb
There is no more "Unknown format" message, which is good.

See also Jmarie's answer to this comment on #1885 (comment)

Import notification

jburel · 2013-12-11T15:42:23Z

--no-rebase

jburel added 2 commits December 5, 2013 15:46

Review notification sequence.

f77e6a9

Remove tabs and whitespace.

bfadcb5

jburel added 2 commits December 9, 2013 17:00

Format.

9d14cb2

Modify constructor to handle directory correctly.

1faea57

ximenesuk and others added 2 commits December 10, 2013 16:26

Add event before upload location is resolved.

a1a222a

Add support for new Prep Upload event

35d017a

ximenesuk mentioned this pull request Dec 11, 2013

Reduce the number of transactions for import (See #10740) #1885

Merged

joshmoore added a commit that referenced this pull request Dec 11, 2013

Merge pull request #1876 from jburel/import-notification

0a2f078

Import notification

joshmoore merged commit 0a2f078 into ome:develop Dec 11, 2013

jburel deleted the import-notification branch February 3, 2014 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Import notification #1876

Import notification #1876

jburel commented Dec 5, 2013

pwalczysko commented Dec 6, 2013

jburel commented Dec 8, 2013

pwalczysko commented Dec 9, 2013

jburel commented Dec 9, 2013

ximenesuk commented Dec 9, 2013

joshmoore commented Dec 9, 2013

jburel commented Dec 10, 2013

ximenesuk commented Dec 11, 2013

bpindelski commented Dec 11, 2013

pwalczysko commented Dec 11, 2013

jburel commented Dec 11, 2013

Import notification #1876

Import notification #1876

Conversation

jburel commented Dec 5, 2013

pwalczysko commented Dec 6, 2013

jburel commented Dec 8, 2013

pwalczysko commented Dec 9, 2013

jburel commented Dec 9, 2013

ximenesuk commented Dec 9, 2013

joshmoore commented Dec 9, 2013

jburel commented Dec 10, 2013

ximenesuk commented Dec 11, 2013

bpindelski commented Dec 11, 2013

pwalczysko commented Dec 11, 2013

jburel commented Dec 11, 2013