Acquisition doesn't acquire images

[impact27]

With micromanager I can acquire a stack of 10 images:

I would like to do the same with pycromanager. I replaced the acquisition engine as instructed in #41 and run this code:

from pycromanager import Acquisition, multi_d_acquisition_events

if __name__ == '__main__': #this is important, don't forget it

    with Acquisition(directory='C:/Users/Quentin.Peter/Desktop/tmp0', name='acquisition_name') as acq:
        events = multi_d_acquisition_events(10)
        acq.acquire(events)

A folder is created (C:/Users/Quentin.Peter/Desktop/tmp0/acquisition_name_1/Full resolution) and the following windows appears:

But then nothing happens. The python code is stuck in Acquisition.await_completion. If I try to close the window, it asks if I want to Finish acquisition? and I I press OK, the python code finishes.

[henrypinkard]

Seems like this was caused by a bug in Java that I still don't understand, but should be now fixed. Will be in Micro-manager nightly builds after (micro-manager/micro-manager#822). In the mean time, you can replace all three of NDTiffStorage.jar, Pycromanager.jar, and AcqEngJ.jar with their latest versions, available here

[henrypinkard]

Is this bug fixed?

[impact27]

With 20200605 I still have the same issue.

[henrypinkard]

Hmmm. Unfortunately I cannot reproduce this anymore on either Windows or OSX. Is it possible one of the older versions of the libraries remains behind? Maybe you could try installing 20200605 into its own new directory? I wonder also if anything suspicious shows up in your core log (in the CoreLogs folder in your mm install directory)

[impact27]

I uninstalled micromanager, emptied the bin, and installed 20200606, and the issue persists (win10). The log is:
CoreLog20200607T165530_pid19912.txt

[impact27]

It is behaving exactly the same if I replace acq.acquire(events) by acq.await_completion(), so it looks like the acquisiting event is never recieved by micro manager.

[henrypinkard]

I see near the top of the core log: MM Studio version: 2.0.x dev. Are you running micro-manager directly from the source code rather than the desktop application?

[henrypinkard]

If so, I would guess that your project may be linking to an older version of the dependency used for writing data. The previous bug I fixed lived inside the NDTiffStorage.jar library.

So the acquisition event is received, and the image is acquired, but it cannot save so it does not display. Ideally, this would throw an error, but unfortunately I couldn't get it to log an error, as it seemed to be bug within Java itself.

[impact27]

I see near the top of the core log: MM Studio version: 2.0.x dev. Are you running micro-manager directly from the source code rather than the desktop application?

I installed it from https://valelab4.ucsf.edu/~MM/nightlyBuilds/2.0.0-gamma/Windows/ and run from the start menu.

[impact27]

The NDTiffStorage.jar in C:/Program file/Micro-Manager-2.0gamma/plugins/Micro-manager has version 0.2.6

[henrypinkard]

That's strange. @nicost do you have any idea why MM Studio version: 2.0.x dev shows up when not running from source, and if this might be connected to library versions?

[henrypinkard]

Actually, I think you might be right that the problem is somewhere in the acquisition events, because I don't see the information about images actually being acquired in your corelog.

I made new versions of both the python and java side with logging related to acquisition events. The python side is in pycromanager 0.3.2. This time run your acquisition with debug=True and it should print stuff about acquisition events as they go. The java side is available in 0.7.5 here. Just drop in and replace as usual. The logging will automatically go into the core log.

[impact27]

I got it to work by replacing import multiprocessing by import multiprocessing.dummy as multiprocessing in acquire.py (https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.dummy)

[nicost]

That's strange. @nicost do you have any idea why MM Studio version: 2.0.x dev shows up when not running from source, and if this might be connected to library versions?

Mine says:
2020-06-07T12:46:46.291727 tid11832 [IFO,App] MM Studio version: 2.0.0-gamma1 20200605
2020-06-07T12:46:46.291727 tid11832 [IFO,App] MMCore version 10.1.0
2020-06-07T12:46:46.291727 tid11832 [IFO,App] Device API version 69, Module API version 10
2020-06-07T12:46:46.291727 tid11832 [IFO,App] Operating System: Windows 10 (amd64) 10.0
2020-06-07T12:46:46.291727 tid11832 [IFO,App] JVM: Java HotSpot(TM) 64-Bit Server VM, version 1.8.0_152, 64-bit

[henrypinkard]

Ah okay great! Then that suggests this is purely on the Python side, and that you're right, the events never actually get sent over. Events are designed to be constantly pulled from Acquisition._event_queue on their own dedicated thread. I use multiprocessing for this instead of threading, with the logic that we wouldn't want to slow down the event sending because some other processing is holding the global interpreter lock. I've since learned thatmultiprocessing behaves differently on windows than unix with unexpected (to me) consequences (#9).

Maybe this is another such instance. Still not sure why I didn't see it on windows though. Which version of Python are you using @impact27?

[impact27]

I think the multiprocessing problem is unrelated to pycromanager so micro-manager/micro-manager#822 solved this issue. It runs with python 3.7.6

For the story:

I am running with ipython and it doesn't work there on windows.
The reason is that pickle can not serialise functions. (Well it pretends it can but in reality it can't), Instead it just remember in which file you can find it. That means the process will import the file, so if you don't have if __name__ == "__main__":, the import will create a new process that will import that will ... and so on. So this works:

import multiprocessing

def say_hello(pipe):
    pipe.send("Hello")
    

if __name__ == "__main__":
    parent_conn, child_conn = multiprocessing.Pipe()
    processor_thread = multiprocessing.Process(target=say_hello,
                                               args=(child_conn,))
    processor_thread.start()
    print(parent_conn.recv())
    processor_thread.join()

But this doesn't:

IPython 7.13.0 -- An enhanced Interactive Python.

In [1]: import multiprocessing
   ...: 
   ...: def say_hello(pipe):
   ...:     pipe.send("Hello")
   ...:     

In [2]: parent_conn, child_conn = multiprocessing.Pipe()
   ...: processor_thread = multiprocessing.Process(target=say_hello,
   ...:                                            args=(child_conn,))
   ...: processor_thread.start()
   ...: print(parent_conn.recv())
   ...: processor_thread.join()

[henrypinkard]

Glad you were able to figure it out. It's unfortunate that multiprocessing is so non-intuitive on windows. Might be worth thinking about alternatives. For acquisition events it would be pretty easy to swap in multiple threads instead of multiple processes, and I doubt there would much (if any) performance hit, but image processors and acquisition hooks run by the same mechanism, and if using both at the same time global interpreter lock blocking could become a problem. I made a new issue to think about this more (#46). Let me know if you have any ideas.

[impact27]

Multiprocessing.dummy is using threads so using threads would be easy as it has the same api as multiprocessing.

[impact27]

Actually I think I was mistaken, the problem is with the editor I am using. I created a PR there (spyder-ide/spyder-kernels#229). I think the multiprocessing part of the bug really doesn't have anything to do with micromanager. Indeed this works:

In [1]: from pycromanager import Acquisition, multi_d_acquisition_events
   ...: with Acquisition(directory='C:/Users/Quentin.Peter/Desktop/tmp0', name='acquisition_name') as acq:
   ...:     events = multi_d_acquisition_events(10)
   ...:     acq.acquire(events)
   ...:     

In [2]:

[henrypinkard]

Ok great. In that case maybe there's not so much of a motivation to switch to threading from multiprocessing after all

[jmmanley]

Hi, I was super excited to see this tool! I seem to be having the same issue described originally by @impact27. Multi-d acquisition works in the desktop app for me, but not when I run the following script:

from pycromanager import Acquisition, multi_d_acquisition_events

if __name__ == '__main__':
        with Acquisition(directory='C:/Users/jmanley/Desktop/test_acq/', name='test_acq', debug=True) as acq:
            events = multi_d_acquisition_events(10)
            acq.acquire(events)

with debug=True, I get the following output:

connecting 4827
connecting 4827
binding 4828
got event(s): [{'axes': {'time': 0}}, {'axes': {'time': 1}}, {'axes': {'time': 2}}, {'axes': {'time': 3}}, {'axes': {'time': 4}}, {'axes': {'time': 5}}, {'axes': {'time': 6}}, {'axes': {'time': 7}}, {'axes': {'time': 8}}, {'axes': {'time': 9}}]
sent events
got event(s):
None

(FYI, I got an error with debug in acquire.py line 20, so I changed to: print('got event(s):', events))

The script completes, but no images seem to be acquired or saved. CoreLog: CoreLog20200619T095812_pid11668.txt

I am using Windows, python 3.7.6, micromanager version 2.0.0-gamma1 20200618, pycromanager version 0.3.3 (and from source). Using multiprocessing.dummy didn't fix.

Any ideas what could be going on? Many thanks!

[henrypinkard]

Not quite sure. What do you mean by "desktop app"? From an IDE? And then when it doesn't work, that's running the script from the command line?

[jmmanley]

Thanks for the quick response! By desktop app, I just meant in the micromanager GUI application itself, where everything works fine. Yes, it is when I run the script in the command line that it doesn't work.

[henrypinkard]

Got it. Those are actually running different libraries under the hood, so working in MM GUI doesn't tell us much. I think this issue might be on the Java side. Could you try running with different python setups to see if that changes anything? In addition to running script, you could try entering python on command line and typing each command in the script individually? And then maybe from an IDE like pycharm?

[henrypinkard]

If none of those work I can add some more logging into the Java side

[henrypinkard]

Just saw and fixed another bug that acquisitions can fail silently if there's not enough disk space. For performance reasons, the saving files allocate much more space than needed (4GB) before truncating. Is it possible you're saving to a HD that's almost full?

[jmmanley]

No luck for me with pycharm or the command line: still no images saved and logs, etc. look the same. I tried saving to another disk with plenty of space and no luck there either.

I agree it would be nice to log the Java side - would the relevant "receiver" of the messages be in the core micro-manager repo or here in the ZMQ Java classes?

[henrypinkard]

It would be dispatched through the ZMQ Java classes to the acquisition library, and eventually be saved by this data writing library, and displayed by this viewer library.

Do you have experience with Java development?

[jmmanley]

I only have a little experience with Java, so maybe above my head..

[henrypinkard]

Okay, could I remote control your system with teamviewer to investigate?

[henrypinkard]

@jmmanley still having an issue with this?

[jmmanley]

Hey @henrypinkard, sorry for the delay! Just tried again. Not sure what’s changed, but things seem to be working with the latest nightly build! :)