Re-enable MacOS support

[gselzer]

Until imagej/pyimagej#197 is resolved, we cannot support Mac for this plugin. The clash between imagej, which needs to run headless on mac, and napari, which makes very little sense running headless. As there seems to be an incomplete separation between the two on Mac, we will not support that OS until the issues with it are resolved.

[ctrueden]

You could try something like:

import imagej, napari
from jpype import createGuiEnvironment
imagej._create_jvm(endpoint, mode='gui')
def run_napari():
    ij = imagej._create_gateway()
    napari.Viewer # or whatever you want to do
setupGuiEnvironment(run_napari)

This replicates what mode='gui' does, but lets you control what happens inside the setupGuiEnvironment callback.

This requires pyobjc-core and pyobjc-framework-cocoa conda packages to be installed.

[gselzer]

You could try something like:

import imagej, napari
from jpype import createGuiEnvironment
imagej._create_jvm(endpoint, mode='gui')
def run_napari():
    ij = imagej._create_gateway()
    napari.Viewer # or whatever you want to do
setupGuiEnvironment(run_napari)

This replicates what mode='gui' does, but lets you control what happens inside the setupGuiEnvironment callback.

This requires pyobjc-core and pyobjc-framework-cocoa conda packages to be installed.

Hmm, I thought that imagej woud hang in these scenarios? Why is this not the normal behavior for imagej.init()?

I feel like I am missing something @ctrueden.

[ctrueden]

@gselzer It is the normal behavior for mode='gui', yes. But it blocks the main thread. The difference here is that you can call napari.Viewer() after the gateway gets created, but only from the callback function passed to setupGuiEnvironment. The setupGuiEnvironment function never returns.

So if you write:

ij = imagej.init(mode='gui')
napari.Viewer()

Then the napari.Viewer() call will never happen.

But if you write:

def do_stuff():
    imagej._create_gateway()
    napari.Viewer()

setupGuiEnvironment(do_stuff)

Then I believe the napari.Viewer() call will still happen after the gateway gets constructed.

Whether this will actually work (as in: both napari and ImageJ2 are usable without weird hangs) is another story. But it's worth a test.

[gselzer]

Alright, I gave it a shot:

tmp2.py:

import imagej 
import napari, skimage 
from jpype import setupGuiEnvironment 
 
imagej._create_jvm("2.5.0", mode='headless', add_legacy=False) 
def setup():  
        ij = imagej._create_gateway() 
        napari.view_image(skimage.data.cells3d()) 
setupGuiEnvironment(setup)

Running it on Mac:

(napari-imagej-dev) gselzer@dyn-144-92-48-223 imagej % python -i tmp2.py
WARNING: no screens available, assuming 24-bit color
Exception in thread "Thread-1" org.jpype.PyExceptionProxy
[java.lang.Enum.toString] Exception in thread "Thread-1" [java.lang.Enum.toString] org.jpype.PyExceptionProxy[java.lang.Enum.toString] 
	at org.jpype.JPypeContext.createException(Unknown Source)
[java.lang.Enum.toString] 	at org.jpype.JPypeContext.createException(Unknown Source)[java.lang.Enum.toString] 
	at org.jpype.proxy.JPypeProxy.hostInvoke(Native Method)
[java.lang.Enum.toString] 	at org.jpype.proxy.JPypeProxy.hostInvoke(Native Method)[java.lang.Enum.toString] 
	at org.jpype.proxy.JPypeProxy.invoke(Unknown Source)
[java.lang.Enum.toString] 	at org.jpype.proxy.JPypeProxy.invoke(Unknown Source)[java.lang.Enum.toString] 
	at com.sun.proxy.$Proxy7.run(Unknown Source)
[java.lang.Enum.toString] 	at com.sun.proxy.$Proxy7.run(Unknown Source)[java.lang.Enum.toString] 
	at java.base/java.lang.Thread.run(Thread.java:834)
[java.lang.Enum.toString] 	at java.base/java.lang.Thread.run(Thread.java:834)[java.lang.Enum.toString]

It then hangs, without a napari popup. I have to close it forcefully, with python warning me about a leaked semaphore 😦.

We will need additional work to avoid the hanging issues on Mac.

Running tmp2.py on my linux machine poses no problems.

[ctrueden]

OK, so the gist is that setupGuiEnvironment makes no difference on the napari/vispy/qt side regarding the bug being triggered. Good to know.

[gselzer]

It doesn't seem to, no.

Part of me wonders if this is something that we could just fix by reaching out to the vispy team. That's the source of the DivideByZero error...

[ctrueden]

Part of me wonders if this is something that we could just fix by reaching out to the vispy team.

Seems unlikely. Of course you can reach out to them, but the true problem is why the screen size is zero: because some graphical subsystem kicks in to headless mode due to some constraint under the hood, triggered by something that happens on the Java side. So even if the vispy team changed their code, I expect that actually showing a napari window would not be possible under these circumstances—it could just be made to fail more gracefully, which is ultimately not what we want (although not a bad thing).

[gselzer]

because some graphical subsystem kicks in to headless mode due to some constraint under the hood, triggered by something that happens on the Java side. So even if the vispy team changed their code, I expect that actually showing a napari window would not be possible under these circumstances—it could just be made to fail more gracefully, which is ultimately not what we want (although not a bad thing).

Yeah, at the end of the day I agree.

I'd also add that multiprocessing might give us some more options with regards to the hanging, and it looks like some shared memory tools were added to the standard library as of 3.8, but I don't think this will fix our bugs with the screen size change 😦

[ctrueden]

Sure, you could test the new interprocess multiprocessing capabilities. Just be sure to test on Windows also (assuming we use a unified cross-OS approach), because @hinerm ran into lots of squid with them. But for this purpose it may work well.

[gselzer]

I'd be happy to test this, but I think it might be best to hold off until the current PR chain (#4, #5, #8, #11, #12, #13, #15) is merged.

I think it is best to prioritize a release of this plugin, even if it is only for Windows and Linux. I don't want the work of those PRs to go stale as this issue is quickly turning into a deep dive 🤿

Do you agree?

[ctrueden]

Sure, that sounds fine. Are you expecting me to review all those PRs? I don't really have the time. Can we just merge and release a 0.1 and then get feedback from the napari devs?

[gselzer]

Are you expecting me to review all those PRs?

Nope, I'm planning to do it all myself, as I am well aware of your time constraints.

I don't really have the time.

Of course if you manage find the time, your review is welcome. But I think that your review time is better used elsewhere.

Can we just merge and release a 0.1 and then get feedback from the napari devs?

I'm not sure that the napari devs have the time either, but yes, more eyes on the project is really my motivation behind prioritizing a release.

[ctrueden]

On Macs in practice, people will be using napari-imagej with a display, and in that scenario it does work (with the warning [JRSAppKitAWT markAppIsDaemon]: Process manager already initialized: can't fully enable headless mode.) as long as you start napari first, so... this issue is really only about getting the CI working, right?

This issue comment is interesting:

export AWT_FORCE_HEADFUL=true

Might work to make it pass? (Originally I was thinking we'd need XQuartz and xvfb but maybe not, actually!)

[gselzer]

(@ctrueden also suggests running brew install xquartz and then running xquartz before running the tests on CI)

[ctrueden]

Yeah, to be clear: on my macOS laptop, if I have XQuartz installed and running when I run python -m pytest -p no:faulthandler, then all the napari-imagej tests pass. Otherwise, without XQuartz running, they fail like this:

$ python -m pytest -p no:faulthandler
process exited early. stderr:b"_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.\n_XSERVTransSocketUNIXCreateListener: mkdir(/tmp/.X11-unix) failed, errno = 2\n_XSERVTransMakeAllCOTSServerListeners: failed to create listener for local\n(EE) \nFatal server error:\n(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) \n"
start failed 1

[gselzer]

Closed thanks to #41