Undeterministic segmentation on with different instances of model

[Czaki]

I try to create cellpose plugin to my tool https://github.com/4DNucleome/PartSeg_cellpose I found that In some scenario different instances of cellpose model returns different predictions. I try to prepare code which deterministic reproduce this, but not always next instance of model fail. I need to recreate it again in third cell to got this result.

But in my plug-in this happens always.

It is possible to fix it?

img02.png is from your example notebook.

[marius10p]

Sorry, I'm not sure I understand. There should not be any randomness in that function call (right @carsen-stringer ?).

[Czaki]

As I expect there should be no randomness. But I found that I got different result in notebook and in gui (my one). Then in test I found that I can reproduce this in notebook.

[Czaki]

Any information?

[carsen-stringer]

In the notebook you are running with diameter=None so it is estimating the diameter for you. In the GUI are you also using the "calibration" button to get the diameter?

I don't understand how it could be different in the same notebook with the same code, can you provide more information if you are still having this issue?

[Czaki]

I don't understand how it could be different in the same notebook with the same code, can you provide more information if you are still having this issue?

I also do not understand this. But I notice that problematic runs take much longer time (9 seconds against 90-160 seconds).
Now it happens time to time when using multiple instances of model from notebook. It does not depend if I set diameter to None or 0 or set positive value 30.

I do not have any idea how debug this. But maybe You have idea why wrong calculation in current version takes longer time?

[Czaki]

I check this on second computer (both ubuntu) and same problem (different segmentation form different instance of model for same file) shows.

[chrisroat]

I have seen similar non-determinism in some unit tests I have, which makes them flaky. It's often on just small, isolated groups of pixels near segment boundaries. From apache/mxnet#13829 (comment), it seems that mxnet autotuning can be non-deterministic.

Setting the following during my tests has fixed my problem:

MXNET_CUDNN_AUTOTUNE_DEFAULT=0

[carsen-stringer]

thanks @chrisroat , closing this for now, perhaps the torch version doesn't have this problem