Does this run headlessly?

[ctr26]

Can you pass it images to process from a ijm script for instance?

[ctr26]

Yes it does,

IJ.run(imp, "DeepImageJ Run", "model=[Noise2Void Denoising] preprocessing=[no preprocessing] postprocessing=[no postprocessing] patch=768 overlap=22 logging=normal");

[ctr26]

Actually, if you do try to run this headlessly using you hit a headless exception, java.awt.HeadlessException. Could this be fixed to run in headless mode?

[carlosuc3m]

Which version of the plugin are you using? Are you just running that command or are you doing something else?
I have just tried it and it works with the last version (https://github.com/deepimagej/deepimagej-plugin/releases/tag/1.1.0).

[ctr26]

Version 1.0.1. Update: Same effect with 1.1.0

I'm running that command in a jypthon script whilst using
fiji --console --headless
in my console

script.py:

from ij import IJ, ImagePlus, ImageStack
print("hello")


blobs = IJ.openImage("https://imagej.net/images/blobs.gif")
imp = blobs.createImagePlus()
IJ.run(imp, "DeepImageJ Run",
            "model=[Noise2Void Denoising] preprocessing=[no preprocessing] postprocessing=[no postprocessing] patch=768 overlap=22 logging=normal");

Console:

/opt/fiji/Fiji.app/ImageJ-linux64 --console --headless --run /mnt/script.py

out:

hello
java.awt.HeadlessException

[carlosuc3m]

I see.
The problem here is not related with the headless mode, but with the size of the patch with respect to the image.
In order for the plugin to work the patch size has to be strictly smaller than 3 times the smaller dimension between x and y.
If you change the patch size from 768 to 512 for example, it should work in both versions.

[ctr26]

I tried this but it's still complaining about a headless exception.

[carlosuc3m]

Does the plugin work when you try to run it from the desktop ImageJ application?

[ctr26]

This does

from ij import IJ, ImagePlus, ImageStack

imp = IJ.openImage("https://imagej.net/images/blobs.gif")
imp.show()

IJ.run(imp, "DeepImageJ Run",
            "model=[Noise2Void Denoising] preprocessing=[no preprocessing] postprocessing=[no postprocessing] patch=256 overlap=22 logging=normal");

This doesn't

from ij import IJ, ImagePlus, ImageStack

imp = IJ.openImage("https://imagej.net/images/blobs.gif")
#imp.show()

IJ.run(imp, "DeepImageJ Run",
            "model=[Noise2Void Denoising] preprocessing=[no preprocessing] postprocessing=[no postprocessing] patch=256 overlap=22 logging=normal");

And if you have imp.show() that automatically crashes FIJI in headless mode.:

File "script.py", line 7, in <module>
    imp.show()
        at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
        at java.awt.Window.<init>(Window.java:536)
        at java.awt.Frame.<init>(Frame.java:420)
        at ij.gui.ImageWindow.<init>(ImageWindow.java:68)
        at ij.gui.ImageWindow.<init>(ImageWindow.java:64)
        at ij.ImagePlus.show(ImagePlus.java:440)
        at ij.ImagePlus.show(ImagePlus.java:412)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)
java.awt.HeadlessException: java.awt.HeadlessException

[imagejan]

@ctr26 as an alternative, you could try running the N2V_Fiji plugin in your case. That plugin is based on CSBDeep and imagej-tensorflow and should not have issues with running headless, as it avoids using ImageJ1 structures. /cc @frauzufall @ctrueden

[ctr26]

What I'm trying to do is setup up a decent docker image of deepimagej because I'd like to be able to run lots of models on the varied data on our cluster. Having a docker image of deepimagej would make it really powerful at scale.BestCraigOn 8 Mar 2020 19:33, Jan Eglinger <notifications@github.com> wrote:@ctr26 as an alternative, you could try running the N2V_Fiji plugin in your case. That plugin is based on CSBDeep and imagej-tensorflow and should not have issues with running headless, as it avoids using ImageJ1 structures. /cc @frauzufall @ctrueden —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

[carlosuc3m]

Thank you for the detailed explanation @ctr26 .
The error is caused because deepImageJ requires an image to be open in ImageJ. I will try to fix this to allow the plugin run headlessly and get back to you when it is fixed.

[frauzufall]

@ctr26

For CARE and N2V trained in python, the recommanded way of running prediction on the cluster is using python directly. This script will do exactly what you asked for, including proper normalization.

How are you doing normalization? The DeepImageJ Noise2Void model preprocessing normalization looks weird, @tibuch @alex-krull is this even correct? @carlosuc3m do you tell people somewhere how this is adjusted for other Noise2Void models? Won't this lead to wrong results?

If you don't use the pre/postprocessing and do normalization yourself, the recommended way for running N2V prediction in Java is CSBDeep.

Have a look at this N2V training notebook, at the end there is this line:

model.export_TF()

.. which exports the model into a ZIP which you can directly plug into CSBDeep to run prediction in Java. It's just a zipped SavedModelBundle, CSBDeep can also run other image-to-image networks.

Here is a script which runs prediction on a whole folder of images: https://github.com/CSBDeep/CSBDeep_fiji/blob/master/script/CARE_generic.py It runs headless. If this does not work, please file an issue here.

We are working on making the N2V python export models also compatible with N2V Fiji so that you can run prediction with proper normalization in Java using N2V for Fiji.

[carlosuc3m]

@ctr26 The issue you commented should already fixed. Please feel free to try the new version and let me know if anything is still wrong with it. Here is the link to the correct version of the plugin https://github.com/deepimagej/deepimagej-plugin/releases/download/1.1.0/DeepImageJ_-1.1.0-SNAPSHOT.jar

[tibuch]

Hello @carlosuc3m,

I just looked into the DeepImageJ Noise2Void model linked by @frauzufall. How did you come up with the preprocessing macro? In Noise2Void we compute mean and standard deviation over all given input training data and for each channel independently. These values are then stored in the config.json file. During prediction we use these numbers to normalize each input image ( (img - mean)/std) ) and after prediction we invert this operation ( (img * std + mean) ).

Are you planning to use the modelzoo descriptions in the future as well?

[esgomezm]

Hi @tibuch and @frauzufall !

Exactly! We took the information from the config.json file (in our case it looks like: {"mean": "0.18905598", "std": "0.18313955", "n_dim": 2, "axes": "YXC", ....}). Then, we reproduced the normalization performed in code of N2V.

We are migrating to the modelzoo description but it is true that there are no specific fields for this kind of parameter. It could be a new proposal for the .yaml file.

On the other hand, we designed the pre and post-processing macros to deal with this kind of problem that is straightforward for the developer.

[tibuch]

Hi @esgomezm,

I see, but I don't understand why you normalize to [0,1] before you normalize with mean and standard deviation. And it is also not immediately clear that these values are supposed to be mean and standard deviation.

// Preprocessing macro
print("Preprocessing");
run("32-bit");
getMinAndMax(min, max);
run("Subtract...", "value=&min");
cocient = max-min;
run("Divide...", "value=&cocient");
run("Subtract...", "value=0.18905598");
run("Divide...", "value=0.18313955");

Additionally this model should only be applied to data which has similar structures and the same noise statistics as the data used for training. Which is impossible to determine with your current format. @ctr26 please make sure that the model you are using is trained on appropriate data!

We address this issue with the modelzoo format, where the author of the yaml file has to provide a link to the used training data. And additionally a test_input and test_output image is provided.

Furthermore the modelzoo yaml files is meant to be language agnostic. The big advantage of a language agnostic model description is we are not bound to a single programming language. Which means that everyone can contribute models and everyone can make use of these models.

This is the current state of the yaml file as it gets exported by the fiji/n2v plugin. :

name: YOUR MODEL NAME HERE
description: YOUR DESCRIPTION HERE
cite:
  text: |-
    Buchholz, T. et al. - Content-aware image restoration for electron microscopy. 
    Methods in Cell Biology, Volume 152 p.277-289, ISSN 0091-679X (2019)
  doi: https://doi.org/10.1016/bs.mch.2019.05.001
authors: [YOUR NAMES HERE]
documentation: README.md
test_input: ./test_input.tif
test_output: ./test_output.tif
covers: [./thumbnail.png]
tags: [denoising, unet2d, n2v]
license: BSD 3
format_version: 0.1.0
language: java
framework: tensorflow
source: de.csbdresden.n2v.train.N2VPrediction
inputs:
- name: raw
  axes: byxc
  data_type: float32
  data_range: [-inf, inf]
  shape:
    min: [1, 4, 4, 1]
    step: [1, 4, 4, 0]
outputs:
- name: denoised
  axes: byxc
  data_type: float32
  data_range: [-inf, inf]
  halo: [0, 32, 32, 0]
  shape:
    reference_input: raw
    scale: [1, 1, 1, 1]
    offset: [0, 0, 0, 0]
training:
  setup:
    reader:
      spec: MISSING IMPLEMENTATION
      kwargs: {uri: MISSING LINK TO PUBLICLY AVAILABLE DATA}
  source: de.csbdresden.n2v.train.N2VTraining
  kwargs: {batchDimLength: 180, batchSize: 64, trainDimensions: 2, neighborhoodRadius: 5, numEpochs: 100,
    numStepsPerEpoch: 300, patchDimLength: 60, stepsFinished: 30000}
prediction:
  preprocess:
    spec: de.csbdresden.n2v.predict.N2VPrediction::preprocess
    kwargs: {mean: 41498.87, stdDev: 15007.021}
  weights: {source: https://github.com/bioimage-io/fiji-bioimage-io/releases/download/v0.1.0/n2v-sem-demo.zip}
  postprocess:
    spec: de.csbdresden.n2v.predict.N2VPrediction::postprocess
    kwargs: {mean: 41498.87, stdDev: 15007.021}
  dependencies: ./dependencies.yaml

As you can see mean and standard deviation are passed as kwargs: {mean: 41498.87, stdDev: 15007.021}. And here she links to java implementations which live in de.csbdresden.n2v and these dependencies are further described in the dependencies.yaml which is just a .pom for maven-plugins.

And you can also see that we are not there yet. This model is missing some crucial information for reproducability as well as save application. author, description and most importantly a link to the publicly available train-data is missing!

I thought that we want to converge to the modelzoo format. For us java users this means that we have to write some additional code which can parse yaml files and look up the correct methods. @frauzufall has already done a lot of work in this direction. But we also need a plugin which asks the user after training to provided the necessary information to write up a proper yaml file. This includes the name of the user (author), a description and the link to the training data. Only with this information the model should be provided to the modelzoo. As far as I can tell you already have some code in this direction where you ask the user for some extra information.

I think it would be fantastic if we could combine our efforts in bringing deep learning to ImageJ. @frauzufall has a lot of experience with the ImageJ2, imagej-tensorflow and CSBDeep as core execution engine of TensorFlow models in Java and you have a nice user interface.

[ctr26]

Hi @esgomezm,

I see, but I don't understand why you normalize to [0,1] before you normalize with mean and standard deviation. And it is also not immediately clear that these values are supposed to be mean and standard deviation.

// Preprocessing macro
print("Preprocessing");
run("32-bit");
getMinAndMax(min, max);
run("Subtract...", "value=&min");
cocient = max-min;
run("Divide...", "value=&cocient");
run("Subtract...", "value=0.18905598");
run("Divide...", "value=0.18313955");

Additionally this model should only be applied to data which has similar structures and the same noise statistics as the data used for training. Which is impossible to determine with your current format. @ctr26 please make sure that the model you are using is trained on appropriate data!

We address this issue with the modelzoo format, where the author of the yaml file has to provide a link to the used training data. And additionally a test_input and test_output image is provided.

Furthermore the modelzoo yaml files is meant to be language agnostic. The big advantage of a language agnostic model description is we are not bound to a single programming language. Which means that everyone can contribute models and everyone can make use of these models.

This is the current state of the yaml file as it gets exported by the fiji/n2v plugin. :

name: YOUR MODEL NAME HERE
description: YOUR DESCRIPTION HERE
cite:
  text: |-
    Buchholz, T. et al. - Content-aware image restoration for electron microscopy. 
    Methods in Cell Biology, Volume 152 p.277-289, ISSN 0091-679X (2019)
  doi: https://doi.org/10.1016/bs.mch.2019.05.001
authors: [YOUR NAMES HERE]
documentation: README.md
test_input: ./test_input.tif
test_output: ./test_output.tif
covers: [./thumbnail.png]
tags: [denoising, unet2d, n2v]
license: BSD 3
format_version: 0.1.0
language: java
framework: tensorflow
source: de.csbdresden.n2v.train.N2VPrediction
inputs:
- name: raw
  axes: byxc
  data_type: float32
  data_range: [-inf, inf]
  shape:
    min: [1, 4, 4, 1]
    step: [1, 4, 4, 0]
outputs:
- name: denoised
  axes: byxc
  data_type: float32
  data_range: [-inf, inf]
  halo: [0, 32, 32, 0]
  shape:
    reference_input: raw
    scale: [1, 1, 1, 1]
    offset: [0, 0, 0, 0]
training:
  setup:
    reader:
      spec: MISSING IMPLEMENTATION
      kwargs: {uri: MISSING LINK TO PUBLICLY AVAILABLE DATA}
  source: de.csbdresden.n2v.train.N2VTraining
  kwargs: {batchDimLength: 180, batchSize: 64, trainDimensions: 2, neighborhoodRadius: 5, numEpochs: 100,
    numStepsPerEpoch: 300, patchDimLength: 60, stepsFinished: 30000}
prediction:
  preprocess:
    spec: de.csbdresden.n2v.predict.N2VPrediction::preprocess
    kwargs: {mean: 41498.87, stdDev: 15007.021}
  weights: {source: https://github.com/bioimage-io/fiji-bioimage-io/releases/download/v0.1.0/n2v-sem-demo.zip}
  postprocess:
    spec: de.csbdresden.n2v.predict.N2VPrediction::postprocess
    kwargs: {mean: 41498.87, stdDev: 15007.021}
  dependencies: ./dependencies.yaml

As you can see mean and standard deviation are passed as kwargs: {mean: 41498.87, stdDev: 15007.021}. And here she links to java implementations which live in de.csbdresden.n2v and these dependencies are further described in the dependencies.yaml which is just a .pom for maven-plugins.

And you can also see that we are not there yet. This model is missing some crucial information for reproducability as well as save application. author, description and most importantly a link to the publicly available train-data is missing!

I thought that we want to converge to the modelzoo format. For us java users this means that we have to write some additional code which can parse yaml files and look up the correct methods. @frauzufall has already done a lot of work in this direction. But we also need a plugin which asks the user after training to provided the necessary information to write up a proper yaml file. This includes the name of the user (author), a description and the link to the training data. Only with this information the model should be provided to the modelzoo. As far as I can tell you already have some code in this direction where you ask the user for some extra information.

I think it would be fantastic if we could combine our efforts in bringing deep learning to ImageJ. @frauzufall has a lot of experience with the ImageJ2, imagej-tensorflow and CSBDeep as core execution engine of TensorFlow models in Java and you have a nice user interface.

To be honest I was surprised that it's possible to run N2V using a pre-trained model as my assumption was that a) it wasn't general in that way and b) you had to train a new N2V model for every new noisy image (or maybe every new noisy environment).

A standardised cross platform model-format in a centralised repository would be very useful for this project, not least because these models are stored on a googledrive and that's hard to access using docker files.

[esgomezm]

Hi @tibuch,

I see, but I don't understand why you normalize to [0,1] before you normalize with mean and standard deviation. And it is also not immediately clear that these values are supposed to be mean and standard deviation.

Additionally this model should only be applied to data which has similar structures and the same noise statistics as the data used for training. Which is impossible to determine with your current format. @ctr26 please make sure that the model you are using is trained on appropriate data!

This is because the training image was normalized between 0 and 1 before training. The image used for the training and the given example image is the same one in this case. However, as @ctr26 said, this specific model is not provided to process new data, but as an example of models that can be loaded in Fiji using DeepImageJ.

Furthermore the modelzoo yaml files is meant to be language agnostic. The big advantage of a language agnostic model description is we are not bound to a single programming language. Which means that everyone can contribute models and everyone can make use of these models.

And you can also see that we are not there yet. This model is missing some crucial information for reproducability as well as save application. author, description and most importantly a link to the publicly available train-data is missing!

I thought that we want to converge to the modelzoo format. For us java users this means that we have to write some additional code which can parse yaml files and look up the correct methods. @frauzufall has already done a lot of work in this direction. But we also need a plugin which asks the user after training to provided the necessary information to write up a proper yaml file. This includes the name of the user (author), a description and the link to the training data. Only with this information the model should be provided to the modelzoo. As far as I can tell you already have some code in this direction where you ask the user for some extra information.

Yes, we developed the config.xml file precisely for this purpose and it partially contains this kind of information. Actually, DeepImageJ_Build_BundledModel was created to write the metadata of the model ( config.xml / YAML) and ensure the correct processing in Fiji.

I think it would be fantastic if we could combine our efforts in bringing deep learning to ImageJ. @frauzufall has a lot of experience with the ImageJ2, imagej-tensorflow and CSBDeep as core execution engine of TensorFlow models in Java and you have a nice user interface.

Definitely! Let's keep discussing it in imagej-modelzoo issue #1

[frauzufall]

However, as @ctr26 said, this specific model is not provided to process new data, but as an example of models that can be loaded in Fiji using DeepImageJ.

This is not what he said, he assumed, since the model was on DeepImageJ, that he can run this on his data on the cluster, and wondered how this works since the authors of N2V say you have to train yourself. The existence of the model of DeepImageJ made him think otherwise (@ctr26 correct me if I'm wrong), and no one from your side told him he should not run this on his data when he posted this issue. Don't you think the model should clearly indicate what you said about it on the website and in Fiji?

[esgomezm]

In general, when using any of the published models to process new data, the user should check the obtained results and verify that they are valid. Same as when you use any other plugin in ImageJ, users are expected to read the information provided by the authors of each model. I think this is general for any bioimage workflow and not only for deep learning models.
In the case of N2V, the model should be trained because the authors recommend so.

We have added a specific note for users in this direction that I hope it serves to avoid this kind of confusion.