Preserve original data anisotropy?

[dpshepherd]

Hi all,

We've got this working fairly well locally. Still struggling with our school Hadoop cluster, which I think is a config issue.

A lot of our data has anisotropic xy pixel size vs z steps. What is the best way to get the BigStitcher-Spark affine fusion to act the same way as the "preserve original data anisotropy" setting in BigStitcher?

One thought I had was to edit the XML to change the calibration before fusion.

Thanks!

[StephanPreibisch]

Hi, that would work. But I also added it for the affine fusion right now with commit e8a66ed

Please let me know if it works. As usual it is self-explanatory from the command line parameters ;)

[dpshepherd]

Thanks!!

Running a fusion now and will update once it finishes. The log shows that the anisotropy factor was correctly calculated from the XML file.

[dpshepherd]

The fusion ran and the log says that preserve anisotropy is true and calculated the anisotropy correctly, but the resulting N5 has the same number of z planes when fused with or without the anisotropy flag activated,.

With --preserveAnisotropy = true, the N5 should have 2557 z planes. However, both true and false result in a fused file with 10644 z planes. The anisotropy factor is 1 um /.24 um = 4.16667. This matches 2557 * (1/.24) = 10644.

Attached is the xml.

c13m34_24_bdv.zip

[StephanPreibisch]

Hi, it should be fixed now ... could you please try again (sorry).

[dpshepherd]

Running with new changes now. Number of blocks to fuse is smaller in the log, which suggests it is working. Will update again once done.

Thanks!

[StephanPreibisch]

Fingers crossed! Do you also need non-rigid?

[dpshepherd]

Nope, rigid is good for our purposes.

[dpshepherd]

Correct number of z planes! But - BDV N5 viewer doesn't find a dataset anymore. I can load it in Python using Zarr/N5 tools, which is how I checked.

And muuuch faster now.

[StephanPreibisch]

When I tested it it worked like a charm ... how exactly are you opening it? The only difference is a different size, so not sure what could cause that?

[dpshepherd]

The previous fusion results previously opened for me just like a charm as well.

BigDataViewer -> N5 Viewer. When I click "detect datasets", it doesn't find the contents of the directory (should be Alexa647/s0). Interestingly, I can open the dataset in Python using Zarr with the N5 backend just fine.

[StephanPreibisch]

Can you please also paste a screenshot of the actual directory? Ideally with permissions?

[StephanPreibisch]

How does File > Import N5 behave?

[StephanPreibisch]

usually I have to go into the Alexa647 directory and then press "detect datasets"

[dpshepherd]

Can you please also paste a screenshot of the actual directory? Ideally with permissions?

(base) dps@qi2labserver:/mnt/opm2/20220120/n5$ tree -L 3 -p output.n5
output.n5
├── [drwxrwxrwx]  Alexa647
│   └── [drwxrwxrwx]  s0
│       ├── [drwxrwxrwx]  0
│       ├── [drwxrwxrwx]  1
│       ├── [drwxrwxrwx]  10
│       ├── [drwxrwxrwx]  2
│       ├── [drwxrwxrwx]  3
│       ├── [drwxrwxrwx]  4
│       ├── [drwxrwxrwx]  5
│       ├── [drwxrwxrwx]  6
│       ├── [drwxrwxrwx]  7
│       ├── [drwxrwxrwx]  8
│       ├── [drwxrwxrwx]  9
│       └── [-rwxrwxrwx]  attributes.json
└── [-rwxrwxrwx]  attributes.json

How does File > Import N5 behave?

Same behavior, no datasets are detected.

usually I have to go into the Alexa647 directory and then press "detect datasets"

Same behavior, no datasets are detected.

[dpshepherd]

I was able to load the dataset in Python using the Zarr N5 API, run our image processing code across all of the blocks, and save the output as a ome.tiff using tifffile.

[StephanPreibisch]

could you please also post the contents of both attributes.json? I just want to understand what is going wrong ...

[dpshepherd]

(base) dps@qi2labserver:/mnt/opm2/20220120/n5/output.n5$ more attributes.json
{"n5":"2.2.1"}

[StephanPreibisch]

and the one in s0?

[dpshepherd]

(base) dps@qi2labserver:/mnt/opm2/20220120/n5/output.n5/Alexa647/s0$ more attributes.json
{"min":[-197,-34119,1579],"compression":{"type":"gzip","useZlib":false,"level":1},"blockSize":[512,512,512],"dataType":"uint16","dimensions":[5479,3665,2557]}

[StephanPreibisch]

this is all correct, that is so weird ...

[dpshepherd]

Let me try a fresh Fiji install.

[StephanPreibisch]

can you send me everything except the directories in s0?

[dpshepherd]

Yes, give me a couple hours. I need to help the group get their posters finished for BPS meeting!

[StephanPreibisch]

good luck :)

[StephanPreibisch]

So I re-created the directory structure with the two attributes.json and it works fine for me ... what is different for you?

I went here "/Users/spreibi/tmp/output.n5/Alexa647/s0" and pressed "detect datasets"

[dpshepherd]

What version of the n5-viewer.jar do you have? I just downloaded a fresh Fiji, updated it, and it is n5-viewer_fiji-4.5.0.jar. On this one, there are more windows for metadata types.

If I don't update Fiji, I can detect a dataset but then get an error

Here is the directory structure without the data directories.
output_nos0.n5.zip

[tpietzsch]

Is this still a problem or can we close this issue? Or rather split it, because the original "preserve anisotropy" issue seems to be fixed?
(I'm just reading through the BigStitcher-Spark issues at the moment to get a sense of the type of problems...)

[dpshepherd]

From our end, the Fiji loading issue is still open. Splitting makes sense to me, as the data is fused with the correct anisotropy and accessible (for us) via Python.

[StephanPreibisch]

This commit should fix it: 6c21396 -- sorry that it took a while, really annoying side-effect of not naming attributes consistently