Improve dummy data when opening mastodon project with missing images (Issue 206)

[stefanhahmann]

Resolves #206

[stefanhahmann]

@maarzt this pull request is in draft mode for now. please test it.

[maarzt]

Hello Stefan,

It works exactly as we discussed it, that's great. I didn't had a look into the code for now. But I noticed that the message dialog that is shown to the user is a little bit confusing:

Maybe a dialog like this would be better:

• Title: Mastodon - Image data not found.

• Icon: Warning Message

• Text:
  Mastodon could not find the images associated with this project.
  This file was not found:
  /path/to/file/not/found/dataset.xml

  It is still possible to open the project. You won't be able to see the image data, but you can still view and modify the tracking data.

  Would you like to open the project without image data?

• Buttons: [Open Project] [Cancel]

[stefanhahmann]

Good idea. Applied suggested changes in a further commit.

[stefanhahmann]

Visualisation of new behaviour:

• 
• 

[maarzt]

I very much like how well this works. I think it's a big improvement in terms of user experience.

There are still some things that I would like to improve. The code for opening a mastodon project is already quite complex, and would like to avoid increasing the complexity. Could you do the requested changes. I would afterwards continue, try to refactor some stuff.

[maarzt]

@stefanhahmann Please write a few sentences, how opening the spim data XML works, if the HDF5 data is missing.
I only understand parts of this. Like that the ImgLoader part has the removed from the XML, and there are some null values that need to be replaced later on. I think it would be useful having this documented in the issue or somewhere.

[tinevez]

Hello @stefanhahmann @maarzt

I am only catching up now. I think that this PR may be duplicating work already done, by @xulman

The beta-26 can deal with missing BDV data but improperly. For instance the number of time-points is not detected correctly and the dummy data implemented with Vlado seems not to work in full (you need to resave the file for the dummy to work).
I have a gigantic dataset that starts at frame 400, but I cannot simply display the tracks nor TrackScheme if I do not have the BDV.

Maybe you want to test Vlado solution?

In Fiji, open Mastodon making sure we use the beta-26. We need the Fiji version, because it has the mastodon-tomancak artifact that contains the file path editor of Vlado.

Open a Mastodon file that points to a missing XML BDV file. Mastodon will warn that it cannot find the associated bdv file.
Go to the File > Fix image path menu item. There is a "I want dummy data instead", then "from spots". This will fix the issue, but only if you save and reload.

Please tell us what you think.

[maarzt]

Hello @tinevez,

We are well aware of the work done so far be @xulman. This PR continues where this previous work stopped.

The goal is to simplify the user interaction. It is no longer required to re-save the Mastodon project and users don't need to enter information like pixel-size etc.

I would also like to move the code in mastodon-tomancak over to the main repository. As it would make the mastodon git repository more self contained.

The PR might fix the problems with the dataset that starts at frame 400. Could you maybe give this a try?

Bests, Matthias

[tinevez]

Yes ok!
When the PR is ready we will ask Vlado if he is ok with transitioning his code with yours to the core, if that's ok.

[stefanhahmann]

@stefanhahmann Please write a few sentences, how opening the spim data XML works, if the HDF5 data is missing. I only understand parts of this. Like that the ImgLoader part has the removed from the XML, and there are some null values that need to be replaced later on. I think it would be useful having this documented in the issue or somewhere.

As discussed: No changes on the spim data XML are necessary (initially, I thought, they would be necessary, but in fact it turned out that they weren't). The process works now more or less as described in the acceptance criteria of #206

[stefanhahmann]

I made some commits to consider your review comments.
Could you please review again?

[maarzt]

Hi @tinevez,

Let me summarize this PR for you, such that you have an easier time reviewing it. The goal of the PR is to streamline opening a Mastodon project, when the image data is missing. The changes will make it a lot easier to open a Mastodon project that one just got send via email, where image data is missing as usual...

The current version of Mastodon, can be started if the image data is missing. But it only starts with partly functional dummy data. In this state, only the first ten time-points of the trackscheme are visible and a new BDV window is zoomed suboptimally, since the image size incorrect. The user needs to click "File > Fix Image Path", resave and restart Mastodon in order to conveniently inspect the data.

This PR improves this situation very much. The main idea is to save a copy of the spim data XML within the Mastodon project. This backup file is called "dataset.xml.backup". If now a Mastodon project is opened and the primary image data is missing, Mastodon can read the "dataset.xml.backup" file to initialize dummy image data with the correct number of time points, image sizes and even affine image transformations. The TrackScheme and BDV windows work as expected with the now correct dummy image data.

The PR also improves the dialog that tells the user about the problem loading the image data. The dialog now tells the user what went wrong. And there is also a convenient "cancel" button.

Here is an overview of the changes made in the code:

• New method DummySpimData.fromSpimDataXml() allows to create dummy data with correct metadata from a spim data XML. This works as follows:
  • Create a temporary copy of the spim data XML, with the <ImageLoader> entry removed
  • Open this modified XML. The result is a SpimData object with imgLoader not set.
  • Finally imgLoader is set to use the DummyImgLoader.
  • Et voila: we got a SpimData object with all metadata set correctly, buy no actual image pixel data required or loaded.
• ProjectManager.openImageData(...) and SharedBigDataViewerData.fromSpimDataXml(...) have been refactored.
• The dialog telling the user about problems loading image data is now in the new method ProjectManager.openWithDialog(...).
  This still allows to open a project without dialog using ProjectManager.open(...), which is useful for batch processing.
• MamutProjectIO is now responsibly for correctly loading Windows file paths with backslashes on Linux
• Newly added tests:
  • ProjectManagerTest: Test if the dataset.xml.backup is saved correctly.
  • StartMastodonWithMissingImageDemo: Demonstrate how Mastodon starts if image data is missing.
  • MamutProjectIOTest: Make sure Mastodon can deal with Windows and Linux file paths.
  • DummySpimDataTest & DummyImgLoaderTest

Bests,
Matthias

[tinevez]

Hello all. High quality code!
Indeed, this new version does handling missing data really well and I would say, the proper way. The idea of creating a temp file is very good and very efficient. I tested the feature with image data hosted remotely and it works as expected.
Next step would probably be merging the tool to fix the image path from Vlado into the core.