Zarr3 download for volume annotation, supporting nd #7288

fm3 · 2023-08-17T13:13:07Z

URL of deployed dev instance (used for testing):

https://ndvolumedownload.webknossos.xyz

Steps to test:

Create annotation on 3d dataset, see download modal, try different options
Create annotation on nd dataset, see download modal, try different options
Download from other places like dashboard should still use wkw for the moment (untill the libs support zarr)
Reupload of a zarr volume annotation should yield the same annotation
Zarr metadata and datasource-properties.json in data.zip should look right.
Unpacking a zarr data.zip and moving it to the a folder in binaryData should yield a usable webknossos dataset (this does not yet work in nd case because only specific axis orders are supported with additional coordinates)

TODOs:

Issues:

fixes Support download+upload of ND volume annotations #7273
contributes to Support all valid values for transpose codec configuration in zarr v3 #7116

Updated changelog
Considered common edge cases
Needs datastore update after deployment

…sions

… additional coordinates as hardcoded ones

…ry ND data

…dimension

…o array of objects (analoguous to backend)

…e cursor is at

MichaelBuessemeyer · 2023-10-11T11:50:55Z

I took care of the frontend part of this pr:
The download modal now gives the user the option to choose between wkw and zarr for annotations with volume data. The UI looks like this in the current version:

In case one of the volume layers is nd, wkw is disabled as a download format and only zarr is available.

The other cases where annotations are downloaded (like tasks or projects), nd datasets are not supported for download right now.

- enforce annotation download in zarr when having nd datasets

philippotto · 2023-10-11T16:02:15Z

frontend/javascripts/admin/admin_rest_api.ts

@@ -1035,21 +1035,28 @@ export async function downloadAnnotation(
  annotationType: APIAnnotationType,
  showVolumeFallbackDownloadWarning: boolean = false,
  versions: Versions = {},
+  volumeDataZipFormat: string = "wkw",


the type could be "zarr3" | "wkw" instead of string, I think.

philippotto · 2023-10-11T16:05:21Z

frontend/javascripts/oxalis/view/action-bar/download_modal_view.tsx

@@ -268,10 +278,13 @@ function _DownloadModalView({
    (state: OxalisState) => state.temporaryConfiguration.isMergerModeEnabled,
  );
  const hasVolumeFallback = tracing.volumes.some((volume) => volume.fallbackLayer != null);
+  const isVolumeNDimensional = tracing.volumes.some((tracing) => tracing.additionalAxes.length > 0);
+  const hasVolumes = hasVolumeTracings(tracing);
+  const initialFilesToDownload = hasVolumes ? (isVolumeNDimensional ? "zarr" : "wkw") : "nml";


I'd rename initialFilesToDownload and filesToDownload to [initial]fileFormatToDownload. otherwise, it sounds like the vars would contain arrays.

philippotto · 2023-10-11T16:08:30Z

frontend/javascripts/oxalis/view/action-bar/download_modal_view.tsx

@@ -547,46 +555,42 @@ function _DownloadModalView({
                }}
              >
                Select the data you would like to download.
+                <Hint style={{ marginTop: 12 }}>
+                  An NML file will always be included with any download.


will the NML always contain the skeletons? if not, then it's not possible to download volume + skeleton at the same time? and if yes, then it's not possible to download volume without skeletons? either way, it seems as this gives less control to the user? is this intended?

Indeed, skeletons are always included. This was already the case before this PR. We might want to change that in the future, but that’s unrelated to this change.

Ah, I see, then never mind :)

app/controllers/AnnotationIOController.scala

...tore/app/com/scalableminds/webknossos/tracingstore/tracings/volume/VolumeDataZipHelper.scala

philippotto and others added 30 commits June 12, 2023 14:50

add UI for 4th dimension and mock 4D data when requesting buckets

4f2be0e

adapt bucket address space to incorporate higher dimension

f80301b

clean up imports

3c8322a

add slider to Q input field; tune mocked 4D data retrieval

a24fb58

change max value of number slider

a38ece5

Merge branch 'master' of github.com:scalableminds/webknossos into 4d

e19dd4d

make skeletons compatible with higher dimensions

168d58e

fix tests; improve typing; create CubeEntry instances lazily

8dd6c5f

fix linting

082af8b

use n-dimensional generalization instead of fourth coordinate everywhere

7b30efe

fix resampling of buckets when annotating volume data in higher dimen…

6cc4372

…sions

add additionalCoordinates parameter to several other places

5f11460

hardcode additionalCoords initialization

275c7af

tackle more todo comments

c25d61d

adapt additionalCoordinates input UI to arbitrary dimensions; use two…

38e0592

… additional coordinates as hardcoded ones

dynamically generate shaders for nodes and edges to deal with arbitra…

b0b5b04

…ry ND data

make edges interpolate from alpha 1 to 0 if edge partner is on other …

1f05545

…dimension

Add additional coordinates for 4+D requests

6b7b3a5

Merge branch 'timeseries' into 4d

ff3e37b

begin integration of backend additionalCoordinate interface

54a4bf1

change frontend format for additional coordinates from number-array t…

48b1605

…o array of objects (analoguous to backend)

Merge branch 'master' into 4d

bfe839c

Add additional coords to db

98c1e15

Fix naming

4c84670

Merge branch 'master' of github.com:scalableminds/webknossos into 4d

ce64fa9

remove hardcoded additional coordinates related to DataCube class

dad1652

properly initialize flycam with additional coordinates

e5047ce

make that arrow keys in vector input increment/decrement element wher…

2b819b0

…e cursor is at

fix onChange in additional-coordinate-input

8f2eb14

use bounds of additional unified coordinates in slider

a63da9e

fm3 added 2 commits October 5, 2023 13:09

cleanup

71c330e

fix chunk path regex

b1d1989

fm3 mentioned this pull request Oct 11, 2023

Support Zarr3 Volume Annotation format scalableminds/webknossos-libs#951

Closed

Add option to select volume download format to annotation download modal

568616b

- enforce annotation download in zarr when having nd datasets

fm3 changed the title ~~WIP: zarr3 download for volume annotation, supporting nd~~ Zarr3 download for volume annotation, supporting nd Oct 11, 2023

fm3 added 2 commits October 11, 2023 14:01

Merge branch 'master' into nd-volume-download

1eda531

changelog

84556cc

fm3 marked this pull request as ready for review October 11, 2023 12:03

fm3 requested review from frcroth and daniel-wer October 11, 2023 12:03

fm3 added 2 commits October 11, 2023 14:23

format

601231a

WIP: include channel dimension

7f14760

fm3 removed request for daniel-wer and frcroth October 11, 2023 13:12

fix channel path component

e09f9fd

philippotto reviewed Oct 11, 2023

View reviewed changes

fm3 added 2 commits October 12, 2023 13:41

re-add saveMultiple route (not unused after all)

650e72c

use array variant of transpose codec spec

770ed8f

fm3 mentioned this pull request Oct 12, 2023

Support all valid values for transpose codec configuration in zarr v3 #7116

Open

implement frontend pr feedback

3f00055

fm3 requested review from philippotto and frcroth October 12, 2023 13:23

fm3 added 2 commits October 12, 2023 15:28

parameter name

14c39d7

lint

17ac787

frcroth approved these changes Oct 16, 2023

View reviewed changes

fm3 and others added 2 commits October 16, 2023 14:11

Merge branch 'master' into nd-volume-download

af34c47

Merge branch 'master' into nd-volume-download

835f7bc

fm3 merged commit ede79a0 into master Oct 17, 2023
2 checks passed

fm3 deleted the nd-volume-download branch October 17, 2023 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zarr3 download for volume annotation, supporting nd #7288

Zarr3 download for volume annotation, supporting nd #7288

fm3 commented Aug 17, 2023 •

edited

Loading

MichaelBuessemeyer commented Oct 11, 2023

philippotto Oct 11, 2023

philippotto Oct 11, 2023

philippotto Oct 11, 2023

fm3 Oct 11, 2023

philippotto Oct 12, 2023

Zarr3 download for volume annotation, supporting nd #7288

Zarr3 download for volume annotation, supporting nd #7288

Conversation

fm3 commented Aug 17, 2023 • edited Loading

URL of deployed dev instance (used for testing):

Steps to test:

TODOs:

Issues:

MichaelBuessemeyer commented Oct 11, 2023

philippotto Oct 11, 2023

Choose a reason for hiding this comment

philippotto Oct 11, 2023

Choose a reason for hiding this comment

philippotto Oct 11, 2023

Choose a reason for hiding this comment

fm3 Oct 11, 2023

Choose a reason for hiding this comment

philippotto Oct 12, 2023

Choose a reason for hiding this comment

fm3 commented Aug 17, 2023 •

edited

Loading