Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segment group batch actions #7164

Merged
merged 51 commits into from
Jul 20, 2023
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
4e86f6d
start: change group color
dieknolle3333 Jun 20, 2023
b97c9d8
change group color
dieknolle3333 Jun 20, 2023
ea18ffd
[very broken] first implementation to hide/load all meshes of a segme…
dieknolle3333 Jun 20, 2023
98e458f
show/hide all msehes for group
dieknolle3333 Jun 23, 2023
071d0e1
add batch actions: download, show/hide, load meshes for segment group
dieknolle3333 Jun 25, 2023
d2d8861
fix code duplication and lint
dieknolle3333 Jun 26, 2023
e6ed640
add warning if position of segments in unknown
dieknolle3333 Jun 26, 2023
5472e53
push WIP before I leave
dieknolle3333 Jun 27, 2023
0a4c7d2
zip multiple meshes
dieknolle3333 Jun 27, 2023
31c3f87
undo zip.js upgrade
dieknolle3333 Jun 27, 2023
e829f1e
batch actions to set colour and remove and refresh meshes option
dieknolle3333 Jun 27, 2023
0f0f402
Merge branch 'master' into segment-group-batch-actions
dieknolle3333 Jun 27, 2023
c879a10
lint
dieknolle3333 Jun 27, 2023
e0440d3
remove TODOs
dieknolle3333 Jun 27, 2023
8cab227
add chagelog entry
dieknolle3333 Jun 27, 2023
ea8b1c9
refactor code
dieknolle3333 Jun 30, 2023
5944d85
WIP: start refactoring
dieknolle3333 Jun 30, 2023
8781956
WIP: add helper method
dieknolle3333 Jul 2, 2023
e809bca
WIP: address review
dieknolle3333 Jul 3, 2023
ca20df1
WIP: add lost method call
dieknolle3333 Jul 3, 2023
a4b89e2
WIP: trying to improve areSegmentsInGroupVisible
dieknolle3333 Jul 3, 2023
b5a2e99
WIP: fix getDerivedStateFromProps
dieknolle3333 Jul 4, 2023
9db1f1e
veryWIP: trying to fix tests by mocking zipjs
dieknolle3333 Jul 4, 2023
6c7bedd
WIP: trying to fix tests
dieknolle3333 Jul 7, 2023
cc866f5
WIP: fix small typo
dieknolle3333 Jul 7, 2023
c8ca503
fix merge conflicts
dieknolle3333 Jul 7, 2023
180bddb
WIP: trying to fix tests again
dieknolle3333 Jul 10, 2023
b9a11ed
WIP: trying to fix tests again
dieknolle3333 Jul 10, 2023
4939928
WIP: trying to fix tests (I really hope its the last time)
dieknolle3333 Jul 10, 2023
5f872b1
fix comment
dieknolle3333 Jul 10, 2023
ecdf655
change order of menu items
dieknolle3333 Jul 10, 2023
fc65635
Merge branch 'master' into segment-group-batch-actions
dieknolle3333 Jul 10, 2023
f3156ea
WIP: address PR review
dieknolle3333 Jul 11, 2023
1198f1d
WIP: address refactoring
dieknolle3333 Jul 12, 2023
9c1c06f
WIP: change icons
dieknolle3333 Jul 14, 2023
cdf453a
address review and lint
dieknolle3333 Jul 14, 2023
2803c61
merge master
dieknolle3333 Jul 14, 2023
76a09af
fix capitalization
dieknolle3333 Jul 14, 2023
720e052
fix spacing
dieknolle3333 Jul 14, 2023
bb1e286
fx method name
dieknolle3333 Jul 14, 2023
3baa2aa
fix icon spacing
dieknolle3333 Jul 17, 2023
caa361a
Merge branch 'master' into segment-group-batch-actions
dieknolle3333 Jul 17, 2023
a78963f
improve typing
philippotto Jul 17, 2023
bebb08f
Merge branch 'segment-group-batch-actions' of github.com:scalablemind…
philippotto Jul 17, 2023
a4dd2cd
WIP: fix spacing of font awesome icons and fix linting
dieknolle3333 Jul 17, 2023
e3892db
Merge branch 'segment-group-batch-actions' of github.com:scalablemind…
dieknolle3333 Jul 17, 2023
4d35fca
address review
dieknolle3333 Jul 17, 2023
eef4546
remove TODO comment and improve naming
dieknolle3333 Jul 18, 2023
414455e
Merge branch 'master' into segment-group-batch-actions
dieknolle3333 Jul 18, 2023
41034cc
merge master
dieknolle3333 Jul 18, 2023
eddc45e
merge master
dieknolle3333 Jul 20, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.unreleased.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
- Added a search feature for segments and segment groups. Listed segments/groups can be searched by id and name. [#7175](https://github.com/scalableminds/webknossos/pull/7175)
- Added support for transformations with thin plate splines. [#7131](https://github.com/scalableminds/webknossos/pull/7131)
- WEBKNOSSOS can now read S3 remote dataset credentials from environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_KEY`. Those will be used, if available, when accessing remote datasets for which no explicit credentials are supplied. [#7170](https://github.com/scalableminds/webknossos/pull/7170)
- Added security.txt according to [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116). The content is configurable and it can be disabled. [#7182](https://github.com/scalableminds/webknossos/pull/7182)
- Added tooltips to explain the task actions "Reset" and "Reset and Cancel". [#7201](https://github.com/scalableminds/webknossos/pull/7201)
- Added batch actions for segment groups, such as changing the color and loading or downloading all corresponding meshes. [#7164](https://github.com/scalableminds/webknossos/pull/7164).

### Changed
Expand Down
8 changes: 4 additions & 4 deletions frontend/javascripts/admin/dataset/dataset_upload_view.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ import Toast from "libs/toast";
import * as Utils from "libs/utils";
import messages from "messages";
import { trackAction } from "oxalis/model/helpers/analytics";
import { BlobReader, ZipReader, Entry } from "@zip.js/zip.js";
import Zip from "libs/zipjs_wrapper";
import {
CardContainer,
DatasetNameFormItem,
Expand Down Expand Up @@ -490,16 +490,16 @@ class DatasetUploadView extends React.Component<PropsWithFormAndRouter, State> {

if (fileExtension === "zip") {
try {
const reader = new ZipReader(new BlobReader(file));
const reader = new Zip.ZipReader(new Zip.BlobReader(file));
const entries = await reader.getEntries();
await reader.close();
const wkwFile = entries.find((entry: Entry) =>
const wkwFile = entries.find((entry: any) =>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this isn't pretty, but Zip.Entry doesnt work because the import of the wrapper is somehow not found

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed a commit (I hope the CI passes), which improves the typing a bit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thank you!

Utils.isFileExtensionEqualTo(entry.filename, "wkw"),
);
const needsConversion = wkwFile == null;
this.handleNeedsConversionInfo(needsConversion);

const nmlFile = entries.find((entry: Entry) =>
const nmlFile = entries.find((entry: any) =>
Utils.isFileExtensionEqualTo(entry.filename, "nml"),
);
if (nmlFile) {
Expand Down
14 changes: 14 additions & 0 deletions frontend/javascripts/libs/zipjs_wrapper.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
class TransFormStream {}

let mockedWindow = false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whats the purpose of this variable? if it's needed for some reason, you can add a comment to it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Omitting it doesn't seem to break anything, so I left it out.
Just for clarification. Technically this could have been used somewhere else because the scope of the variable is global, right? (https://www.w3schools.com/js/js_scope.asp)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically this could have been used somewhere else because the scope of the variable is global, right? (https://www.w3schools.com/js/js_scope.asp)

The linked article is a bit confusing (generally, w3schools ist not the best resource; I recommend MDN instead). However, the specific MDN article is also not perfect (it mentions module scope, but doesn't really explain when it comes into play).
To answer your question: no, it could not have been used somewhere else because the variable is module-scoped. That scoping is achieved during the compilation/bundling of all the module files. To get a truly global variable, one has to extend window (in a browser context) or global (in a node context).


// Mock zip.js and TransformStream during tests
if (!global.window) {
// @ts-expect-error
global.TransformStream = TransFormStream;
mockedWindow = true;
}

const Zip = require("@zip.js/zip.js");

export default Zip;
14 changes: 7 additions & 7 deletions frontend/javascripts/oxalis/model/actions/annotation_actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ export type StartedLoadingIsosurfaceAction = ReturnType<typeof startedLoadingIso
export type FinishedLoadingIsosurfaceAction = ReturnType<typeof finishedLoadingIsosurfaceAction>;
export type UpdateMeshFileListAction = ReturnType<typeof updateMeshFileListAction>;
export type UpdateCurrentMeshFileAction = ReturnType<typeof updateCurrentMeshFileAction>;
export type ImportIsosurfaceFromStlAction = ReturnType<typeof importIsosurfaceFromStlAction>;
export type ImportIsosurfaceFromSTLAction = ReturnType<typeof importIsosurfaceFromSTLAction>;
export type RemoveIsosurfaceAction = ReturnType<typeof removeIsosurfaceAction>;
export type AddAdHocIsosurfaceAction = ReturnType<typeof addAdHocIsosurfaceAction>;
export type AddPrecomputedIsosurfaceAction = ReturnType<typeof addPrecomputedIsosurfaceAction>;
Expand Down Expand Up @@ -74,7 +74,7 @@ export type AnnotationActionTypes =
| FinishedLoadingIsosurfaceAction
| UpdateMeshFileListAction
| UpdateCurrentMeshFileAction
| ImportIsosurfaceFromStlAction
| ImportIsosurfaceFromSTLAction
| RemoveIsosurfaceAction
| AddAdHocIsosurfaceAction
| AddPrecomputedIsosurfaceAction
Expand Down Expand Up @@ -213,23 +213,23 @@ export const maybeFetchMeshFilesAction = (
} as const);

export const triggerIsosurfaceDownloadAction = (
cellName: string,
segmentName: string,
segmentId: number,
layerName: string,
) =>
({
type: "TRIGGER_ISOSURFACE_DOWNLOAD",
cellName,
segmentName,
segmentId,
layerName,
} as const);

export const triggerIsosurfacesDownloadAction = (
surfaceArray: { cellName: string; segmentId: number; layerName: string }[],
segmentsArray: Array<{ segmentName: string; segmentId: number; layerName: string }>,
) =>
({
type: "TRIGGER_ISOSURFACES_DOWNLOAD",
surfaceArray,
segmentsArray,
} as const);

export const refreshIsosurfacesAction = () =>
Expand Down Expand Up @@ -275,7 +275,7 @@ export const updateCurrentMeshFileAction = (
meshFileName,
} as const);

export const importIsosurfaceFromStlAction = (layerName: string, buffer: ArrayBuffer) =>
export const importIsosurfaceFromSTLAction = (layerName: string, buffer: ArrayBuffer) =>
({
type: "IMPORT_ISOSURFACE_FROM_STL",
layerName,
Expand Down
57 changes: 27 additions & 30 deletions frontend/javascripts/oxalis/model/sagas/isosurface_saga.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import type { Action } from "oxalis/model/actions/actions";
import type { Vector3 } from "oxalis/constants";
import { MappingStatusEnum } from "oxalis/constants";
import {
ImportIsosurfaceFromStlAction,
ImportIsosurfaceFromSTLAction,
UpdateIsosurfaceVisibilityAction,
RemoveIsosurfaceAction,
RefreshIsosurfaceAction,
Expand Down Expand Up @@ -74,7 +74,7 @@ import processTaskWithPool from "libs/task_pool";
import { getBaseSegmentationName } from "oxalis/view/right-border-tabs/segments_tab/segments_view_helper";
import { RemoveSegmentAction, UpdateSegmentAction } from "../actions/volumetracing_actions";
import { ResolutionInfo } from "../helpers/resolution_info";
import { BlobReader, BlobWriter, ZipWriter } from "@zip.js/zip.js";
import Zip from "libs/zipjs_wrapper";

export const NO_LOD_MESH_INDEX = -1;
const MAX_RETRY_COUNT = 5;
Expand Down Expand Up @@ -104,7 +104,7 @@ function marchingCubeSizeInMag1(): Vector3 {
: [128, 128, 128];
}
const modifiedCells: Set<number> = new Set();
export function isIsosurfaceStl(buffer: ArrayBuffer): boolean {
export function isIsosurfaceSTL(buffer: ArrayBuffer): boolean {
const dataView = new DataView(buffer);
const isIsosurface = stlIsosurfaceConstants.isosurfaceMarker.every(
(marker, index) => dataView.getUint8(index) === marker,
Expand Down Expand Up @@ -948,14 +948,7 @@ function* downloadIsosurfaceCellById(
}

try {
const stlDataViews = exportToStl(geometry);
// Encode isosurface and cell id property
const { isosurfaceMarker, segmentIdIndex } = stlIsosurfaceConstants;
isosurfaceMarker.forEach((marker, index) => {
stlDataViews[0].setUint8(index, marker);
});
stlDataViews[0].setUint32(segmentIdIndex, segmentId, true);
const blob = new Blob(stlDataViews);
const blob = getSTLBlob(geometry, segmentId);
yield* call(saveAs, blob, `${cellName}-${segmentId}.stl`);
} catch (exception) {
ErrorHandling.notify(exception as Error);
Expand All @@ -965,12 +958,12 @@ function* downloadIsosurfaceCellById(
}

function* downloadIsosurfaceCellsAsZIP(
arrayOfCells: { cellName: string; segmentId: number; layerName: string }[],
segments: Array<{ segmentName: string; segmentId: number; layerName: string }>,
): Saga<void> {
const { segmentMeshController } = getSceneController();
const zipWriter = new ZipWriter(new BlobWriter("application/zip"));
const zipWriter = new Zip.ZipWriter(new Zip.BlobWriter("application/zip"));
try {
const promises = arrayOfCells.map((element) => {
const addFileToZipWriterPromises = segments.map((element) => {
const geometry = segmentMeshController.getIsosurfaceGeometryInBestLOD(
element.segmentId,
element.layerName,
Expand All @@ -983,35 +976,39 @@ function* downloadIsosurfaceCellsAsZIP(
});
return;
}
const stlDataViews = exportToStl(geometry);
// Encode isosurface and cell id property
const { isosurfaceMarker, segmentIdIndex } = stlIsosurfaceConstants;
isosurfaceMarker.forEach((marker, index) => {
stlDataViews[0].setUint8(index, marker);
});
stlDataViews[0].setUint32(segmentIdIndex, element.segmentId, true);
const stlDataReader = new BlobReader(new Blob(stlDataViews));
return zipWriter.add(`${element.cellName}-${element.segmentId}.stl`, stlDataReader);
const stlDataReader = new Zip.BlobReader(getSTLBlob(geometry, element.segmentId));
return zipWriter.add(`${element.segmentName}-${element.segmentId}.stl`, stlDataReader);
});
yield all(promises);
yield all(addFileToZipWriterPromises);
const result = yield* call([zipWriter, zipWriter.close]);
saveAs(result, "mesh-export.zip");
yield* call(saveAs, result as Blob, "mesh-export.zip");
} catch (exception) {
ErrorHandling.notify(exception as Error);
console.error(exception);
Toast.error("Could not export to STL. See console for details");
Toast.error("Could not export meshes as STL files. See console for details");
}
}

const getSTLBlob = (geometry: THREE.Group, segmentId: number): Blob => {
const stlDataViews = exportToStl(geometry);
// Encode isosurface and cell id property
const { isosurfaceMarker, segmentIdIndex } = stlIsosurfaceConstants;
isosurfaceMarker.forEach((marker, index) => {
stlDataViews[0].setUint8(index, marker);
});
stlDataViews[0].setUint32(segmentIdIndex, segmentId, true);
return new Blob(stlDataViews);
};

function* downloadIsosurfaceCell(action: TriggerIsosurfaceDownloadAction): Saga<void> {
yield* call(downloadIsosurfaceCellById, action.cellName, action.segmentId, action.layerName);
yield* call(downloadIsosurfaceCellById, action.segmentName, action.segmentId, action.layerName);
}

function* downloadIsosurfaceCells(action: TriggerIsosurfacesDownloadAction): Saga<void> {
yield* call(downloadIsosurfaceCellsAsZIP, action.surfaceArray);
yield* call(downloadIsosurfaceCellsAsZIP, action.segmentsArray);
}

function* importIsosurfaceFromStl(action: ImportIsosurfaceFromStlAction): Saga<void> {
function* importIsosurfaceFromSTL(action: ImportIsosurfaceFromSTLAction): Saga<void> {
const { layerName, buffer } = action;
const dataView = new DataView(buffer);
const segmentId = dataView.getUint32(stlIsosurfaceConstants.segmentIdIndex, true);
Expand Down Expand Up @@ -1081,7 +1078,7 @@ export default function* isosurfaceSaga(): Saga<void> {
yield* takeEvery(loadPrecomputedMeshActionChannel, loadPrecomputedMesh);
yield* takeEvery("TRIGGER_ISOSURFACE_DOWNLOAD", downloadIsosurfaceCell);
yield* takeEvery("TRIGGER_ISOSURFACES_DOWNLOAD", downloadIsosurfaceCells);
yield* takeEvery("IMPORT_ISOSURFACE_FROM_STL", importIsosurfaceFromStl);
yield* takeEvery("IMPORT_ISOSURFACE_FROM_STL", importIsosurfaceFromSTL);
yield* takeEvery("REMOVE_ISOSURFACE", removeIsosurface);
yield* takeEvery("REMOVE_SEGMENT", handleRemoveSegment);
yield* takeEvery("REFRESH_ISOSURFACES", refreshIsosurfaces);
Expand Down