merge master

CHANGELOG.unreleased.md

@@ -15,10 +15,13 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - In addition to drag and drop, the selected tree(s) in the Skeleton tab can also be moved into another group by right-clicking the target group and selecting "Move selected tree(s) here". [#7005](https://github.com/scalableminds/webknossos/pull/7005)
 - Added support for remote datasets encoded with [brotli](https://datatracker.ietf.org/doc/html/rfc7932). [#7041](https://github.com/scalableminds/webknossos/pull/7041)
 - Teams can be edited more straight-forwardly in a popup in the team edit page. [#7043](https://github.com/scalableminds/webknossos/pull/7043)
+- Annotations with Editable Mappings (a.k.a Supervoxel Proofreading) can now be merged. [#7026](https://github.com/scalableminds/webknossos/pull/7026)
+- The file size and inodes of artifacts are now aggregated and shown in the Voxelytics workflow list. [#7071](https://github.com/scalableminds/webknossos/pull/7071)
 
 ### Changed
 - Loading of precomputed meshes got significantly faster (especially when using a mesh file for an oversegmentation with an applied agglomerate mapping). [#7001](https://github.com/scalableminds/webknossos/pull/7001)
 - Improved speed of proofreading by only reloading affected areas after a split or merge. [#7050](https://github.com/scalableminds/webknossos/pull/7050)
+- The minimum length of layer names in datasets was set from 3 to 1, enabling single-character names for layers. [#7064](https://github.com/scalableminds/webknossos/pull/7064)
 
 ### Fixed
 - Fixed that changing a segment color could lead to a crash. [#7000](https://github.com/scalableminds/webknossos/pull/7000)
@@ -32,6 +35,10 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
 - Fixed a bug where users could sometimes not access their own time tracking information. [#7055](https://github.com/scalableminds/webknossos/pull/7055)
 - Fixed a bug in the wallTime calculation of the Voxelytics reports. [#7059](https://github.com/scalableminds/webknossos/pull/7059) 
 - Fixed a bug where thumbnails and raw data requests with non-bucket-aligned positions would show data at slightly wrong positions. [#7058](https://github.com/scalableminds/webknossos/pull/7058)
+- Fixed rare rendering bug for datasets with multiple layers and differing magnifications. [#7066](https://github.com/scalableminds/webknossos/pull/7066)
+- Fixed a bug where duplicating annotations with Editable Mappings could lead to a server-side endless loop. [#7026](https://github.com/scalableminds/webknossos/pull/7026)
+- Fixed the datasource-properties.json route for zarr-streaminge export of datasets that are not wkw/zarr.  [#7065](https://github.com/scalableminds/webknossos/pull/7065)
+- Fixed an issue where you could no longer invite users to your organization even though you had space left. [#7078](https://github.com/scalableminds/webknossos/pull/7078)
 - Fixed displayed units of used storage in the organization's overview page. [#7057](https://github.com/scalableminds/webknossos/pull/7057)
 
 ### Removed

MIGRATIONS.unreleased.md

@@ -10,5 +10,6 @@ User-facing changes are documented in the [changelog](CHANGELOG.released.md).
 - FossilDB needs to be opened with new additional column families editableMappingsInfo, editableMappingsAgglomerateToGraph, editableMappingsSegmentToAgglomerate.
 - For instances with existing editable mapping (a.k.a supervoxel proofreading) annotations: To keep those annotations alive, a python migration has to be run with access to your tracingstore’s FossilDB. It is recommended to do this during a webknossos downtime to avoid data loss. It needs python 3.8+ and the pip packages installable by `pip install grpcio-tools grpcio-health-checking`. Run it with `python tools/migrate-editable-mappings/migrate-editable-mappings.py -v -w -o localhost:7155`. Omit -o for a faster migration but no access to older versions of the editable mappings. The migration is idempotent.
 - The datastore now needs `brotli`. For Debian-based systems, this can be installed with `apt-get install libbrotli1`.
+- New FossilDB version 0.1.23 (`master__448` on Dockerhub) is required, compare [FossilDB PR](https://github.com/scalableminds/fossildb/pull/38).
 
 ### Postgres Evolutions:

app/controllers/AnnotationIOController.scala

@@ -365,14 +365,15 @@ Expects:
       } yield result
     }
 
-  // TODO: select versions per layer
   private def downloadExplorational(annotationId: String,
                                     typ: String,
                                     issuingUser: Option[User],
                                     skeletonVersion: Option[Long],
                                     volumeVersion: Option[Long],
                                     skipVolumeData: Boolean)(implicit ctx: DBAccessContext) = {
 
+    // Note: volumeVersion cannot currently be supplied per layer, see https://github.com/scalableminds/webknossos/issues/5925
+
     def skeletonToTemporaryFile(dataSet: DataSet,
                                 annotation: Annotation,
                                 organizationName: String): Fox[TemporaryFile] =

app/controllers/DataSetController.scala

@@ -46,7 +46,6 @@ object DatasetUpdateParameters extends TristateOptionJsonHelper {
 class DataSetController @Inject()(userService: UserService,
                                   userDAO: UserDAO,
                                   dataSetService: DataSetService,
-                                  dataSetDataLayerDAO: DataSetDataLayerDAO,
                                   dataStoreDAO: DataStoreDAO,
                                   dataSetLastUsedTimesDAO: DataSetLastUsedTimesDAO,
                                   organizationDAO: OrganizationDAO,
@@ -161,7 +160,7 @@ class DataSetController @Inject()(userService: UserService,
             reportMutable += "Error when exploring as layer set: Resulted in zero layers."
             None
           case f: Failure =>
-            reportMutable += s"Error when exploring as layer set: ${exploreRemoteLayerService.formatFailureForReport(f)}"
+            reportMutable += s"Error when exploring as layer set: ${Fox.failureChainAsString(f)}"
             None
           case Empty =>
             reportMutable += "Error when exploring as layer set: Empty"

app/models/binary/DataSet.scala

@@ -671,8 +671,12 @@ class DataSetDataLayerDAO @Inject()(
 
   def findAllForDataSet(dataSetId: ObjectId): Fox[List[DataLayer]] =
     for {
-      rows <- run(DatasetLayers.filter(_._Dataset === dataSetId.id).result).map(_.toList)
-      rowsParsed <- Fox.combined(rows.map(parseRow(_, dataSetId)))
+      rows <- run(q"""SELECT _dataSet, name, category, elementClass, boundingBox, largestSegmentId, mappings,
+                          defaultViewConfiguration, adminViewConfiguration
+                      FROM webknossos.dataset_layers
+                      WHERE _dataset = $dataSetId
+                      ORDER BY name""".as[DatasetLayersRow])
+      rowsParsed <- Fox.combined(rows.toList.map(parseRow(_, dataSetId)))
     } yield rowsParsed
 
   private def insertLayerQuery(dataSetId: ObjectId, layer: DataLayer): SqlAction[Int, NoStream, Effect] =

app/models/binary/explore/ExploreRemoteLayerService.scala

@@ -16,7 +16,7 @@ import com.scalableminds.webknossos.datastore.storage.{DataVaultsHolder, RemoteS
 import com.typesafe.scalalogging.LazyLogging
 import models.binary.credential.CredentialService
 import models.user.User
-import net.liftweb.common.{Box, Empty, Failure, Full}
+import net.liftweb.common.{Empty, Failure, Full}
 import net.liftweb.util.Helpers.tryo
 import oxalis.security.WkEnv
 import play.api.libs.json.{Json, OFormat}
@@ -199,21 +199,12 @@ class ExploreRemoteLayerService @Inject()(credentialService: CredentialService)
             reportMutable += s"Found ${layersWithVoxelSizes.length} ${currentExplorer.name} layers at $remotePath."
             Fox.successful(layersWithVoxelSizes)
           case f: Failure =>
-            reportMutable += s"Error when reading $remotePath as ${currentExplorer.name}: ${formatFailureForReport(f)}"
+            reportMutable += s"Error when reading $remotePath as ${currentExplorer.name}: ${Fox.failureChainAsString(f)}"
             exploreRemoteLayersForRemotePath(remotePath, credentialId, reportMutable, remainingExplorers)
           case Empty =>
             reportMutable += s"Error when reading $remotePath as ${currentExplorer.name}: Empty"
             exploreRemoteLayersForRemotePath(remotePath, credentialId, reportMutable, remainingExplorers)
         }
     }
 
-  def formatFailureForReport(failure: Failure): String = {
-    def formatChain(chain: Box[Failure]): String = chain match {
-      case Full(failure) =>
-        " <~ " + failure.msg + formatChain(failure.chain)
-      case _ => ""
-    }
-    failure.msg + formatChain(failure.chain)
-  }
-
 }

app/models/voxelytics/VoxelyticsDAO.scala

@@ -397,7 +397,9 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
         VoxelyticsRunState.STALE)} THEN 1 ELSE 0 END) AS failed,
           SUM(CASE WHEN COALESCE(running_tasks.state, any_tasks.state) = ${VoxelyticsRunState.SKIPPED} THEN 1 ELSE 0 END) AS skipped,
           SUM(CASE WHEN COALESCE(running_tasks.state, any_tasks.state) = ${VoxelyticsRunState.COMPLETE} THEN 1 ELSE 0 END) AS complete,
-          SUM(CASE WHEN COALESCE(running_tasks.state, any_tasks.state) = ${VoxelyticsRunState.CANCELLED} THEN 1 ELSE 0 END) AS cancelled
+          SUM(CASE WHEN COALESCE(running_tasks.state, any_tasks.state) = ${VoxelyticsRunState.CANCELLED} THEN 1 ELSE 0 END) AS cancelled,
+          SUM(COALESCE(running_tasks.fileSize, 0)) AS fileSize,
+          SUM(COALESCE(running_tasks.inodeCount, 0)) AS inodeCount
         FROM webknossos.voxelytics_workflows w
         JOIN ( -- Aggregating the task states of workflow runs (including skipped and pending)
           SELECT DISTINCT ON (workflow_hash, taskName) r.workflow_hash workflow_hash, t.name taskName, ts.state state
@@ -407,22 +409,39 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
           ORDER BY workflow_hash, taskName, ts.beginTime DESC
         ) any_tasks ON any_tasks.workflow_hash = w.hash
         LEFT JOIN ( -- Aggregating the task states of workflow runs (excluding skipped and pending)
-          SELECT DISTINCT ON (workflow_hash, taskName) r.workflow_hash workflow_hash, t.name taskName, ts.state state
+          SELECT
+            DISTINCT ON (workflow_hash, taskName)
+            r.workflow_hash workflow_hash,
+            t.name taskName,
+            ts.state state,
+            COALESCE(ta.fileSize, 0) fileSize,
+            COALESCE(ta.inodeCount, 0) inodeCount
           FROM webknossos.voxelytics_tasks t
           JOIN (${visibleRunsQ(currentUser, allowUnlisted = false)}) r ON t._run = r._id
           JOIN task_states ts ON ts._id = t._id
+          LEFT JOIN (
+            SELECT a._task _task, SUM(a.fileSize) fileSize, SUM(a.inodeCount) inodeCount
+            FROM webknossos.voxelytics_artifacts a
+            GROUP BY a._task
+          ) ta ON ta._task = t._id
           WHERE ts.state NOT IN ${SqlToken.tupleFromValues(VoxelyticsRunState.SKIPPED, VoxelyticsRunState.PENDING)}
           ORDER BY workflow_hash, taskName, ts.beginTime DESC
         ) running_tasks ON running_tasks.workflow_hash = w.hash AND running_tasks.taskName = any_tasks.taskName
         WHERE w.hash IN ${SqlToken.tupleFromList(workflowHashes)} AND w._organization = $organizationId
         GROUP BY w.hash
-        """.as[(String, Long, Long, Long, Long, Long)])
+        """.as[(String, Long, Long, Long, Long, Long, Long, Long)])
     } yield
       r.toList
         .map(
           row =>
             (row._1,
-             TaskCounts(total = row._2, failed = row._3, skipped = row._4, complete = row._5, cancelled = row._6)))
+             TaskCounts(total = row._2,
+                        failed = row._3,
+                        skipped = row._4,
+                        complete = row._5,
+                        cancelled = row._6,
+                        fileSize = row._7,
+                        inodeCount = row._8)))
         .toMap
   }
 
@@ -446,7 +465,9 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
           COALESCE(tasks.failed, 0) AS tasksFailed,
           COALESCE(tasks.skipped, 0) AS tasksSkipped,
           COALESCE(tasks.complete, 0) AS tasksComplete,
-          COALESCE(tasks.cancelled, 0) AS tasksCancelled
+          COALESCE(tasks.cancelled, 0) AS tasksCancelled,
+          COALESCE(tasks.fileSize, 0) AS fileSize,
+          COALESCE(tasks.inodeCount, 0) AS inodeCount
         FROM (${visibleRunsQ(currentUser, allowUnlisted = false)}) r
         JOIN (${runsWithStateQ(staleTimeout)}) rs ON rs._id = r._id
         LEFT JOIN (
@@ -457,9 +478,16 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
           .tupleFromValues(VoxelyticsRunState.FAILED, VoxelyticsRunState.STALE)} THEN 1 ELSE 0 END) AS failed,
             SUM(CASE WHEN ts.state = ${VoxelyticsRunState.SKIPPED} THEN 1 ELSE 0 END) AS skipped,
             SUM(CASE WHEN ts.state = ${VoxelyticsRunState.COMPLETE} THEN 1 ELSE 0 END) AS complete,
-            SUM(CASE WHEN ts.state = ${VoxelyticsRunState.CANCELLED} THEN 1 ELSE 0 END) AS cancelled
+            SUM(CASE WHEN ts.state = ${VoxelyticsRunState.CANCELLED} THEN 1 ELSE 0 END) AS cancelled,
+            SUM(COALESCE(ta.fileSize, 0)) AS fileSize,
+            SUM(COALESCE(ta.inodeCount, 0)) AS inodeCount
           FROM webknossos.voxelytics_tasks AS t
           JOIN (${tasksWithStateQ(staleTimeout)}) ts ON ts._id = t._id
+          LEFT JOIN (
+            SELECT a._task _task, SUM(a.fileSize) fileSize, SUM(a.inodeCount) inodeCount
+            FROM webknossos.voxelytics_artifacts a
+            GROUP BY a._task
+          ) ta ON ta._task = t._id
           GROUP BY t._run
         ) tasks ON tasks._run = r._id
         WHERE r._organization = $organizationId
@@ -476,6 +504,8 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
                 Long,
                 Long,
                 Long,
+                Long,
+                Long,
                 Long)])
       results <- Fox.combined(
         r.toList.map(
@@ -498,7 +528,9 @@ class VoxelyticsDAO @Inject()(sqlClient: SqlClient)(implicit ec: ExecutionContex
                   failed = row._11,
                   skipped = row._12,
                   complete = row._13,
-                  cancelled = row._14
+                  cancelled = row._14,
+                  fileSize = row._15,
+                  inodeCount = row._16
                 )
             )))
     } yield results

app/models/voxelytics/VoxelyticsService.scala

@@ -57,7 +57,13 @@ object WorkflowEntry {
   implicit val jsonFormat: OFormat[WorkflowEntry] = Json.format[WorkflowEntry]
 }
 
-case class TaskCounts(total: Long, failed: Long, skipped: Long, complete: Long, cancelled: Long)
+case class TaskCounts(total: Long,
+                      failed: Long,
+                      skipped: Long,
+                      complete: Long,
+                      cancelled: Long,
+                      fileSize: Long,
+                      inodeCount: Long)
 
 object TaskCounts {
   implicit val jsonFormat: OFormat[TaskCounts] = Json.format[TaskCounts]

app/oxalis/mail/MailchimpTicker.scala

@@ -18,7 +18,6 @@ import scala.concurrent.duration._
 class MailchimpTicker @Inject()(val lifecycle: ApplicationLifecycle,
                                 val system: ActorSystem,
                                 multiUserDAO: MultiUserDAO,
-                                userDAO: MultiUserDAO,
                                 mailchimpClient: MailchimpClient)(implicit ec: ExecutionContext)
     extends IntervalScheduler
     with LazyLogging {

docker-compose.yml

@@ -270,11 +270,11 @@ services:
 
   # FossilDB
   fossildb:
-    image: scalableminds/fossildb:master__410
+    image: scalableminds/fossildb:master__448
     command:
       - fossildb
       - -c
-      - skeletons,skeletonUpdates,volumes,volumeData,volumeUpdates,editableMappings,editableMappingUpdates
+      - skeletons,skeletonUpdates,volumes,volumeData,volumeUpdates,editableMappings,editableMappingUpdates,editableMappingsInfo,editableMappingsAgglomerateToGraph,editableMappingsSegmentToAgglomerate
     user: ${USER_UID:-fossildb}:${USER_GID:-fossildb}
 
   fossildb-persisted:

fossildb/version

@@ -1 +1 @@
-0.1.22
+0.1.23

frontend/javascripts/admin/dataset/dataset_components.tsx

@@ -34,7 +34,7 @@ export function CardContainer({
 }
 export const layerNameRules = [
   {
-    min: 3,
+    min: 1,
   },
   // Note that these rules are also checked by the backend
   {
@@ -57,6 +57,7 @@ export const getDatasetNameRules = (
     required: true,
     message: messages["dataset.import.required.name"],
   },
+  { min: 3, message: messages["dataset.name_length"] },
   ...layerNameRules,
   {
     validator: async (_rule: any, value: string) => {

frontend/javascripts/admin/onboarding.tsx

@@ -306,7 +306,7 @@ export class InviteUsersModal extends React.Component<
 
   render() {
     const isInvitesDisabled =
-      this.props.currentUserCount + this.extractEmailAddresses().length >=
+      this.props.currentUserCount + this.extractEmailAddresses().length >
       this.props.maxUserCountPerOrganization;
 
     return (

frontend/javascripts/admin/voxelytics/artifacts_view.tsx

@@ -4,7 +4,7 @@ import { Button, Card, message } from "antd";
 import { CopyOutlined } from "@ant-design/icons";
 import { VoxelyticsArtifactConfig } from "types/api_flow_types";
 import { getVoxelyticsArtifactChecksums } from "admin/admin_rest_api";
-import { formatBytes } from "libs/format_utils";
+import { formatCountToDataAmountUnit } from "libs/format_utils";
 import { copyToClipboad, isObjectEmpty, useTheme } from "./utils";
 
 function renderArtifactPath(artifact: VoxelyticsArtifactConfig) {
@@ -128,7 +128,7 @@ function ArtifactsView({
       <>
         <span>{artifactName}</span>
         <span style={{ fontSize: "10px", marginLeft: 10 }}>
-          \\ version {artifact.version}, {formatBytes(artifact.fileSize)},{" "}
+          \\ version {artifact.version}, {formatCountToDataAmountUnit(artifact.fileSize)},{" "}
           {artifact.inodeCount.toLocaleString()} inodes
         </span>
       </>

frontend/javascripts/admin/voxelytics/statistics_tab.tsx

@@ -4,7 +4,7 @@ import { SyncOutlined } from "@ant-design/icons";
 import { getVoxelyticsChunkStatistics } from "admin/admin_rest_api";
 import { usePolling } from "libs/react_hooks";
 import {
-  formatBytes,
+  formatCountToDataAmountUnit,
   formatCPU,
   formatDistanceStrict,
   formatDurationStrict,
@@ -127,21 +127,21 @@ export default function StatisticsTab({
                   {row.memory?.max != null && (
                     <>
                       <span className="stats-label">Max</span>{" "}
-                      {formatBytes(row.memory.max * 1024 * 1024)}
+                      {formatCountToDataAmountUnit(row.memory.max * 1024 * 1024)}
                     </>
                   )}
                   <br />
                   {row.memory?.median != null && (
                     <>
                       <span className="stats-label">Median</span>{" "}
-                      {formatBytes(row.memory.median * 1024 * 1024)}
+                      {formatCountToDataAmountUnit(row.memory.median * 1024 * 1024)}
                     </>
                   )}
                   <br />
                   {row.memory?.stddev != null && (
                     <>
                       <span className="stats-label">Stddev</span>{" "}
-                      {formatBytes(row.memory.stddev * 1024 * 1024)}
+                      {formatCountToDataAmountUnit(row.memory.stddev * 1024 * 1024)}
                     </>
                   )}
                 </td>

frontend/javascripts/admin/voxelytics/task_list_view.tsx

@@ -666,7 +666,13 @@ function aggregateTaskInfos(
         taskName: task.key,
         state: VoxelyticsRunState.SKIPPED,
         currentExecutionId: null,
-        chunkCounts: { total: 0, failed: 0, skipped: 0, complete: 0, cancelled: 0 },
+        chunkCounts: {
+          total: 0,
+          failed: 0,
+          skipped: 0,
+          complete: 0,
+          cancelled: 0,
+        },
         beginTime: null,
         endTime: null,
         runs: [],