Skip to content

0.4.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 08 Jul 05:24
· 112 commits to refs/heads/main since this release
0.4.0
82ffc52
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

This release adds new types of labels like spots, overlapping cells, and divisions and rearchitects the client-server relationship to manage project data on the client.

The core change is switching to a new file format, where all the files for a project are bundled into a zip. Each file is a modular type of labels, like the segmentation, cells, divisions, and spots. We intend to extend this format, adding further labels that can leverage existing labels or add new types of annotations entirely.

Merge 0.4-dev before releasing 0.4 @tddough98 (#352)

This brings all the work on DeepCell Label so far this year into the main branch for the 0.4 release. As the release drafter hasn't been including updates from PRs into this branch, I added some high level bullet points with the largest features and improvements such as

  • overlapping labels (#323)
  • spots labels (#317)
  • cells labels that can reassign segmented regions without rewriting the segmentation image (#330)
  • division labels to replace the lineage (#331)
  • replacing the database driven architecture with a client side approach where the frontend stores and updates the labeled data locally (#311)
    • client side undo redo (#325)
    • client side data persistence in IndexedDB (#326)
  • end-to-end integration tests with cypress (#314)
Switch master to main @tddough98 (#354)

I changed the default branch to repo to be named main. This updates references to master to main instead.

Add c to cells @tddough98 (#350)

Addresses #347. I decided this was a more critical issue as loading and editing any mesmer results exposes the issue. I did not add "channel mode" editing controls to edit cells across multiple segmentation channels because mesmer does not have consistent cells between channels anyways.

Please user test this (do a handful of actions, undo them, redo them, reload them, combinations of these) on label-dev.deepcell.org!

Fix undoing edits to the segmentation @tddough98 (#349)

This fixes undo after editing the segmentation. It also catches a bug with the cells not persisting on reload immediately after editing the segmentation and adds TODO comments for #347.

Add python-magic-bin @tddough98 (#351)

When setting up the deepcell label repository with Kevin Yu this week, our incoming schmidt software engineer, there was an issue with using python-magic on windows. Installing python-magic-bin as a python dependency solved the issue.

Bump pillow version to address security alerts @tddough98 (#346)

Bumps pillow version to 9.0.1 to deal with dependabot security alerts

Finish merging dev-documentation in 0.4-dev @tddough98 (#345)

This is the same issue as #332 where I forgot to delete a branch and then merged more PRs into the hanging branch.

Reorganize subprojects @tddough98 (#343)

This renames the visualizer folder to frontend and the deepcell_label folder to backend. It also moves resources specific to the Python backend into the backend folder.

Fix mesmer loading @tddough98 (#344)

This fixes an issue with loading results from mesmer from deepcell.org/predict. It came from slight differences in dimension order between the segmentation pipeline and the mesmer pipeline, as documented here: vanvalenlab/kiosk-frontend#202

Add developer documentation @tddough98 (#342)

This adds docstrings to all state machine files and adds ARCHITECUTE.md to cover the high level architecture of the frontend and backend, with closer attention to the tree of state machines.

Fix loading from deepcell.org/predict @tddough98 (#341)

This addresses some quick fixes for loading from deepcell.org/predict (now online!).

The first issue was from an incorrect test case that caused loading zips of tiffs with batches to drop the X dimension.
The second issue was assuming the smallest dimension is the channels when axes are not provided, causing tracking timelapses to use their time axis as channels instead.

Add label zip format documentation @tddough98 (#337)

This adds a markdown file explaining the contents of the project zips created and exported by DeepCell Label. It also includes a section on the supported input for /api/project to create these zips.

There is a new documentation folder to store this and other markdown files.

Handle batches when loading TIFFs and zips of TIFFs @tddough98 (#319)

As we support batch dimensions on the kiosk-frontend, we need a way to handle differently shaped input images and predicted labels from the predict page.

  • Adds back a dimension order for the image file
  • Checks if tiff filenames contain batch_XX when loading a zip of tiffs and groups and loads files by their batch number
Add forceLoadOutput URL parameter @tddough98 (#334)

To better support the Anolytics job workflow, we add a query string forceLoadOutput. When the query string is true, the page loads from the deepcell-label-output bucket. If the project is already in the local browser DB, we show a modal to confirm overwriting the local data. When the project has not been submitted, we show a missing project page with instructions to submit the project first.

For the backend to receive the query string to load from the output bucket, CloudFront needs to forward query strings, which it does not do by default. Setting the Origin request policy for the Elastic Beanstalk backend to AllViewer will forward all cookies, headers, and query strings. I will document this and all other cloudfront/s3/elastic beanstalk configuration in an upcoming deployment guide.

Bug fixes

  • prevent canvas from moving when changing tabs
  • bug fix for undoing an action and then reloading before doing another action
  • fix for loading PNGs with I or F mode (32 bit single channels)
  • fix for setting dynamic range to 0 in grayscale mode
Add deployment guide @tddough98 (#335)

This adds a markdown file to walk through creating and configuring the cloud resources to put DeepCell Label online. As I'll be leaving next month, it's intended to serve as a resource for future work on the project to make sure it can be redeployed and updated. As we discover new required configuration and switch the resources we use, we should update this guide with the up to date steps for deployment.

Update layout, instructions, exporting and remove visualizers @tddough98 (#333)

This contains some of the final changes needed before cutting the 0.4 release of DeepCell Label. It includes

  • updating the in app drop down instructions
  • move the display controls into a tab
  • switching between tabs when using a keybind from another tab
  • fixing bugs with too many WebGL contexts by having a shared GPU object with a single context

The changes are online at label-dev.deepcell.org

Tweak .env.example to avoid environment variables set to "" @tddough98 (#336)

When a line like SQLALCHEMY_DATABASE_URI= is in .env, the environment variable is set to "" instead of using the default value. By commenting these fields out in the example environment, this common setup issue is avoided.

This also switches fmd_config.cfg.example to fmd_config.cfg in .env.example to further prevent using fmd_config.cfg.example accidentally.

Replace lineage with divisions and add division editing @tddough98 (#331)

Overview

This overhauls how divisions labels are encoded, changing to a event focused format divisions.json

[{"parent": 1, "daughters": [2, 3], "t": 1}]

from the existing lineage.json format

{
  "1": {
    "label": 1,
    "frames": [0],
    "frame_div": 1,
    "capped": true,
    "daughters": [2, 3],
    "parent": null,
  },
  "2": {
    "label": 2,
    "frames": [1],
    "frame_div": null,
    "capped": false,
    "daughters": [],
    "parent": 1,
  },
  "3": {
    "label": 3,
    "frames": [1],
    "frame_div": null,
    "capped": false,
    "daughters": [],
    "parent": 1,
  },
}

Loading

The loader on the backend now looks for divisions.json in the input zip. For backwards compatibility, it supports loading lineage.json from .trk files and converts the lineage to divisions.json.

Editing

Editing the divisions is handled by divisionsMachine. There is another separate editDivisionMachine to manage the UI state of editing divisions, like whether the user is being prompted to add a daughter to a division, which then sends events to divisionsMachine. Once the divisions change, we send a DIVISIONS event to the divisions event bus.

When the cell labels are edited, we forward these events (REPLACE, SWAP, or DELETE) so the divisions can also replace, swap, or delete the same cells present in division labels. "Replace with parent" and "create new cell" have been removed from the divisions tab as they are found in the cells tab instead and update the divisions.

Editing the segmentation also may change the cell labels. The segment edit API sends updates for the cell labels, but these are not yet propagated to the divisions. I'm still working on how to update divisions after segmentation edits. For now, as long as a parent or daughter cell is not deleted entirely using the segment API, the divisions stay valid.

Edits to the divisions are also recorded in a labelHistoryMachine, allowing division edits to be undone and redone. The divisionsMachine sends a SNAPSHOT event to its history machine to record what the divisions were before and after the edit.

The idbMachine listens to the divisions event bus for DIVISIONS event to persist the current divisions in IndexedDB.

Other changes

Components that display divisions now use the hooks useParentDivisions(cell) and useDaughterDivisions(cell) to get the division events that the cell is involved in as a parent or daughter.

Swapping and replacing cell labels may make a cell a parent or daughter in multiple divisions. We now show all divisions where the selected cell appears.

Minor code changes

  • Rename Tracking to Divisions in UI text and component names
  • Rename LabelControls to EditControls. This is the column just to the left of the canvas with the tools to edit the segmentation, cells, or divisions
  • Rename numFrames to duration and frame to t or time
  • Switch z to t in cell labels
  • In label.py, new_cell and new_value are dynamically calculated instead of set at initialization so actions don't need to update them within the Edit class

Update 6/9/2022

I've been squashing bugs with @msschwartz21 and tweaking functionality since putting label-dev.deepcell.org online. Here's a summary of new changes

  • Display
    • Add color prop for OutlineCellCanvas
    • Show the daughter to be added on the canvas
    • Show outer outline of selected cell when it is the flood cell
    • Center each box in the cell timelines around markers on sliders
    • Prevent vertical white lines from appearing in timeline
  • Performance
    • Run idbMachine in webworker
    • Switch from drawing canvases to drawing image bitmaps to improvement drawImage performance
  • Usability
    • Handle segment edit API errors gracefully
    • Prevent swap and replace with self
    • Create two stages for add daughter prompt text
    • Clear second selected cell for flood, replace and swap when pressing Esc
    • Select a cell when using flood, replace, and swap if none selected
    • Add keybinds to use cell tools (V for select, Backspace for delete, S for swap, and R for replace)
    • Allow cell tools to be triggered by button presses, keybinds, clicks, or Enter
    • Switch opacity sliders from range slider to single slider
    • Click on cell timeline to jump between frames
  • New functionality
    • Add divisions to export zip
    • Store spots in IndexedDB
    • Remove erased values before writing cells.json when completing an edit in label.py
    • Update divisions from EDITED_CELLS after editing segmentation
    • Enable flooding holes in cells by selecting the flood label as the select label
    • Code quality
    • Rename useSegment to useEditSegment
  • Bug fixes
    • Fix creating too many WebGL contexts by sharing one canvas and GPU instance for all kernels that need to use premultipliedAlpha. Note: this may cause a memory leak as we can’t call kernel.destroy() without affecting the shared canvas
    • Fix loading divisions as None
    • Fix watershed to create a new cell
    • Enable raw based tools when in grayscale mode
    • Prevent re-registering edit actor with undo
Finish merging into 0.4-dev @tddough98 (#332)

Something weird happened with what branches are merging into what. I think in #325, I forgot to delete the client-side-undo branch after merging into 0.4-dev, so the cascading PRs didn't get updated. This closes that loop and doesn't need any code review.

Create cells labels @tddough98 (#330)

This replaces the "overlap matrix" with a cells.json file of cell labels. Each cell label looks like

{ value: 1, cell: 2, z: 3 }

which says that the value 1 in frame 3 encodes cell 2. This is basically an association table (https://en.wikipedia.org/wiki/Associative_entity) for the many-to-many relationship between values and cells. Each value can encode many cells (so we can have overlapping cells at a pixel) and each cell can be encoded by many values.

With the additional z field, each frame has a separate association table between cells and labels. By adding the z field, we can reassign the cells on each frame without changing the values written in the segmentation images, allowing for very snappy client side editing of the cell assignments. Note: in #331, z changes to t.

Undo/redo has major changes so edits to the segmentation and the cell labels are propagated. Communicating changes between dependent label types and grouping them for undo/redo is key functionality to generalize to new label types. Edits to one category of labels must propagate to the labels that reference them so the dependent labels can update themselves. This manifests in #331, as divisions contain cells, so updating the cell labels needs to be reflected in the division labels.

Editing the segmentation can change the cells by adding or deleting values, so we send the cell labels in cells.json along with the segmentation images together and get updates for both the segmentation image and the cell labels.

Export edited labels @tddough98 (#327)

This PR closes the loop for editing labels on the dev branch by letting us exporting edited data. It provides two routes /api/download to download the edited labels to the users computer and /api/upload to upload them to an S3 bucket.

As the frontend cannot easily create a TIFF file, we send a zip with binary array data to the server which repackages these binary files into OME TIFF files that are more user friendly and compatible with other tools like Fiji.

Now that all the pieces are in place for a full data annotation cycle, I will put a demo instance online to test out all the new features. I still need to work more on editing the lineage on this branch before we can deploy a tracking job. Tracking labels can be loaded and displayed, but the editing is not functional yet.

Store project data in IndexedDB on client @tddough98 (#326)

On the main branch, every change is persisted in a database on the server as soon as it happens. On the dev branch ip, as the database no longer participates in label editing, we lose all the changes that happen in the browser. This PR persists the changes in IDB and checks for the project before loading it from the server so we can save and reload the local changes between sessions.

The core changes to review here are

  • the addition of idbMachine.js which open IDB and creates an object store for projects, checks for the current projectId in the store, and if it's there, loads its data instead of loading over the network
  • loadMachine has been folded into projectMachine so that loads over the network only after checking for the project in IDB. The Load and LoadReview components now just instantiate projectMachine instead of both projectMachine and loadMachine.

There a couple minor changes that are causing changes across many files like

  • renaming of rawArrays to raw and labeledArrays to labeled
  • reverting the sliders from 'small' to their default size. I found the small sliders too small to consistently grab with the cursor, and the small sliders don't save much space.

There are also a few bug fixes

  • fix for useMousetrapRef so that sliders with a range (two inputs on the same slider) get the mousetrap class added to the both of its input elements.
  • fix for useAlphaKernelCanvas to update the size of the canvases. When the canvas size was not updating, changes to the canvases were being computed but not drawn.
  • reduce the initial height and width in canvas machine to 1 (instead of 512) so the (x, y) coordinates of the cursor are always inside the labeled array once it loads.

Lastly, here's a improvement that I threw in too

  • LabeledCanvas now uses the opacity of non-selected labels to affect how much to screen the colors. When the non-selected labels are completely transparent, they do not affect the color of the selected label at all, while when they are fully opaque, there is standard screening composition. Screening is described here

The pixels are inverted, multiplied, and inverted again. A lighter picture is the result (opposite of multiply)

Add client side undo for labeled data @tddough98 (#325)

This is a first pass implementation of client side undo redo. It extends the past undo/redo management for UI state and adds history machine variant for labeled data.

It's a first pass implementation because it's missing a few must have and nice to have features. It currently only saves and restores label array and overlap array edits. Lineage editing is not yet functional in this branch, so it's not captured in undo. Client side persistence is also not functional, but once it's there, I hope to persist the undo history as well so that actions can be undone even after closing and reopening the page.

Send zip to edit route @tddough98 (#324)

Here we replace the form with multiple files sent to edit route with a single zip file of all the data created with zip.js. This compresses the data on the client before sending it over the network, reducing the network bandwidth. This also provides an extensible way to edit more labels on the server by adding another file to the zip sent to and from the server.

Merge overlapping labels branch @tddough98 (#323)

My work on overlapping labels occurred before getting the spots and caliban visualizers ready and online. As these visualizers are moving into deployment, I want to use the work in the visualizer branches as the core for the dev branch. Here, I branch off the caliban-visualizer branch and merge in the overlap-label-images branch and work out the issues to get them to play nice.

There are a few principal changes here

  1. introduce an overlaps array stored in overlaps.json
  2. refactoring label.py to be aware of overlapping cells when editing
  3. updating components and state machines to be aware of overlaps (canvases, interactive tools)

1. Loading overlaps.json

To implement overlapping labels, I add an overlaps.json file to the project archive that contains an overlaps array. The (i, j) element of the array is 1 if value i in a segmentation image encodes cell j. Value 0 should encode no cells and cell 0 is still treated as the background and should be encoded by no values.

There are some code snippets commented out that index the overlaps array by frame and feature. At the moment, we use the same overlaps array for the whole project, but it might be worth transitioning to a separate array for each frame if the array becomes unwieldy.

As overlaps array is sparse, we could improve the size of overlaps.json by only writing the positions where , but I leave it in the expanded form for now as both numpy and GPU.js use this expanded form when editing and displaying the image arrays.

2. Refactoring label.py to be aware of overlaps

Instead of writing a cell directly to the segmentation image, label.py now change the values in the segmentation image to add or remove a cell. I also introduce three writing modes:

  • overlap: add or remove a cell without affecting other cells that overlap the edited area
  • overwrite: remove all existing cells in the edited area and write one cell in their place
  • exclude: write a cell only where there are no other cells

There are some cute icons I made to represent these writing modes where there's a center circles that either overlaps with, covers, or goes under two other circles.

3. Make the frontend aware of overlaps

All the components that need to be aware of the overlapping labels use the useOverlaps hook. Canvas components provide this overlaps array to its GPU.js kernel functions. I also add some new canvases for watershed to show the seed coordinates and for flood to show what label is being flooded. The select tool now cycles through all the labels encoded at a position. Other tools check that value at a position encodes the selected cell before executing the action.

The selected labels system is simplified from a foreground & background label to just one label. Some actions, namely flood, still have a "background" label and it has its own canvas and state to show and manage the label that will be flooded.

Next steps

I'll be making a few followup PRs to this one to work towards implementing client side undo/redo including:

  • preparing and sending a zip file to the server instead of a form
  • adding lineage.json to the zip file when editing the lineage
Add visualizer for caliban @tddough98 (#322)

Similar to #317, this introduces a new build of DeepCell Label for visualizing results created by the caliban model on deepcell.org/predict. I also work through the changes

There are also changes to work towards editing lineage labels. Editing the tracks is not fully functional at the moment. However, as this does
I plan on addressing this together with overlapping labels, as editing overlapping labels and editing the lineage requires sending back multiple files in a ZIP. This approach is already mocked up in the overlapping labels branch, so by merging it in, I'll be able to extend it to included updated lineages as well.

The core pieces of work remaining before the full editing app can be used in production are

  • client side undo/redo
  • client side persistence of changes

The changes core to visualizing caliban output include

  • Load and write lineage.json to the project archive
  • Load lineage.json on the client
  • Add lineageMachine to manage the lineage
  • Replace useTracking hook with useLineage and useEditLineage
    • Accessing the lineage happens through useLineage
    • Editing the lineage happens through useEditLineage
  • Manage which cell is selected with lineageMachine
  • Introduce a useEditing hook that returns whether the app is a visualizer or the full editing build
  • Refactor components defined in Timeline.js into separate files to make them easier to work with and reuse

Some other changes included here are

  • Fix issue with proxying the flask server while developing by adding setupProxy.js instead of the proxy field in package.json
  • Improve adding mousetrap class to input elements with useMousetrapRef hook
  • Parameterize which S3 bucket project files are uploaded to with the S3_BUCKET environment variable
  • Remove an extra copy of FrameSlider.js

This build is hosted at tracks.deepcell.org. You submit caliban output to be visualized with the same form and API route as the spots visualizer

curl --location --request POST 'tracks.deepcell.org/api/project' \
--form 'images="URL_TO_DOWNLOAD_CALIBAN_INPUT"' \
--form 'labels="URL_TO_DOWNLOAD_CALIBAN_OUTPUT"'
Spots visualizer @tddough98 (#317)

Here we add a variant of DeepCell Label client to display the spots consumer output and rework the backend to load data into a standardized archive to be loaded by the client.

The visualizer is online at spots.deepcell.org. You can create a project by sending a form with two URLs. The form should have a images field with a download URL for a file with the raw image data and an optional labels field with a download URL for a zip of the labels. The spots.deepcell.org address hosts both the client and the server. Here's an example CURL command to create a project (links may expire):

curl --location --request POST 'spots.deepcell.org/api/project' \
--form 'images="https://storage.googleapis.com/deepcell-prod/uploads/2ebc4a3339b0fc260b8c4815a47888fd.png"' \
--form 'labels="https://drive.google.com/uc?export=download&id=1tRCvgcs_1MzZzVGJukgu1pQr-m2vrTs5"'

The request will return a project ID like l5lCB8ervg7C that can be viewed at spots.deepcell.org/project?projectId=l5lCB8ervg7C.

Here are the changes to visualize spots on the client:

  • Add a SpotsCanvas to display spots
  • Add a spotsMachine to manage the spots data and display controls
  • Add SpotsControls to change how spots are displayed
  • Use a REACT_APP_SPOTS_VISUALIZER environment variable feature flag so that the complete labeling app and the simplified spots viewer exist in the same code base

These are other significant changes to support spots:

  • Make other controls more compact
  • Create a new simplified backend that generates a zip from input files, uploads it to an S3 bucket, and serves the files to the client
  • Introduce a new loadMachine on the client to read this zip

There are also changes to codebase quality included in the mix like:

  • Add index.js to component folders to simplify the imports (e.g. changing Footer/Footer to Footer)
  • Rename QualityControl___ components to Review___
  • Combine useSelector hooks into one hook that compares results with equal from 'fast-deep-equal'
  • Remove a intermediate drawing canvas from components that don't need them
  • Separate components defined in App.js into their own files

As there are many changed files, here's a guide to the relevant files for each changes, excluding trivial changes (renaming, moving a file).

  • backend rework: bleuprints.py, models.py, loaders.py, labelmaker.py
  • testing new backend conftest.py, blueprints_test.py, models_test.py, loaders_test.py, labelmaker_test.py
  • end to end tests for Label: label.spec.js, rgb.zip, tests.yaml
  • e2e tests for Spots: spots.spec.js, spots.zip, tests.yaml
  • loading spots on client: package.json, loadMachine.ts, spotsMachine.js
  • displaying spots: SpotsCanvas.js, ProjectContext.js, canvasMachine.js,
  • controlling spots display: SpotColorToggle.js, SpotOpacitySlider.js, SpotOutlineToggle.js, SpotRadiusSlider.js, SpotsCheckbox.js, SpotsControls.js
  • refactoring App.js: InvalidId.js, Display.js, Load.js, LoadReview.js, Project.js,
Add imagecodecs to requirements @tddough98 (#318)

I have been seeing this error come up in the backend logs when users try to load tiffs from the Label homepage.

ValueError: <COMPRESSION.LZW: 5> requires the 'imagecodecs' package

This adds imagecodecs to the requirements for the backend. It also drops Python 3.7 from our testing matrix as the latest release of imagecodecs is for Python 3.8+ per this NEP (like PEP for numpy) https://numpy.org/neps/nep-0029-deprecation_policy.html. We now use Python 3.8 for hosting on Elastic Beanstalk, so this matches our use.

Load project data in bulk and decouple label image editing from database @tddough98 (#311)

This PR is a two-in-one. Earlier work on the PR focuses on how data loading into client. Previously, we loaded each frame in series and we now load all data for a project in bulk at the start of the application.

Loading data in bulk aligns with the longer term goal of decoupling the client from the server and removes a significant amount of complexity from state machines that used to load data (see imageMachine and its child/grandchild actors). Instead, we load and manage all array data in the arrayMachine and component that need the data access it through useArrays. We also introduce labelsMachine to manage the cell info, which will be the footprint for the state machines for the new label format. We don't use the new label format yet in this PR, but rather lay the groundwork to substitute the prototype state machines from other branches for the new label format.

Later work on the PR focuses on decoupling the Edit module on the server from the database, which also aligns with decoupling the client and server. Instead of telling the server which image to read from the database, we send the image itself and get back the updated image. By loading the array data in bulk, we can treat the client copy of the project data as the ground truth, so to edit and iterate on the labels, we need to send the current labels to an editing API and save the updates on the client.

This PR does break some key features, namely undo/redo and tracking. To return to feature parity, we'll need to

  • persist label image snapshots in IndexedDB on the client
  • reimplement tracking operations for the new label format

Each of these are significant development efforts that deserve dedicated effort and review, so I think it's better to review and merge this PR into the develop branch now and add back these features piecewise. I also want to call out some new directions/features that are highlighted/enabled by this PR

  • separating API calls from the state management tree so we can easily test the state tree by mocking the API
  • running tests with jest-electron
  • improving the label image editing API by sending and receiving zip files with zip.js
Use event buses to facilitate cross-actor message passing @tddough98 (#302)

This PR adds event buses to coordinate message passing between actors. Previously, message passing between siblings in the actor tree had to pass events up to the parent and the parent had to forward events to the appropriate children.

Event bus make message passing more flexible. They allow actors to broadcast events without knowing who need to receive the event and they enable two way communication.

This change declutters the message passing management previously in the root projectMachine and opens up future flexibility for message passing. We'll need to refine message passing as we move forward with the new label format, but the event bus pattern should be a good starting point.

The EventBus and fromEventBus source code and pattern is taken from xstate-behaviors, which recommends

This package is not yet release as it is in a very early alpha state. If you would like to use any of these helpers now its recommended to copy and paste them into your codebase for the time being!

Here are the state machines with an event bus, which events they send to the buses, and which machines use those events.

  • image
    • sends FRAME, FEATURE, and CHANNEL events
    • apiMachine uses these events append to outgoing EDIT events so the backend edit the correct image slice
  • labeled
    • sends LABELED_ARRAY and LABELS events
    • canvas uses LABELED_ARRAY to find the label under the cursor
    • select uses LABELS to cycle through labels
  • canvas
    • sends HOVERING, COORDINATES, mouseup, and mousedown events
    • toolMachine forwards mouseup and mousedown events to the active tool
    • tools (brush, threshold, flood, etc.) use HOVERING and COORDINATES to track the cursor
  • select
    • sends FOREGROUND, BACKGROUND, and SELECTED events
    • tools use these to know what labels to edit
  • raw
    • sends GRAYSCALE and COLOR events
    • toolMachine uses these to activate or deactivate grayscale specific tools (like threshold, autofit, and watershed)
  • api
    • sends EDITED events to labeled, so labeled knows to update it's data after an edit
    • tool machines send EDIT events to initiate an edit on the backend

The undo eventBus is an exception. The undoMachine does not send events to other actors on the event bus. Instead, the bus allows other machines to register actors in the undo/redo history with ADD_ACTOR events.

Here are some of places where I expect event buses will be useful for the new label format

  • defining additional label types
    • event buses let new labels to send & receive message to existing actors without the existing actors needing to know about the new label type
    • when editing one label, dependent labels need to receive the edited labels to know how to update themselves. An event bus lets the edited label broadcast changes to all actors listening to the bus
    • if a new label needs application state like the canvas position, the new labels can listen to the canvas bus without the canvas actor needing to know
  • undo/redo
    • undo/redo needs to capture all the changes that propagate from a single label change. If all changes are passed on the same event bus, undo/redo can listen to the event bus and know all the labels that changed
    • undo/redo needs to restore UI state across many actors. An event bus could ensure all actors that saved their state to be undone/redo receive messages to restore their state
End to end testing with Cypress @tddough98 (#314)

Here I add an e2e-tests step the the tests workflow that runs end-to-end tests with Cypress.

There are two true end-to-end tests, one for a tissue segmentation job and one for a tracking job, that picks an example file from the homepage, download the files on the backend, load them into the database, and serve them to the frontend, providing high value confidence that the full user workflows will work in production. True end-to-end tests are slower more expensive to run, so other tests instead use stubbed data to quickly load and interact with a page. We have near complete coverage of the React components, while most of the uncovered code is interactions defined in state machines. Future work can cover this code by interacting with components that dispatch events to state machines.

We measure code coverage on the frontend for these end-to-end tests and add that coverage to coveralls. At the moment, the Cypress tests do not capture backend code coverage. I started to explore running the flask backend with coverage. The flask backend is a long running process which needs to be terminated gracefully with SIGTERM to generate a .coverage file (https://coverage.readthedocs.io/en/latest/subprocess.html). As GitHub Actions manages the processes in each step, and the Cypress GHA runs the flask backend under, I couldn't find a way to send SIGTERM to the process before the Cypress step ended and terminated the process itself without generating a .coverage file. For now, we're losing out on measuring this backend coverage, but the end-to-end testing value is still as tests will fail if there are issues in the covered backend code.

Test React component tree and state management by mocking load events @tddough98 (#313)

Here I add a separate loadMachine to fetch data from the Flask backend. We separately create a state management tree in a waiting state and then send a LOADED event with project data once the loadMachine reaches its final state. By separating the API calls from the rest of the state management, we can adds tests that covers the full component tree by sending a mocked LOADED event to the state tree.

The end-to-end tests introduced in #314 provide a similar level of test coverage, with the added benefit of running the backend server as well and running in a real browser. Here, the tests are run by jest instead of cypress, which makes them more lightweight and easy to run often.

It's worth keeping both testing approaches in the codebase so we have the option to choose which we want to run, even if the Cypress tests are more powerful overall.

Add separate README with instructions for first time setup @tddough98 (#309)

I've added a separate README with detailed instructions on setting up Python and Node for first-time users. This adds details for users who have not used Python or Node before without adding more to the main README that is not as useful for experienced or repeat users. I also tweaked the language and order of some of the README.

Add yarn.lock @tddough98 (#307)

Yesterday I helped @msschwartz21 debug some issues with running Label locally, and I believe the issue came down to dependency mismatches. I've learned that for applications like Label, it is best practice to track yarn.lock in the repo as it ensures the exact version of dependencies are installed across machines.

Here's a stack overflow answer confirming that yarn.lock should be tracked for applications: https://stackoverflow.com/a/40206145/14458070

Here's an issue on the yarn repo as well: yarnpkg/yarn#1583

Simplify frontend testing matrix @tddough98 (#308)

There also have been some testing issues popping up in our frontend testing action. Node 14 & 16 started failing due to the Install Xi step, which was previously added because Node 17 did not have a version of node-gyp available to install and had to build it, which needed Xi. We only use node for testing, so having a complete testing matrix is overkill and node is not used anywhere in production. Cutting down the matrix to just one node version, v16 which is currently recommended for most users, speeds up the workflow, cuts down on the GitHub Actions minutes used, and avoids issues like libraries that are not available on every node version without losing any value from the test suite.

Fix quality control canvases @tddough98 (#301)

#294 added hooks useAlphaKernelCanvas and useDrawCanvas to encapsulate the logic to set up canvases. When doing QC with DCL (switching between multiple projects on the same page), these canvases were not switched out properly, leaving the canvas for the previous project on screen.

Here, we add the top level project machine to these dependencies of these hooks, so when the project switches, the canvases will be set up properly for the new project.

Upgrade to MUI v5 @tddough98 (#304)

Material-UI rebranded and updated to MUI v5 a few months ago. Here, I mostly follow their migration guide, then handled the breaking changes with styling by opting for their new sx prop. Here's an overview of how sx works in MUI v5: https://mui.com/system/the-sx-prop/.

These migration guide steps include

  • upgrading dependencies
  • adding a ThemeProvider at the root of the component tree
  • running their codemod

Afterwards, I fixed some layout issues that arose and handled the style conversion to using sx instead of useStyles.

One persistent issue is that the styling for disabled button in the toolbar and actions buttons is off. In order to show tooltips over disabled buttons, we switch out the button element for a div, but that interferes with how MUI styles the group, causing some minor issues with the button outlines. We can keep the styling issues, drop the tooltips for disabled buttons, or stop disabling tools/buttons by supporting watershed/thresholding/autofit when in RGB mode. For now, I left the styling issues as they cause not functional issues.

Set file upload size limit to 50 megabytes @tddough98 (#306)

As reported in #305, drag and drop is not working on the label.deepcell.org homepage due to the 413 errors reporting that attached files are too large.

I believe with our update to Amazon Linux 2 for the backend in #287, the behavior for nginx changed. With this new platform, the configuration looks a bit different/ With the previous platform, we would set up this file with a .config file but with Amazon Linux 2, this approach no longer works and instead we put the conf file in the .platform folder.

Remove node 12 from testing matrix @tddough98 (#303)

Our frontend testing suite has started failing on the yarn install step because Node 12 has reached its end of life. This PR drops node 12 from the testing matrix.

Fix minified brush kernel function @tddough98 (#300)

Similar to #299, the minified kernel functions for the brush canvas was causing breaking changes for the kernel compilation. This quick fix moves the brush kernel function into a template string and moves a custom function into a local declaration so that it cannot get renamed during minification.

label.deepcell.org has been updated with this fix and it can be tested there

🧰 Maintenance

Bump async from 2.6.3 to 2.6.4 in /visualizer @dependabot (#321)

Bumps async from 2.6.3 to 2.6.4.

Changelog

Sourced from async's changelog.

v2.6.4

  • Fix potential prototype pollution exploit (#1828)
Commits
Maintainer changes

This version was pushed to npm by hargasinski, a new releaser for async since your current version.


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Bump ejs from 3.1.6 to 3.1.7 in /visualizer @dependabot (#320)

Bumps ejs from 3.1.6 to 3.1.7.

Release notes

Sourced from ejs's releases.

v3.1.7

Version 3.1.7

Commits
  • 820855a Version 3.1.7
  • 076dcb6 Don't use template literal
  • faf8b84 Skip test -- error message vary depending on JS runtime
  • c028c34 Update packages
  • e4180b4 Merge pull request #629 from markbrouwer96/main
  • d5404d6 Updated jsdoc to 3.6.7
  • 7b0845d Merge pull request #609 from mde/dependabot/npm_and_yarn/glob-parent-5.1.2
  • 32fb8ee Bump glob-parent from 5.1.1 to 5.1.2
  • f21a9e4 Merge pull request #603 from mde/mde-null-proto-where-possible
  • a50e46f Merge pull request #606 from akash-55/main
  • Additional commits viewable in compare view

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Bump minimist from 1.2.5 to 1.2.6 in /visualizer @dependabot (#316)

Bumps minimist from 1.2.5 to 1.2.6.

Commits

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Bump node-forge from 1.2.1 to 1.3.0 in /visualizer @dependabot (#315)

Bumps node-forge from 1.2.1 to 1.3.0.

Changelog

Sourced from node-forge's changelog.

1.3.0 - 2022-03-17

Security

  • Three RSA PKCS#1 v1.5 signature verification issues were reported by Moosa Yahyazadeh (moosa-yahyazadeh@uiowa.edu).
  • HIGH: Leniency in checking digestAlgorithm structure can lead to signature forgery.
  • HIGH: Failing to check tailing garbage bytes can lead to signature forgery.
  • MEDIUM: Leniency in checking type octet.
    • DigestInfo is not properly checked for proper ASN.1 structure. This can lead to successful verification with signatures that contain invalid structures but a valid digest.
    • CVE ID: CVE-2022-24773
    • GHSA ID: GHSA-2r2c-g63r-vccr

Fixed

  • [asn1] Add fallback to pretty print invalid UTF8 data.
  • [asn1] fromDer is now more strict and will default to ensuring all input bytes are parsed or throw an error. A new option parseAllBytes can disable this behavior.
    • NOTE: The previous behavior is being changed since it can lead to security issues with crafted inputs. It is possible that code doing custom DER parsing may need to adapt to this new behavior and optional flag.
  • [rsa] Add and use a validator to check for proper structure of parsed ASN.1 RSASSA-PKCS-v1_5 DigestInfo data. Additionally check that the hash algorithm identifier is a known value from RFC 8017 PKCS1-v1-5DigestAlgorithms. An invalid DigestInfo or algorithm identifier will now throw an error.
    • NOTE: The previous lenient behavior is being changed to be more strict since it could lead to security issues with crafted inputs. It is possible that code may have to handle the errors from these stricter checks.

... (truncated)

Commits

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Bump follow-redirects from 1.14.7 to 1.14.8 in /visualizer @dependabot (#312)

Bumps follow-redirects from 1.14.7 to 1.14.8.

Commits
  • 3d81dc3 Release version 1.14.8 of the npm package.
  • 62e546a Drop confidential headers across schemes.
  • See full diff in compare view

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Contributors

msschwartz21, dependabot, and tddough98
Assets 2
Loading