Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make NML import more performant and resilient #4541

Merged
merged 17 commits into from
Apr 23, 2020
Merged

Conversation

daniel-wer
Copy link
Member

@daniel-wer daniel-wer commented Apr 9, 2020

  • Better performance, by keeping trees and groups mutable until put into the store (From ~5s to ~1.5s for this part for a 200,000 node tracing)
  • Allow missing tree names and assign a new tree name for missing and empty tree names
  • Align tree timestamp assignment to backend NML parser. Replace invalid timestamps with a default timestamp of 0.
  • Fixed that validation errors are shown for NML-like files
  • Split unconnected trees during NML import and wrap them in a group.

Ideally, I'd like to avoid the redefinition of the MutableTree type and, therefore, experimented with recursive read-only instructions for flow, but I couldn't get those to work for the diffable maps :/
(for reference, the most success I had was with the solution outlined in facebook/flow#5844 (comment))

URL of deployed dev instance (used for testing):

  • https://___.webknossos.xyz

Steps to test:

  • knossos_nmls.zip
  • Import these knossos NMLs in the tracing view.
    • Unconnected trees should be split
    • There should be no errors during saving
  • Try to import an NML where the trees have no name or the nodes have no attributes (except for position and id)
  • Check that validation errors are shown for NML-like files, e.g. by removing the id attribute of one of the nodes before upload.

Issues:


@daniel-wer daniel-wer self-assigned this Apr 9, 2020
@daniel-wer daniel-wer changed the title [WIP] Make NML import more performant and resilient Make NML import more performant and resilient Apr 20, 2020
Copy link
Member

@philippotto philippotto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Top notch work 🎉 I only left some minor performance ideas which should be easy to integrate.

/edit: Also tested the provided NMLs and everything worked fine 👍

frontend/javascripts/oxalis/model/helpers/nml_helpers.js Outdated Show resolved Hide resolved
frontend/javascripts/oxalis/model/helpers/nml_helpers.js Outdated Show resolved Hide resolved
frontend/javascripts/oxalis/model/helpers/nml_helpers.js Outdated Show resolved Hide resolved
@daniel-wer
Copy link
Member Author

Thanks for the review, I updated flow to a somewhat newer version, because the flatMap type definition was missing - fixed the new errors as good as possible. The newest flow version adds lots of new errors, so that'll be a project for another day ^^

@daniel-wer
Copy link
Member Author

@philippotto Could you have a look at my new changes and maybe test whether yarn flow test works for you locally with this version? :)

@philippotto
Copy link
Member

@philippotto Could you have a look at my new changes and maybe test whether yarn flow test works for you locally with this version? :)

After I upgraded node to v12, everything worked fine :) Maybe post a PSA in slack after this is merged?

@daniel-wer
Copy link
Member Author

Absolutely, will do, thanks for testing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Frontend NML parser: show error details in toast
2 participants