Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DataLoaders 7: support for custom DataLoaders #4566

Merged
merged 5 commits into from
Dec 19, 2023
Merged

DataLoaders 7: support for custom DataLoaders #4566

merged 5 commits into from
Dec 19, 2023

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Dec 17, 2023 •
edited by github-actions bot
Loading

Add support for registering custom loaders, the same way we offer support for registering custom space views and store subscribers.

Checks:

  • cargo r -p custom_data_loader -- Cargo.toml

Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
    • Full build: app.rerun.io
    • Partial build: app.rerun.io - Useful for quick testing when changes do not affect examples in any way
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

@teh-cmc teh-cmc added 📺 re_viewer affects re_viewer itself include in changelog labels Dec 17, 2023
@teh-cmc teh-cmc changed the title DataLoaders 6: support for custom DataLoaders DataLoaders 7: support for custom DataLoaders Dec 17, 2023
This was referenced Dec 17, 2023
Merged
Merged
Closed
Closed
Closed
@teh-cmc teh-cmc added the do-not-merge Do not merge this PR label Dec 17, 2023
@emilk emilk self-requested a review December 19, 2023 10:39
emilk
emilk approved these changes Dec 19, 2023
View reviewed changes
examples/rust/custom_data_loader/src/main.rs Outdated Show resolved Hide resolved
examples/rust/custom_data_loader/src/main.rs Outdated Show resolved Hide resolved
Base automatically changed from cmc/dataloaders_6_unsupported_support to main December 19, 2023 18:50
teh-cmc added a commit that referenced this pull request Dec 19, 2023
@teh-cmc
DataLoaders 6: first-class support for NotSupported (#4565)
b7f7a55 
Introduce a first-class protocol for `DataLoader`s -- whether they are
builtin, custom, or external -- to announce that they don't support
loading a given piece of data.

This fixes one of the oldest issue when loading files in Rerun: loading
unsupported data would always bottom out in either the image or
websocket paths, leading to unrelated errors very confusing to users.

The loading process is now a two-pass process:
1. Dispatch the data to be loaded to all loaders, which will respond
ASAP whether they can do it or not.
1.1. If at least one compatible loader is found, go to 2
1.2. Otherwise, fail early with a nice error message for the end user
2. Dispatch the actual data loading.

This has important/subtle ramifications on the threading model, but
other than that is what you'd expect.


Checks:
- [x] Native: CLI examples/assets/*
- [x] Native: CLI examples/assets
- [x] Native: CLI examples/assets/* containing unsupported files
- [x] Native: CLI examples/assets containing unsupported files
- [x] Native: File>Open examples/assets/*
- [x] Native: File>Open examples/assets
- [x] Native: File>Open examples/assets/* containing unsupported files
- [x] Native: File>Open examples/assets containing unsupported files
- [x] Native: Drag-n-drop examples/assets/*
- [x] Native: Drag-n-drop examples/assets
- [x] Native: Drag-n-drop examples/assets/* containing unsupported files
- [x] Native: Drag-n-drop examples/assets containing unsupported files
- [x] Web: File>Open examples/assets/*
- [x] Web: Drag-n-drop examples/assets/*
- [x] Web: File>Open examples/assets/* containing unsupported files
- [x] Web: Drag-n-drop examples/assets/* containing unsupported files



---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- #4565
- #4566
- #4567
@teh-cmc teh-cmc removed the do-not-merge Do not merge this PR label Dec 19, 2023
@teh-cmc teh-cmc merged commit 47bec77 into main Dec 19, 2023
40 of 41 checks passed
@teh-cmc teh-cmc deleted the cmc/dataloaders_7_custom branch December 19, 2023 19:00
teh-cmc added a commit that referenced this pull request Dec 22, 2023
@teh-cmc @nikolausWest
DataLoaders 8: docs, guide, etc (#4567)
d72a4c9 
Some docs for the whole thing.

It heavily relies on linking to the examples for exact details and to
avoid rot.

It's probably extremely bad. Help welcome.


---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- #4565
- #4566
- #4567

Co-authored-by: Nikolaus West <niko@rerun.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emilk emilk emilk approved these changes

Assignees
No one assigned
Labels
include in changelog 📺 re_viewer affects re_viewer itself
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@teh-cmc @emilk