Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DataLoaders 5: add support for external binary DataLoaders (PATH) #4521

Merged
merged 14 commits into from Dec 15, 2023
Merged

DataLoaders 5: add support for external binary DataLoaders (PATH) #4521

merged 14 commits into from Dec 15, 2023

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Dec 13, 2023 •
edited by github-actions bot

Turn any executable in your PATH into a DataLoader by logging data to stdout.

Includes examples for all supported languages.

image

Checks:

  • rerun-loader-rust-file just prints help message
  • rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs successfully does nothing
  • rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs | rerun - works
  • rerun-loader-python-file just prints help message
  • rerun-loader-python-file examples/python/external_data_loader/main.py successfully does nothing
  • rerun-loader-python-file examples/python/external_data_loader/main.py | rerun - works
  • rerun-loader-cpp-file just prints help message
  • rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp successfully does nothing
  • rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp | rerun - works
  • cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets
  • Native: File > Open > examples/assets/
  • Native: Drag-n-drop > examples/assets/

Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
    • Full build: app.rerun.io
    • Partial build: app.rerun.io - Useful for quick testing when changes do not affect examples in any way
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

@teh-cmc teh-cmc added 📺 re_viewer affects re_viewer itself do-not-merge Do not merge this PR include in changelog CLI Related to the Rerun CLI labels Dec 13, 2023
@teh-cmc teh-cmc changed the base branch from main to cmc/dataloaders_4_folders December 13, 2023 19:04
@teh-cmc teh-cmc changed the title DataLoaders 4: add support for external binary DataLoaders (PATH) DataLoaders 5: add support for external binary DataLoaders (PATH) Dec 13, 2023
This was referenced Dec 14, 2023
Merged
Merged
Merged
Merged
Merged
@teh-cmc teh-cmc force-pushed the cmc/dataloaders_5_external branch 2 times, most recently from ce95b4d to 788acd1 Compare December 14, 2023 14:17
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 14, 2023 •
edited

Size changes

Name cmc/dataloaders_4_folders 4521/merge Change
detect_and_track_objects.rrd (none) 55.21 MiB +100%
plots.rrd 204.80 kiB 194.56 kiB -5.00%

@teh-cmc teh-cmc force-pushed the cmc/dataloaders_5_external branch 2 times, most recently from 14e7ea7 to 3e4babe Compare December 14, 2023 17:02
@teh-cmc teh-cmc marked this pull request as ready for review December 14, 2023 17:57
@teh-cmc
Copy link
Member Author

teh-cmc commented Dec 15, 2023

I forgot to update the DataLoader docstring to link the new loader, will do so in the docs/howto guide PR.

emilk
emilk approved these changes Dec 15, 2023
View reviewed changes
crates/re_data_source/src/data_loader/loader_external.rs Outdated Show resolved Hide resolved
crates/re_data_source/src/data_loader/loader_external.rs Show resolved Hide resolved
examples/rust/external_data_loader/src/main.rs Outdated Show resolved Hide resolved
crates/re_data_source/src/load_file.rs Outdated Show resolved Hide resolved
crates/re_data_source/src/data_loader/loader_external.rs
@@ -0,0 +1,188 @@
use std::io::Read;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should document somewhere what we expect from a loader, eg in a //!-level docstring

In particular:

  • what arguments to expect (--recording-id)
  • what exit code to use for unsupported file types
  • what and where to output (.rrd data to stdout)

This should also be added as a how-to guide on rerun.io/docs!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have an upcoming PR that adds a web guide, I'll add a link to that guide in the doc string then

teh-cmc added a commit that referenced this pull request Dec 15, 2023
@teh-cmc
DataLoaders 0: utility for hierarchical EntityPath from file path (…
5a93347 
…#4516)

A trivial PR that I extracted into its own in case this ends up being
controversial.

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521
teh-cmc added a commit that referenced this pull request Dec 15, 2023
@teh-cmc
DataLoaders 1: introduce, and migrate to, DataLoaders (#4517)
b51df6a 
**COMMIT PER COMMIT!**

_That's the nasty one... there's always a nasty one 😬_ 

Introduces the `DataLoader` trait, which can load any file, either from
a filepath by reading from the local filesystem, or from a pre-loaded
file's contents.

`DataLoader`s are run in parallel where possible, and have complete say
over what data they are interested in or not (i.e. they are not
registered based on an extension, which is very limited in practice).

I've decided to commit small example assets for the types we support,
cause I feel we really need a fast way of checking whether everything
still works from time to time.
It's pretty light but might be controversial, I figure.

Checks:
- [x] `cargo r -p rerun-cli --no-default-features --features
native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd}`
- [x] Native: `File > Open > examples/assets/*`
- [x] Native: `Drag-n-drop > examples/assets/*`
- [x] Web: `File > Open > examples/assets/*`
- [x] Web: `Drag-n-drop > examples/assets/*`

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- TODO: register custom loaders
- TODO: high level docs and guides for everything related to loading
files
teh-cmc added a commit that referenced this pull request Dec 15, 2023
@teh-cmc
DataLoaders 2: add text-based DataLoader (.txt, .md) (#4518)
8f4f4e9 
What the title says.


![image](https://github.com/rerun-io/rerun/assets/2910679/68f2e499-f4df-4e75-95f6-0ed7f479c5e6)


Checks:
- [x] `cargo r -p rerun-cli --no-default-features --features
native_viewer --
examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md}`
- [x] Native: `File > Open > examples/assets/*`
- [x] Native: `Drag-n-drop > examples/assets/*`
- [x] Web: `File > Open > examples/assets/*`
- [x] Web: `Drag-n-drop > examples/assets/*`

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- TODO: register custom loaders
- TODO: high level docs and guides for everything related to loading
files
teh-cmc added a commit that referenced this pull request Dec 15, 2023
@teh-cmc
DataLoaders 3: add 3D point cloud DataLoader (.ply) (#4519)
82f17dc 
`.ply` support, but point clouds only, at least for now.

No feature flag: I had some, and tbh it only made thing needlessly hard
to work with for very little gains.
`.ply` are so common anyway, they always come up in user requests.

Here's an example from
[there](https://github.com/HuangCongQing/Point-Clouds-Visualization/blob/master/2open3D/data/fragment.ply):

![image](https://github.com/rerun-io/rerun/assets/2910679/8242c5a2-037d-46c6-9816-ab9b240a7473)


Checks:
- [x] `cargo r -p rerun-cli --no-default-features --features
native_viewer --
examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md,ply}`
- [x] Native: `File > Open > examples/assets/*`
- [x] Native: `Drag-n-drop > examples/assets/*`
- [x] Web: `File > Open > examples/assets/*`
- [x] Web: `Drag-n-drop > examples/assets/*`

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- TODO: register custom loaders
- TODO: high level docs and guides for everything related to loading
files
Base automatically changed from cmc/dataloaders_4_folders to main December 15, 2023 14:09
teh-cmc added a commit that referenced this pull request Dec 15, 2023
@teh-cmc
DataLoaders 4: add generic folder DataLoader (#4520)
5931809 
Load folders filled with whatever. Go nuts.

The loader has no particular semantics (e.g. HIVE partitioning): it just
loads everything in a folder using the appropriate loader(s) for each
file.

Checks:
- [x] `cargo r -p rerun-cli --no-default-features --features
native_viewer -- examples/assets`
- [x] Native: `File > Open > examples/assets/`
- [x] Native: `Drag-n-drop > examples/assets/`

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- TODO: register custom loaders
- TODO: high level docs and guides for everything related to loading
files
@teh-cmc teh-cmc removed the do-not-merge Do not merge this PR label Dec 15, 2023
@teh-cmc teh-cmc merged commit 8ebaf8d into main Dec 15, 2023
43 of 44 checks passed
@teh-cmc teh-cmc deleted the cmc/dataloaders_5_external branch December 15, 2023 16:37
This was referenced Dec 17, 2023
Merged
Merged
Merged
Closed
Closed
Closed
teh-cmc added a commit that referenced this pull request Dec 19, 2023
@teh-cmc
DataLoaders 6: first-class support for NotSupported (#4565)
b7f7a55 
Introduce a first-class protocol for `DataLoader`s -- whether they are
builtin, custom, or external -- to announce that they don't support
loading a given piece of data.

This fixes one of the oldest issue when loading files in Rerun: loading
unsupported data would always bottom out in either the image or
websocket paths, leading to unrelated errors very confusing to users.

The loading process is now a two-pass process:
1. Dispatch the data to be loaded to all loaders, which will respond
ASAP whether they can do it or not.
1.1. If at least one compatible loader is found, go to 2
1.2. Otherwise, fail early with a nice error message for the end user
2. Dispatch the actual data loading.

This has important/subtle ramifications on the threading model, but
other than that is what you'd expect.


Checks:
- [x] Native: CLI examples/assets/*
- [x] Native: CLI examples/assets
- [x] Native: CLI examples/assets/* containing unsupported files
- [x] Native: CLI examples/assets containing unsupported files
- [x] Native: File>Open examples/assets/*
- [x] Native: File>Open examples/assets
- [x] Native: File>Open examples/assets/* containing unsupported files
- [x] Native: File>Open examples/assets containing unsupported files
- [x] Native: Drag-n-drop examples/assets/*
- [x] Native: Drag-n-drop examples/assets
- [x] Native: Drag-n-drop examples/assets/* containing unsupported files
- [x] Native: Drag-n-drop examples/assets containing unsupported files
- [x] Web: File>Open examples/assets/*
- [x] Web: Drag-n-drop examples/assets/*
- [x] Web: File>Open examples/assets/* containing unsupported files
- [x] Web: Drag-n-drop examples/assets/* containing unsupported files



---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- #4565
- #4566
- #4567
teh-cmc added a commit that referenced this pull request Dec 19, 2023
@teh-cmc
DataLoaders 7: support for custom DataLoaders (#4566)
47bec77 
Add support for registering custom loaders, the same way we offer
support for registering custom space views and store subscribers.

Checks:
- [x] cargo r -p custom_data_loader -- Cargo.toml

---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- #4565
- #4566
- #4567
teh-cmc added a commit that referenced this pull request Dec 22, 2023
@teh-cmc @nikolausWest
DataLoaders 8: docs, guide, etc (#4567)
d72a4c9 
Some docs for the whole thing.

It heavily relies on linking to the examples for exact details and to
avoid rot.

It's probably extremely bad. Help welcome.


---

Part of a series of PRs to make it possible to load _any_ file from the
local filesystem, by any means, on web and native:
- #4516
- #4517 
- #4518 
- #4519 
- #4520 
- #4521 
- #4565
- #4566
- #4567

Co-authored-by: Nikolaus West <niko@rerun.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@emilk emilk emilk approved these changes

Assignees
No one assigned
Labels
CLI Related to the Rerun CLI include in changelog 📺 re_viewer affects re_viewer itself
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@teh-cmc @emilk