Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document our crate organization in ARCHITECTURE.md #4458

Merged
merged 6 commits into from
Dec 10, 2023
Merged

Document our crate organization in ARCHITECTURE.md #4458

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
daf1dd5
Added crate list and diagram to ARCHITECTURE.md
abey79 Dec 7, 2023
3f1a7c2
Add lint to ensure all crates are covered in ARCHITECTURE.md
abey79 Dec 7, 2023
fed2605
Update scripts/lint.py
abey79 Dec 7, 2023
38fb72d
Reorder sections
abey79 Dec 7, 2023
52fb906
Several improvement to lint:
abey79 Dec 7, 2023
61649d5
Add "interner" to cspell
abey79 Dec 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 136 additions & 18 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,6 @@ The web viewer can load `.rrd` files (just drag-drop them into the browser), or
NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version.


## Crates
In order to get an overview of our in-house crates and how they depend on each other, we recommend you run:

```
cargo install cargo-depgraph
cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
open deps.png
```

and:

```
cargo doc --no-deps --open
```

and then browse through the `re_*` crates.


## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
Expand Down Expand Up @@ -99,3 +81,139 @@ Immediate mode is also a forcing function, forcing us to relentlessly optimize o
This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.

Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.


## Crates

Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!

This image must be updated each time a crate is added/removed/updated.

FigJam document: https://www.figma.com/file/Umob8ztK1HmYKLUMSq8aPb/Crates-org

Update instructions:
1) Update the FigJame document
2) Select all -> right-click -> Copy as PNG
3) `just upload --name crates`
4) Copy/paste the resulting HTML
-->

### SDK/CLI/Wasm top-level crates

| Crate | Description |
|-----------|--------------------------------------|
| rerun-cli | Rerun native CLI binary crate |
| rerun | Rerun Rust SDK and viewer shim crate |
| rerun_c | Rerun C SDK |
| rerun_py | Rerun Python SDK |
| re_sdk | Rerun logging SDK |


### Viewer crates

##### UI crates

| Crate | Description |
|-----------------------------|----------------------------------------------------------------------------------------|
| re_viewer | The Rerun viewer |
| re_viewport | The central viewport panel of the Rerun viewer. |
| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |
| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |
| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |
| re_ui | Rerun GUI theme and helpers, built around egui |
| re_renderer | A wgpu-based renderer for all your visualization needs. |
| re_space_view | Types & utilities for defining Space View classes and communicating with the Viewport. |
| re_space_view_bar_chart | A Space View that shows a single bar chart. |
| re_space_view_spatial | Space Views that show entities in a 2D or 3D spatial relationship. |
| re_space_view_tensor | A Space View dedicated to visualizing tensors with arbitrary dimensionality. |
| re_space_view_text_document | A simple Space View that shows a single text box. |
| re_space_view_text_log | A Space View that shows text entries in a table and scrolls with the active time. |
| re_space_view_time_series | A Space View that shows plots over Rerun timelines. |


### Application-level store

| Crate | Description |
|-----------------|-----------------------------------------------------------------|
| re_data_store | In-memory storage of Rerun log data, indexed for fast queries. |
| re_query | Querying data in the re_arrow_store |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |


### Low-level store

| Crate | Description |
|----------------|-----------------------------------------------------------------------------|
| re_arrow_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |


### Data flow

| Crate | Description |
|----------------------|--------------------------------------------------------------------------------------------------------|
| re_sdk_comms | TCP communication between Rerun SDK and Rerun Server |
| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |
| re_ws_comms | WebSocket communication library (encoding, decoding, client, server) between a Rerun server and viewer |
| re_data_source | Handles loading of Rerun data |


### Build support

| Crate | Description |
|----------------------------|---------------------------------------------------------------|
| re_build_info | Information about the build. Use together with re_build_tools |
| re_build_tools | build.rs helpers for generating build info |
| re_types_builder | Generates code for Rerun's SDKs from flatbuffers definitions. |
| re_build_examples | Build rerun example RRD files |
| re_build_examples_manifest | Build the rerun examples manifest JSON file |
| re_build_web_viewer | Build the rerun web-viewer Wasm from source |


### Utilities

| Crate | Description |
|--------------------|--------------------------------------------------------------------------------------|
| re_analytics | Rerun's analytics SDK |
| re_log | Helpers for setting up and doing text logging in the Rerun crates. |
| re_error | Helpers for handling errors. |
| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |
| re_tuid | 128-bit Time-based Unique Identifier |
| re_string_interner | Yet another string interning library |
| re_tracing | Helpers for tracing/spans/flamegraphs and such. |
| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |
| re_smart_channel | A channel that keeps track of latency and queue length. |
| re_int_histogram | A histogram with `i64` keys and `u32` counts, supporting both sparse and dense uses. |
| re_memory | Run-time memory tracking and profiling. |



### Dependencies and docs

In order to get a dependency graph for our in-house crates and their docs, we recommend you run:

```
cargo install cargo-depgraph
cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
open deps.png
```

and:

```
cargo doc --no-deps --open
```

and then browse through the `re_*` crates.
1 change: 1 addition & 0 deletions docs/cspell.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@
"icosphere",
"imgmsg",
"impls",
"interner",
"intrinsics",
"ipynb",
"ipython",
Expand Down
43 changes: 42 additions & 1 deletion scripts/lint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
import os
import re
import sys
from typing import Any, Iterator
from pathlib import Path
from typing import Any, Callable, Iterator

import frontmatter
from gitignore_parser import parse_gitignore
Expand Down Expand Up @@ -748,6 +749,43 @@ def lint_file(filepath: str, args: Any) -> int:
return num_errors


def lint_crate_docs(should_ignore: Callable[[Any], bool]) -> int:
"""Make sure ARCHITECTURE.md talks about every single crate we have."""

crates_dir = Path("crates")
architecture_md_file = Path("ARCHITECTURE.md")

architecture_md = architecture_md_file.read_text()

# extract all crate names ("re_...") from ARCHITECTURE.md to ensure they actually exist
listed_crates: dict[str, int] = {}
for i, line in enumerate(architecture_md.split("\n"), start=1):
for crate_name in re.findall(r"\bre_\w+", line):
if crate_name not in listed_crates:
listed_crates[crate_name] = i

error_count = 0
for cargo_toml in crates_dir.glob("**/Cargo.toml"):
crate = cargo_toml.parent
crate_name = crate.name

if crate_name in listed_crates:
del listed_crates[crate_name]

if should_ignore(crate):
continue

if not re.search(r"\b" + crate_name + r"\b", architecture_md):
print(f"{architecture_md_file}: missing documentation for crate {crate.name}")
error_count += 1

for crate_name, line_nr in sorted(listed_crates.items(), key=lambda x: x[1]):
print(f"{architecture_md_file}:{line_nr}: crate name {crate_name} does not exist")
error_count += 1

return error_count


def main() -> None:
# Make sure we are bug free before we run:
test_lint_line()
Expand Down Expand Up @@ -832,6 +870,9 @@ def main() -> None:
continue
num_errors += lint_file(filepath, args)

# Since no files have been specified, we also run the global lints.
num_errors += lint_crate_docs(should_ignore)

if num_errors == 0:
print(f"{sys.argv[0]} finished without error")
sys.exit(0)
Expand Down
Loading