Docs codegen #3445

jprochazk · 2023-09-25T10:48:39Z

What

Add docs as a new "language" in codegen
Replace old docs with new ones
Minor output polish work
- Components and APIs section should say "Required Components" instead of just "Required" (for all kinds)
- Use actual headings for example titles
- Use images even in .rs/.py files

Follow-up work: #3503

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)

### What * Fixes #2461 The new link format is `https://static.rerun.io/{original_image_name_minus_extension}/{content_hash}/{width}.{ext}` For example, `https://static.rerun.io/ae4b8970edba8480431cb71e57b8cddd9e1769c7_clock_480w.png` is now `https://static.rerun.io/clock/ae4b8970edba8480431cb71e57b8cddd9e1769c7/480w.png`. What makes this more discoverable: - For machines, it's the fact that the hash is the same for all sizes. This means given an image URL, you can retrieve all the other sizes. This is important for #3445. - For humans, the top-level directory is the name of the image, which makes it _much_ easier to browse through GCS looking for that image. You can compare the `updated at` date of each hash to find the latest one. `upload_image.py` has been updated to use the new format. We're nowhere near the point of using too much space on GCS, so all the old links are preserved. This PR also fixes some other issues: - `thumbnails.py` did not handle updating `thumbnail_dimensions` correctly if the property was already present in the frontmatter - `upload_image.py` would fail if the user does not have `delete` permission when uploading the same image twice, because it would attempt to overwrite the existing image. if the content hash does not change, then we do not need to upload the image again, so we just skip uploading to GCS if the object already exists. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3477) (if applicable) - [PR Build Summary](https://build.rerun.io/pr/3477) - [Docs preview](https://rerun.io/preview/67a25d43432ddf6245f04ca52806fd976a4dac4e/docs)  - [Examples preview](https://rerun.io/preview/67a25d43432ddf6245f04ca52806fd976a4dac4e/examples)  - [Recent benchmark results](https://ref.rerun.io/dev/bench/) - [Wasm size tracking](https://ref.rerun.io/dev/sizes/)

nikolausWest · 2023-09-27T20:43:56Z

Should probably just say "Components" in the heading.

Could we make this section tighter? Perhaps using a table or something like

Components

Required: position3d
Recommended: color, radius
Optional: text, class_id, keypoint_id, instance_key

emilk

Awesome!

crates/re_types/build.rs

crates/re_types/src/lib.rs

crates/re_types_builder/src/codegen/common.rs

crates/re_types_builder/src/codegen/docs/mod.rs

crates/re_types_builder/src/codegen/python.rs

emilk · 2023-09-28T06:26:14Z

crates/re_types_builder/src/codegen/rust/api.rs

+            if let Some(title) = example.base.title {
+                lines.push(format!(" ### {title}"));
+            }
+            lines.push(" ```ignore".into());


Why ignore? All the code examples should properly parse.

no_run should work just as well

Suggested change

lines.push(" ```ignore".into());

lines.push(" ```no_run".into());

ignore sounds correct to me.

All of these are already compiled and ran as part of the code_examples crate; recompiling them here would only slow down the CI for very little benefit?

fair; I guess we don't need the extra sanity check

emilk · 2023-09-28T06:27:51Z

scripts/upload_image.py

@@ -58,6 +58,8 @@
 from google.cloud import storage
 from PIL.Image import Image, Resampling

+# NOTE: We depend on the stack using the exact sizes they do right now in:
+#       - docs codegen (`image_url_stack`)


Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>

nikolausWest · 2023-09-28T09:16:54Z

We also need to remove the old archetype pages and update the index page as well

jprochazk · 2023-09-28T09:45:35Z

I updated the index page, ~~but I think we should do any potential iterations over the docs in a separate PR, because this one is already way too large~~

I see you already opened #3504 - I'll just merge this then once CI passes

jprochazk added 2 commits September 25, 2023 11:29

fix \examples not outputting anything in cpp

f94b152

support \example args with image + actually generate docs

08608cf

jprochazk mentioned this pull request Sep 26, 2023

New image link format #3477

Merged

3 tasks

Merge branch 'main' into jan/codegen-docs

879818e

jprochazk added 📖 documentation Improvements or additions to documentation 🧑‍💻 dev experience developer experience (excluding CI) codegen/idl include in changelog labels Sep 27, 2023

jprochazk added 10 commits September 27, 2023 12:45

generate image stacks

24df56a

unrelated diff

6c20cf4

update annotation_context

271c4ab

remove affix_fuzzer

62e6254

add images to fbs files from existing docs

618af58

surround component names in backticks

3cd6ecf

remove unnecessary blank lines

7e2a9dc

fix lint

5cce931

fix spelling

418e1e3

Merge branch 'main' into jan/codegen-docs

d3013fc

jprochazk marked this pull request as ready for review September 27, 2023 11:24

temporarily copy data_types top-level docs

9aa0410

jprochazk marked this pull request as draft September 27, 2023 12:17

jprochazk added 8 commits September 27, 2023 21:23

update doc output dir + titles

1539b2c

update old docs

f1db4ea

use proper headings for example titles in rs/cpp

adbb7d3

update codegen

47c1123

update docs + delete data_types_test

2efc288

remove pointless alt text (doesnt describe the image)

f49d064

show images in rs and py output

7fc84c3

update codegen

cc103c3

jprochazk mentioned this pull request Sep 27, 2023

Generate documentation for components/data types #3503

Closed

emilk approved these changes Sep 28, 2023

View reviewed changes

jprochazk and others added 19 commits September 28, 2023 09:33

docstring join macro

24d6bea

update components section output

bad9696

Update crates/re_types/src/lib.rs

d6dc1e8

Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>

Update crates/re_types_builder/src/codegen/common.rs

539d960

Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>

Update crates/re_types_builder/src/codegen/common.rs

64590cc

Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>

Example.content -> lines

f77496b

remove commented out code

7c63393

move required value into variable

dd69cb3

remove trailing whitespace from docstring line

21e399e

use ok() instead of let _ =

a4960dd

use helper for testing archetype detection

693e8b0

quote title in frontmatter

98e55e7

components_and_apis -> write_archetype_fields

875b628

use PascalCase component names

21f9bce

remove commented out code

77048e7

examples_list -> write_examples_list + handle plural section title

fe6f0a0

Merge branch 'main' into jan/codegen-docs

0953d10

use :? instead of manual quotes

4fd889e

Merge branch 'main' into jan/codegen-docs

19a0201

jprochazk added 3 commits September 28, 2023 11:22

remove old archetype docs

311e49f

fix links on data_types index

cb70526

update broken links + index page

3bd57da

jprochazk requested a review from nikolausWest September 28, 2023 09:42

jprochazk merged commit 65bad7e into main Sep 28, 2023
27 checks passed

jprochazk deleted the jan/codegen-docs branch September 28, 2023 09:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs codegen #3445

Docs codegen #3445

jprochazk commented Sep 25, 2023 •

edited by github-actions bot

nikolausWest commented Sep 27, 2023

emilk left a comment

emilk Sep 28, 2023

teh-cmc Sep 28, 2023

emilk Sep 28, 2023

emilk Sep 28, 2023

nikolausWest commented Sep 28, 2023

jprochazk commented Sep 28, 2023 •

edited

	lines.push(" ```ignore".into());
	lines.push(" ```no_run".into());

Docs codegen #3445

Docs codegen #3445

Conversation

jprochazk commented Sep 25, 2023 • edited by github-actions bot

What

Checklist

nikolausWest commented Sep 27, 2023

Components

emilk left a comment

Choose a reason for hiding this comment

emilk Sep 28, 2023

Choose a reason for hiding this comment

teh-cmc Sep 28, 2023

Choose a reason for hiding this comment

emilk Sep 28, 2023

Choose a reason for hiding this comment

emilk Sep 28, 2023

Choose a reason for hiding this comment

nikolausWest commented Sep 28, 2023

jprochazk commented Sep 28, 2023 • edited

jprochazk commented Sep 25, 2023 •

edited by github-actions bot

jprochazk commented Sep 28, 2023 •

edited