Skip to content

Commit

Permalink
Make codegen I/O-free and agnostic to output location (#3888)
Browse files Browse the repository at this point in the history
**Commit by commit**

This is necessary refactoring work for the upcoming
`attr.rust.custom_crate` attribute, itself necessary for the upcoming
serde-codegen support, itself necessary for the upcoming blueprint
experimentations as well as #3741.

### Changes

1. The `CodeGenerator` trait as well as all post-processing helpers
(gitattributes, orphan detection...) are now I/O-free.
  ```rust
pub type GeneratedFiles =
std::collections::BTreeMap<camino::Utf8PathBuf, String>;
  
  pub trait CodeGenerator {
      fn generate(
          &mut self,
          reporter: &crate::Reporter,
          objects: &crate::Objects,
          arrow_registry: &crate::ArrowRegistry,
      ) -> GeneratedFiles;
  }
  ```
2. All post-processing helpers are now agnostic to the location output.
This is very important as it makes it possible to generate e.g. rust
code out of the `re_types` crate without everything crumbling down.
A side-effect is that gitattributes files are now finer-grained.
3. The Rust codegen pass is now crate agnostic: it is driven by the
workspace path rather than a specific crate path.
Necessary for the upcoming `attr.rust.custom_crate`.
4. All codegen passes now follow the exact same 4-step structure:
  ```
  // 1. Generate in-memory code files.
  let mut gen = MyGenerator::new();
  let mut files = gen.generate(reporter, objects, arrow_registry);
  // 2. Generate in-memory attribute files.
  generate_gitattributes_for_generated_files(&mut files);
  // 3. Write all in-memory files to disk.
  write_files(&gen.pkg_path, &gen.testing_pkg_path, &files);
  // 4. Remove orphaned files.
  crate::codegen::common::remove_orphaned_files(reporter, &files);
  ```
5. The documentation codegen pass now removes its orphans, which is why
some `md` files were removed in this PR.

---

- Unblocks #3741 
- Unblocks #3495
  • Loading branch information
teh-cmc committed Oct 17, 2023
1 parent cbbe859 commit a364405
Show file tree
Hide file tree
Showing 45 changed files with 1,063 additions and 1,328 deletions.
133 changes: 0 additions & 133 deletions crates/re_types/.gitattributes

This file was deleted.

27 changes: 27 additions & 0 deletions crates/re_types/src/archetypes/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions crates/re_types/src/blueprint/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

36 changes: 36 additions & 0 deletions crates/re_types/src/components/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

34 changes: 34 additions & 0 deletions crates/re_types/src/datatypes/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions crates/re_types/src/testing/archetypes/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion crates/re_types/src/testing/blueprint/mod.rs

This file was deleted.

25 changes: 25 additions & 0 deletions crates/re_types/src/testing/components/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions crates/re_types/src/testing/datatypes/.gitattributes

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 1 addition & 3 deletions crates/re_types_builder/src/bin/build_re_types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ const DEFINITIONS_DIR_PATH: &str = "crates/re_types/definitions";
const ENTRYPOINT_PATH: &str = "crates/re_types/definitions/rerun/archetypes.fbs";
const DOC_EXAMPLES_DIR_PATH: &str = "docs/code-examples";
const CPP_OUTPUT_DIR_PATH: &str = "rerun_cpp";
const RUST_OUTPUT_DIR_PATH: &str = "crates/re_types/.";
const PYTHON_OUTPUT_DIR_PATH: &str = "rerun_py/rerun_sdk/rerun";
const PYTHON_TESTING_OUTPUT_DIR_PATH: &str = "rerun_py/tests/test_types";
const DOCS_CONTENT_DIR_PATH: &str = "docs/content/reference/types";
Expand Down Expand Up @@ -72,7 +71,6 @@ fn main() {
let definitions_dir_path = workspace_dir.join(DEFINITIONS_DIR_PATH);
let entrypoint_path = workspace_dir.join(ENTRYPOINT_PATH);
let cpp_output_dir_path = workspace_dir.join(CPP_OUTPUT_DIR_PATH);
let rust_output_dir_path = workspace_dir.join(RUST_OUTPUT_DIR_PATH);
let python_output_dir_path = workspace_dir.join(PYTHON_OUTPUT_DIR_PATH);
let python_testing_output_dir_path = workspace_dir.join(PYTHON_TESTING_OUTPUT_DIR_PATH);
let docs_content_dir_path = workspace_dir.join(DOCS_CONTENT_DIR_PATH);
Expand Down Expand Up @@ -120,7 +118,7 @@ fn main() {
),
|| re_types_builder::generate_rust_code(
&reporter,
rust_output_dir_path,
workspace_dir,
&objects,
&arrow_registry
),
Expand Down
Loading

0 comments on commit a364405

Please sign in to comment.