Make codegen I/O-free and agnostic to output location

[teh-cmc]

Commit by commit

This is necessary refactoring work for the upcoming attr.rust.custom_crate attribute, itself necessary for the upcoming serde-codegen support, itself necessary for the upcoming blueprint experimentations as well as #3741.

Changes

1. The CodeGenerator trait as well as all post-processing helpers (gitattributes, orphan detection...) are now I/O-free.

pub type GeneratedFiles = std::collections::BTreeMap<camino::Utf8PathBuf, String>;

pub trait CodeGenerator {
    fn generate(
        &mut self,
        reporter: &crate::Reporter,
        objects: &crate::Objects,
        arrow_registry: &crate::ArrowRegistry,
    ) -> GeneratedFiles;
}

2. All post-processing helpers are now agnostic to the location output.
   This is very important as it makes it possible to generate e.g. rust code out of the re_types crate without everything crumbling down.
   A side-effect is that gitattributes files are now finer-grained.
3. The Rust codegen pass is now crate agnostic: it is driven by the workspace path rather than a specific crate path.
   Necessary for the upcoming attr.rust.custom_crate.
4. All codegen passes now follow the exact same 4-step structure:

// 1. Generate in-memory code files.
let mut gen = MyGenerator::new();
let mut files = gen.generate(reporter, objects, arrow_registry);
// 2. Generate in-memory attribute files.
generate_gitattributes_for_generated_files(&mut files);
// 3. Write all in-memory files to disk.
write_files(&gen.pkg_path, &gen.testing_pkg_path, &files);
// 4. Remove orphaned files.
crate::codegen::common::remove_orphaned_files(reporter, &files);

5. The documentation codegen pass now removes its orphans, which is why some md files were removed in this PR.

---

• Unblocks Tracking issue: arrow cleanup & migration away from arrow2{-convert} #3741
• Unblocks Add a --check argument to cargo codegen #3495

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested demo.rerun.io (if applicable)
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

• PR Build Summary
• Docs preview
• Examples preview
• Recent benchmark results
• Wasm size tracking

[emilk]

I don't understand why you moved out the language-specific formatting out of the language-specific modules?

To me it makes more sense if GeneratedFiles would be the final files, with no extra processing required.
Then these functions would be a lot more similar to each other (identical?)

[emilk]

…and it would make the PR diff smaller too :)

[teh-cmc]

I just don't like the the mixup of concerns: the codegen pass should only concern itself with generating code, not any post-processing (including formatting) nor I/O.
Especially so with the current implementation where the codegen pass is nice and pure while the formatting pass requires I/O, process spawning, etc.

I will likely introduce a formalized formatting pass in #3495 though, which will make all these generate_XXX methods go away and make --checking trivial.

[emilk]

that makes sense!