add first version of rgr sandwich

[arlyon]

Hi!

This is a proof-of-concept for the rust-go-rust sandwich. It successfully ports the GetTurboDataDir function onto the rust side, and exposes it via a single statically-linked library turborepo_ffi along with bindings for it. It is expected that anything we need via FFI and the accompanying glue code can live here, so we can put the needed functionality into the right places in rust-land and just re-expose a simple API here.

There are still a number of things that we need to do to finish this work:

• set up gitignore and decide how to integrate the various parts into the toolchain. we use swig to generate the go bindings which needs to be re-run when changing any function signatures, so we'll have to decide if we want that to be a required part of the toolchain or optional. things that need to be ignored: libturborepo_ffi.a, bindings.h (?), messages.pb.go (?), swig_wrap.cxx (?), turborepo_ffi.go (?)
• update the build.rs file for turborepo to statically build and move libturborepo_ffi and associated artifacts into the go module (handled in the makefile)
• set up proto generation properly for the go side
• clean up go.mod (some bits leaked in from testing)
• figure out a strategy for opting in by setting the go build -tag (perhaps enable it by default for canary builds...?)

That said, it's working on both ends using a nice generic interface.

turbo/crates/turborepo-ffi/src/lib.rs

Lines 5 to 32 in 66f3d52

	#[repr(C)]
	#[derive(Debug)]
	pub struct Buffer {
	len: u32,
	data: *mut u8,
	}
	impl<T: prost::Message> From<T> for Buffer {
	fn from(value: T) -> Self {
	let mut bytes = value.encode_to_vec();
	println!("rust bytes: {:x?}", &bytes);
	let data = bytes.as_mut_ptr();
	let len = bytes.len() as u32;
	std::mem::forget(bytes);
	Buffer { len, data }
	}
	}
	impl Buffer {
	fn into_proto<T: prost::Message + Default>(self) -> Result<T, prost::DecodeError> {
	// SAFETY
	// protobuf has a fairly strict schema so overrunning or underrunning the byte
	// array will not cause any major issues, that is to say garbage in
	// garbage out
	let mut slice = unsafe { std::slice::from_raw_parts(self.data, self.len as usize) };
	T::decode(&mut slice)
	}
	}

turbo/cli/internal/ffi/util.go

Lines 10 to 26 in 66f3d52

	func Parse[C proto.Message](b Buffer, c C) {
	bytes := toBytes(b)
	proto.Unmarshal(bytes, c)
	}
	func toBytes(b Buffer) []byte {
	var out []byte
	len := (*uint32)(unsafe.Pointer(b.GetLen().Swigcptr()))
	sh := (*reflect.SliceHeader)(unsafe.Pointer(&out))
	sh.Data = b.GetData().Swigcptr()
	sh.Len = int(*len)
	sh.Cap = int(*len)
	return out
	}

Usage looks something like the following:

turbo/crates/turborepo-ffi/src/lib.rs

Lines 41 to 50 in 66f3d52

	pub extern "C" fn get_turbo_data_dir() -> Buffer {
	// note: this is _not_ recommended, but it the current behaviour go-side
	// ideally we should use the platform specific convention
	// (which we get from using ProjectDirs::from)
	let dirs =
	directories::ProjectDirs::from_path("turborepo".into()).expect("user has a home dir");
	let dir = dirs.data_dir().to_string_lossy().to_string();
	proto::TurboDataDirResp { dir }.into()
	}

turbo/cli/internal/fs/get_turbo_data_dir_rust.go

Lines 11 to 18 in 66f3d52

	// GetTurboDataDir returns a directory outside of the repo
	// where turbo can store data files related to turbo.
	func GetTurboDataDir() turbopath.AbsoluteSystemPath {
	buffer := ffi.Get_turbo_data_dir()
	resp := ffi_proto.TurboDataDirResp{}
	ffi.Parse(buffer, resp.ProtoReflect().Interface())
	return turbopath.AbsoluteSystemPathFromUpstream(resp.Dir)
	}

Implementation Details

• we use go build tags to define which code path to use. When defining the 'go' tag, then the native go implementation is used. When defining the 'rust' tag, then we use ffi. This can be controlled in make via the cli using GO_TAG=rust
• the Buffer type is a simple rust struct that contains a lenth and a pointer to some data. we can wrap that in some simple code to serialize and deserialize a particular protobuf message, handling cleanup etc.

How to try it out:

make turbo GO_TAG=rust

The rust implementation of GetTurboDataDir will print out the path whenever it is called, to verifiy it is being used.

[vercel]

@arlyon is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

You must have Developer access to commit code to Vercel on Vercel. If you contact an administrator and receive Developer access, commit again to see your changes.

Learn more: https://vercel.com/docs/concepts/teams/roles-and-permissions#enterprise-team-account-roles

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
examples-basic-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-cra-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-designsystem-docs	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-kitchensink-blog	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-native-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-nonmonorepo	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-svelte-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-tailwind-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
examples-vite-web	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)
turbo-site	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Feb 3, 2023 at 5:24PM (UTC)

1 Ignored Deployment

Name	Status	Preview	Comments	Updated
turbo-vite-web	⬜️ Ignored (Inspect)			Feb 3, 2023 at 5:24PM (UTC)

[NicholasLYang]

Looks great! Only a nit about some commented out code.

[nathanhammond]

This landed while I was mid-review.

[nathanhammond]

I believe that these are unused right now? I believe we should drop them from this PR.

[nathanhammond]

It's a transitive dependency, but Go doesn't know how to build it.

I don't have a strong opinion here.

[nathanhammond]

Mild preference for this, which is untested but I think it works:

Suggested change

	cd ../crates/turborepo-ffi && protoc --go_out=. messages.proto
	cd ../crates/turborepo-ffi && protoc --go_out=$(CLI_DIR)/ffi/proto messages.proto

[nathanhammond]

Is a bufferfire similar to a dumpster fire?

[nathanhammond]

I think this is our first generic into the Go codebase. I don't think it matters anymore, but I want to make sure we didn't just break linting.

/cc @gsoltis

[nathanhammond]

Can we rename these variables? We're not at a limit on characters and b C.Buffer, c M is unnecessarily cryptic.

[nathanhammond]

This is clever. I still kinda prefer GoBytes in spite of the bonus copy.