Skip to content

Implement embedded FS directly with strings #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 25, 2025

Conversation

jakebailey
Copy link
Member

In profiling #275, I discovered that a good chunk of time and memory allocation is due to the embedded FS. Our VFS interface reads file contents in terms of strings, but the underlying io/fs.FS interface uses mutable byte slices, so that incurs a copy for each file read. If you keep going down, it turns out that go:embed is implemented in terms of immutable in-binary strings, which are then copied out of byte slices! So, each time we access the embedded VFS, we are converting a string to a byte slice, just to turn it back into a string.

If we skip the io/fs.FS layer and instead directly implement our own vfs.FS interface in terms of go:embed'd strings, we can skip this entirely.

On my machine, this brings the total runtime of internal/testutil/runner from 9.3s to 6.3s, reducing memory allocation during the test from 42.7GB to 10.25GB.

I'm not 100% happy with the implementation itself (mainly if we choose to add more stuff, though I doubt I'd do the translations as a FS layer), but, I think it's okay for now.

@jakebailey jakebailey merged commit 226ae75 into main Feb 25, 2025
16 checks passed
@jakebailey jakebailey deleted the jabaile/direct-bundled-vfs branch February 25, 2025 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants