Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filesystem caching #281

Merged
merged 10 commits into from
Sep 14, 2022
Merged

Filesystem caching #281

merged 10 commits into from
Sep 14, 2022

Conversation

matthewmueller
Copy link
Contributor

@matthewmueller matthewmueller commented Sep 14, 2022

This PR is a big one. This PR implements filesystem caching. While the generator sync is currently quite fast, I want to start introducing more expensive generators that can generate code and spawn subprocesses. This requires generators to only run once and ideally not unless they're needed.

There are currently two new levels of caching:

  1. Multiple reads to the same file or generator are cached during a sync, but reset before each sync.
  2. Generator caching persists over multiple syncs but is in memory so reset at the end of the process.

One of the unique features of Bud is its virtual generator filesystem. A nice property that came out of using a virtual filesystem as the primary abstraction is that caching can be granular and implemented in a fairly straightforward way:

  • If a generator reads, stats or globs a file or directory, then that generator depends on that file or directory.

There's a bit more nuance around reacting to newly created nested files and I'm sure it's not 100% perfect yet, but it's good enough to build upon. As far as I can tell, this core piece of Bud is basically done™️ in terms of requirements so I'm excited to get back to building user-facing features on top of Bud's virtual filesystem!

⚠️ I temporarily removed plugin support (which only worked so far for the public/ generator). I'll rework this and add it back in a subsequent PR!

@matthewmueller matthewmueller mentioned this pull request Sep 14, 2022
3 tasks
@matthewmueller matthewmueller merged commit de97779 into main Sep 14, 2022
@matthewmueller matthewmueller deleted the filesystem-caching branch September 14, 2022 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant