Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider adding support for caching deps and build artifacts #293

Open
starbelly opened this issue Jun 22, 2024 · 5 comments
Open

Consider adding support for caching deps and build artifacts #293

starbelly opened this issue Jun 22, 2024 · 5 comments
Labels
help wanted Extra attention is needed

Comments

@starbelly
Copy link
Member

I believe we discussed this before and did not think it was appropriate, mainly because cache invalidation is the second hardest problem in computer science 😉

Setting up cache in a project on your own is a trivial matter, though people disdain the redundancy of copy / pasta on every project, which I can understand.

setup-beam would either have to leverage a composite action, which doesn't sound great to me, or utilize github-toolkit (as we currently depend on today).

I think if this was added, it should be enabled by default. We'd also need to provide a way to bust the cache (i.e., a cache_version input). Github provides a way to delete cache in the UI, but it also annoying to go through IMHO.

The main caveats are here is when the cache invalid (e.g., an upload to github went weird), thus the need for a way to bust the cache.

Here is a very old example of a composite action, while I don't think a composite action is the way to go here, it still serves as some kind of example.

https://gist.github.com/starbelly/f423c2a3e059fa9c5e2647e4b5a7b175

@starbelly starbelly changed the title Consider adding support for caching deps and build artificates Consider adding support for caching deps and build artifacts Jun 22, 2024
@paulo-ferraz-oliveira
Copy link
Collaborator

You mean cache e.g. _build and stuff generated in the consumer's pipeline (like Dialyzer cache?)? What if they're not using rebar3, what assumptions are we to make? I maybe understood this wrong, but:

  1. what issue are you trying to solve?
  2. what interface do you propose to solve it?

@starbelly
Copy link
Member Author

  1. The issue that others want to solve is repetetive copy/pasta (i.e., copy a cache solution from one project to another, get it wrong, edit, push, repeat until right).
  2. I'm not proposing an interface myself yet, this has to be thought about more is there are hidden complexities when it comes to caching, especially across different tools (rebar3, hex, etc). and repository structures (i.e., lib, single app, and umbrella). Rather, this is about a simple discussion of whether we'd like to start to think about those problems.

@paulo-ferraz-oliveira paulo-ferraz-oliveira added the help wanted Extra attention is needed label Jun 27, 2024
@paulo-ferraz-oliveira
Copy link
Collaborator

There's a lot to unpack here, for sure:

  1. what to we want to cache
  2. when to we want to cache
  3. how much control over the cache we want to give over configuration options
  4. ...

@rhcarvalho
Copy link

As reference points, these two "language/ecosystem" actions have built-in cache integration:

I don't think caching here needs to cover all possible use cases, as long as users can opt-out and go use actions/cache or something else when desired.

Here's a current cache usage I have in a workflow:

      - uses: actions/cache@v4
        with:
          path: |
            deps
            _build
          key: ${{ runner.os }}-elixir-${{ hashFiles('**/mix.lock') }}
          restore-keys: |
            ${{ runner.os }}-elixir-

@paulo-ferraz-oliveira
Copy link
Collaborator

Yeah, but then this turns into a bigger problem: we already support Elixir, Erlang and Gleam and intend on eventually supporting LFE (possibly Luerl). So now you have to maintain 4 cache "types", plus "when to bust them?" (?)

Looking at your example:

  1. you might be missing, for example, the Elixir version, since that should bust the cache
  2. the same goes for the Erlang/OTP version
  3. I've had many issues with restore-keys in the past, especially when stuff moves in the main branch and provokes incompatibilities, etc.

There's a host of issues the action will not solve (and would get unnecessarily complicated for) that we could probably curate in a community -driven Gist of good practices, or similar.

Then again, this could also be a different action that'd assume setup-beam exists, and you'd have something like

- uses: erlef/setup-beam-cache
  with:
    for: elixir

and then inside the specific Elixir rules would be maintained.

Or even several actions, if the differences are too big.

I just don't think, at this moment, this is a specialisation the action needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

3 participants