Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make global transforms cache versioned #12075

Open
lptr opened this issue Jan 30, 2020 · 6 comments
Open

Make global transforms cache versioned #12075

lptr opened this issue Jan 30, 2020 · 6 comments
Labels
a:chore Minor issue without significant impact in:artifact-transforms

Comments

@lptr
Copy link
Member

lptr commented Jan 30, 2020

Currently the output of global artifact transforms are stored under $GRADLE_HOME/caches/transforms-2 which is a cross-version cache location. However, the locations of the individual workspaces are bound to the Gradle version executing the related transforms. Thus there is no chance of different versions reusing content from this cache directory.

To complicate matters more, the state of the workspaces is tracked via the global version-bound executionHistory.

Therefore the cache should actually be a non-cross-version cache, and be moved under $GRADLE_HOME/$GRADLE_VERSION/caches/transforms (or similar).

One downside of doing this is that after upgrading to a new version of Gradle the old caches won't be cleaned up anymore, which could end up wasting disk space. This is not a special case for the transforms cache, but it's a potentially large cache where the effect is perhaps more visible. We should handle this problem separately though.


cc: @gradle/build-cache

@wolfs
Copy link
Member

wolfs commented Jan 30, 2020

I think we already clean up version specific caches of older Gradle versions when the older version hasn't been used for a certain amount of time.

@lptr
Copy link
Member Author

lptr commented Jan 30, 2020

As discussed with @wolfs we should probably also migrate to a different layout. Right now we have the following layout:

- $GRADLE_HOME/caches/transforms-2/
  - files-2.1/
    - <HASH>/  <---- output directory for the transform
    - <HASH>.bin <-- result file for the transform

We could move to this:

- $GRADLE_HOME/$GRADLE_VERSION/caches/transforms/
  - <HASH>  <-------- workspace root
    - output/  <----- output directory for the transform
    - results.bin <-- result file for the transform

There is no real need for the files-2.1 directory, so let's do away with that. OTOH, it would be nicer to have a single root/entity for every workspace. It just makes more sense to me for one, and if we want to make the cache atomic (so a failed execution won't be able to corrupt it), we need a single root anyway.

@lptr
Copy link
Member Author

lptr commented Jan 30, 2020

I've added a generic workspace allocator in this (currently draft) PR that we can eventually migrate to: #12072

@stale
Copy link

stale bot commented Jan 29, 2021

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

@stale stale bot added the stale label Jan 29, 2021
@lptr
Copy link
Member Author

lptr commented Jan 29, 2021

We should do this.

@stale stale bot removed the stale label Jan 29, 2021
@stale
Copy link

stale bot commented Apr 17, 2022

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

@stale stale bot added the stale label Apr 17, 2022
@lptr lptr removed the stale label Apr 20, 2022
@lptr lptr added the a:chore Minor issue without significant impact label Sep 12, 2023
@lptr lptr removed the @execution label Feb 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:chore Minor issue without significant impact in:artifact-transforms
Projects
None yet
Development

No branches or pull requests

3 participants