Skip to content

Conversation

alan-agius4
Copy link
Collaborator

@alan-agius4 alan-agius4 commented Apr 23, 2020

Cache fallback can be useful in some cases, but in our case it is hindering the CI performance as the cache is growing everyday which results in slower CI times because of the restoring and saving phases.

A fresh cache size is about 450Mb, while currently with fallbacking enabled it has grown to 3.8Gb

Below are some timings taken were we see that having cache fallbacking is doesn't help CI times:

Caching Cache Size Restore Cache Yarn Install Save Cache Total Time Elapsed
Cache w/ Fallback (Miss) 3.81Gb 2m 50s 30s 7m 20s 10m 40s
Cache w/ Fallback (Hit) 3.81Gb 2m 50s 20s 0 3m 10s
Cache wo/ Fallback (Miss) 0 0 40s 30s 1m 10s
Cache wo/ Fallback (Hit) 458mb 20s 20s 0 40s

@alan-agius4 alan-agius4 changed the title ci: timings ci: remove circle ci cache fallback Apr 23, 2020
@alan-agius4 alan-agius4 marked this pull request as ready for review April 23, 2020 08:50
Cache fallback can be useful in some cases, but in our case it is hindering the CI performance as the cache is growing everyday which results in slower CI times because of the restoring and saving phases.

A fresh cache size is about 450Mb, while currently with fallbacking enabled it has grown to 3.8Gb

Below are some timings taken were we see that having cache fallbacking is doesn't help CI times:

| Caching                   | Cache Size | Restore Cache | Yarn Install | Save Cache | Total Time Elapsed |
|---------------------------|------------|---------------|--------------|------------|--------------------|
| Cache w/ Fallback (Miss)  | 3.81Gb     | 2m 50s        | 30s          | 7m 20s     | 10m 40s            |
| Cache w/ Fallback (Hit)   | 3.81Gb      | 2m 50s        | 20s          | 0          | 3m 10s             |
| Cache wo/ Fallback (Miss) | 0          | 0             | 40s          | 30s        | 1m 10s              |
| Cache wo/ Fallback (Hit)  | 458mb      | 20s           | 20s          | 0          | 40s                |
@alan-agius4 alan-agius4 added the target: patch This PR is targeted for the next patch release label Apr 23, 2020
@filipesilva
Copy link
Contributor

Is the assumption here is that, even if we using a smaller backup cache, it still would be slower than just ignoring it? The 3.81gb is surely bloated, but we can discard it by just changing key prefixes and thus go back to a similar cache size.

If the backup cache is also ~450mbs, then it should have have a similar total time of Cache wo/ Fallback (Hit) plus 30s of saving cache because we used the fallback and need to save for this key. That would bring total to same as Cache wo/ Fallback (Miss).

@alan-agius4
Copy link
Collaborator Author

@filipesilva,

Is the assumption here is that, even if we using a smaller backup cache, it still would be slower than just ignoring it?

Yes, because there is the overhead of restoring the fallback cache and saving the new cache.

The 3.81gb is surely bloated, but we can discard it by just changing key prefixes and thus go back to a similar cache size.

We'd need to change the keys quite often, based on how many packages have been updated per week.

Earlier this week:

Found a cache from build 154706 at angular_devkit-0.12.0-ymJ325P4dGZIgoMbjUPVe05i_3Z27cHSu721Y7xbKv4=
Size: 3.1 GiB
Cached paths:
  * /home/circleci/.cache/yarn

Downloading cache archive...
Unarchiving cache...

Today:

No cache is found for key: angular_devkit-0.12.0-nE0g1dDpAmYnOaeX2jkZ0CVaUMKR+TIXe8H_ogCwYxM=
Found a cache from build 154694 at angular_devkit-0.12.0
Size: 3.8 GiB
Cached paths:
  * /home/circleci/.cache/yarn

Downloading cache archive...
Unarchiving cache...

If the backup cache is also ~450mbs, then it should have have a similar total time of Cache wo/ Fallback (Hit) plus 30s of saving cache because we used the fallback and need to save for this key. That would bring total to same as Cache wo/ Fallback (Miss).

Indeed the timings would be very similar, but what would be the benefit seeing that without cache fallback is as performant, and we'd need to keep re-invalidated the cache key every couple of days manually, since we don't have any other means to re-invalid the fallback cache.

@filipesilva
Copy link
Contributor

Eish, wasn't expecting our package church to be that high. I agree it's better to just skip the fallback then.

@alan-agius4
Copy link
Collaborator Author

I think you meant package cache 😛

@alan-agius4 alan-agius4 added the action: merge The PR is ready for merge by the caretaker label Apr 24, 2020
@mgechev mgechev merged commit 5a0683d into angular:master Apr 24, 2020
@alan-agius4 alan-agius4 deleted the ci-timings branch April 24, 2020 17:01
@angular-automatic-lock-bot
Copy link

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

This action has been performed automatically by a bot.

@angular-automatic-lock-bot angular-automatic-lock-bot bot locked and limited conversation to collaborators May 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
action: merge The PR is ready for merge by the caretaker target: patch This PR is targeted for the next patch release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants