Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TRAVIS: enable ccache compression #2243

Merged
merged 1 commit into from May 27, 2020
Merged

TRAVIS: enable ccache compression #2243

merged 1 commit into from May 27, 2020

Conversation

@janisozaur
Copy link
Contributor

janisozaur commented May 12, 2020

Current builds saturate Travis CI caches:

$ diff -U999 /tmp/ccache_before /tmp/ccache_after || true
--- /tmp/ccache_before	2020-05-08 16:34:12.987955074 +0000
+++ /tmp/ccache_after	2020-05-08 16:45:45.570927610 +0000
@@ -1,13 +1,13 @@
 cache directory                     /home/travis/.ccache
 primary config                      /home/travis/.ccache/ccache.conf
 secondary config      (readonly)    /etc/ccache.conf
 cache hit (direct)                  2816
 cache hit (preprocessed)            1045
-cache miss                       5751026
-called for link                    85473
-called for preprocessing            3832
-compile failed                     11462
-no input file                       2558
-files in cache                     11334
-cache size                         451.9 MB
+cache miss                       5755826
+called for link                    85542
+called for preprocessing            3835
+compile failed                     11471
+no input file                       2559
+files in cache                     11430
+cache size                         458.1 MB
 max cache size                     500.0 MB

I doubt it's possible to get more space allocated, though there are no
docs on what exact limits are. I suspect that default explicit setting
matches what's provided. Instead of trying to get more storage,
compress the objects with zlib (default level 6) to try increasing hit
rate.

Testing with compression on/off provides following stats:

--- compression_off	2020-05-12 21:53:41.597030300 +0200
+++ compression_on	2020-05-12 21:53:52.887293915 +0200
@@ -1,18 +1,18 @@
 cache directory                     /home/janisozaur/.ccache
 primary config                      /home/janisozaur/.ccache/ccache.conf
 secondary config      (readonly)    /etc/ccache.conf
-stats updated                       Tue May 12 20:42:00 2020
-stats zeroed                        Tue May 12 20:29:24 2020
+stats updated                       Tue May 12 20:21:36 2020
+stats zeroed                        Tue May 12 20:09:03 2020
 cache hit (direct)                     0
 cache hit (preprocessed)               0
 cache miss                          3383
 cache hit rate                      0.00 %
 called for link                       41
 called for preprocessing               3
 compile failed                         9
 no input file                          1
 cleanups performed                     0
 files in cache                     13544
-cache size                         377.3 MB
+cache size                         211.9 MB
 max cache size                       5.0 GB
-make -j 4  1997,44s user 332,76s system 336% cpu 11:31,51 total
+make -j 4  2009,84s user 333,39s system 333% cpu 11:41,85 total

43% reduction of cached files' size for 10s increase of wall clock compilation time (on my machine). This should hopefully increase the hit rate significantly, which now nearly doesn't exist.

After having merged this, I would suggest clearing the Travis' caches (cf. https://docs.travis-ci.com/user/caching/#clearing-caches) to ensure stale objects don't prevent newer compressed ones from getting in.

Note: this is a re-open of my previous PR which was closed with comment I don't really understand: #2230 (comment)

Current builds saturate Travis CI caches:

```
$ diff -U999 /tmp/ccache_before /tmp/ccache_after || true
--- /tmp/ccache_before	2019-11-19 21:45:52.491212479 +0000
+++ /tmp/ccache_after	2019-11-19 21:58:32.335898083 +0000
@@ -1,10 +1,13 @@
 cache directory                     /home/travis/.ccache
 primary config                      /home/travis/.ccache/ccache.conf
 secondary config      (readonly)    /etc/ccache.conf
 cache hit (direct)                     0
 cache hit (preprocessed)               0
-cache miss                             0
-no input file                          1
-files in cache                         0
-cache size                           0.0 kB
+cache miss                          4156
+called for link                       66
+called for preprocessing               3
+compile failed                         9
+no input file                          2
+files in cache                     10568
+cache size                         434.1 MB
 max cache size                     500.0 MB
```

I doubt it's possible to get more space allocated, though there are no
docs on what exact limits are. I suspect that default explicit setting
matches what's provided. Instead of trying to get more storage,
compress the objects with zlib (default level 6) to try increasing hit
rate.
@mduggan
Copy link
Contributor

mduggan commented May 13, 2020

The travis macos build particularly takes so long at the moment it often times out, so this seems like something that would be worth a try.

@sev- sev- merged commit 4c831ea into scummvm:master May 27, 2020
2 checks passed
2 checks passed
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@janisozaur
Copy link
Contributor Author

janisozaur commented May 27, 2020

@sev- can you also reset the caches on Travis, if not done yet?

@mduggan
Copy link
Contributor

mduggan commented May 27, 2020

Dobe - I just cleared all the travis caches.

@janisozaur
Copy link
Contributor Author

janisozaur commented Jun 1, 2020

I'd say this is a success. After a few days, since merging, collecting the logs from latest travis build:

$ diff -U999 /tmp/ccache_before /tmp/ccache_after || true
--- /tmp/ccache_before	2020-06-01 19:55:46.910877939 +0000
+++ /tmp/ccache_after	2020-06-01 19:56:58.886877939 +0000
@@ -1,13 +1,13 @@
 cache directory                     /home/travis/.ccache
 primary config                      /home/travis/.ccache/ccache.conf
 secondary config      (readonly)    /etc/ccache.conf
-cache hit (direct)                307538
+cache hit (direct)                312367
 cache hit (preprocessed)             123
-cache miss                          6371
-called for link                     4485
-called for preprocessing             195
-compile failed                       585
-no input file                        131
-files in cache                     24315
+cache miss                          6373
+called for link                     4554
+called for preprocessing             198
+compile failed                       594
+no input file                        132
+files in cache                     24321
 cache size                         367.3 MB
 max cache size                     500.0 MB

312367 direct only hits now vs 3861 direct and preprocessed combined before. And the cache is still smaller than before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.