Caching problems on OS X travis machines

[springmeyer]

Currently OS X builds may fail because downloading and unpacking the cache is so slow that it hits the travis 10m limit for one operation (https://docs.travis-ci.com/user/common-build-problems/#My-builds-are-timing-out). This is a side effect of #2684

Because the travis caching is not in our control I don't see many solutions other than:

• reducing the amount of things we cache to reduce the cache size
• re-instating the cacher timeout to prevent builds from failing and instead allow them to continue without a cache.

Example of a failed job: https://travis-ci.org/Project-OSRM/osrm-backend/jobs/149749254#L47

[springmeyer]

re-instating the cacher timeout to prevent builds from failing and instead allow them to continue without a cache.

I think we should try this first, and this is done in #2743. The idea is that when the OS X machines are not running to slow they will be able to unpack the cache in under 10m and benefit from the cache during builds and testing. When they are slow (network likely) and cannot unpack the cache in under 10 minutes then the cacher will timeout (failing silently) before the entire job times out and the build will proceed, just slowly since the cache is missing.

/cc @daniel-j-h to review and merge if this seems like a worthwhile approach.

[daniel-j-h]

I cleared all caches yesterday before the 5.3.1 release. They were over 2 GB in size. The main problem is not the timeout or the caching per se, the main problem is there is no way to clear the cache other than going into the UI and doing it manually.

[springmeyer]

They were over 2 GB in size

@daniel-j-h keep in mind that the UI at https://travis-ci.org/Project-OSRM/osrm-backend/caches lists the aggregate size for all jobs. Each cache's size can be viewed using the travis cache command:

$ travis cache
Detected repository as Project-OSRM/osrm-backend, is this correct? |yes| yes
On branch 5.3:
cache-linux-trusty-d8cdc401316dcf244ab7e0817905cf4de736cfa7b1460328cb75afe417dffc1b--compiler-gcc-4.8-debug.tgz         last modified: 2016-08-04 07:40:00  size: 97.71 MiB

On branch PR.2739:
cache-linux-trusty-32a2d0186778ea57ff9e06e2419ef5c589715dc864d001d648e641592afdd6ff--compiler-gcc-5-release-shared.tgz  last modified: 2016-08-05 06:11:04  size: 125.0 MiB
cache-linux-trusty-59fe2cf4dcd64e5f6be6cbf3414297b1dbcd9e68dad7fea1e4c6d2d45b307983--compiler-gcc-5-debug.tgz           last modified: 2016-08-05 06:07:47  size: 3.25 MiB
cache-linux-trusty-7c7a9aaa96f099cd97d8e419f3b274c8a86d41cf764386223d783d2bd4126927--compiler-gcc-5-release.tgz         last modified: 2016-08-05 06:07:40  size: 125.0 MiB
cache-linux-trusty-7e76f4c34f91b271042cb132b400fb5a33c84b0b7ace424f86a2ea6a8a7d315a--compiler-clang-3.8-debug.tgz       last modified: 2016-08-05 06:00:10  size: 40.59 MiB
cache-linux-trusty-b24ecea759992f6c301799c8c3f8b805e628b986a4a0fb6708408007fb23c82e--compiler-gcc-5-release-i686.tgz    last modified: 2016-08-05 06:12:57  size: 123.66 MiB
cache-osx-trusty-xcode7.3-326d0a261e2b63827a6817f892ea0f514e9a44920c8fe7086a1512ce37597137--compiler-clang.tgz          last modified: 2016-08-05 06:10:12  size: 43.87 MiB

On branch PR.2741:
cache-linux-trusty-32a2d0186778ea57ff9e06e2419ef5c589715dc864d001d648e641592afdd6ff--compiler-gcc-5-release-shared.tgz  last modified: 2016-08-04 14:59:35  size: 182.22 MiB
cache-linux-trusty-59fe2cf4dcd64e5f6be6cbf3414297b1dbcd9e68dad7fea1e4c6d2d45b307983--compiler-gcc-5-debug.tgz           last modified: 2016-08-04 15:09:42  size: 3.23 MiB
cache-linux-trusty-7c7a9aaa96f099cd97d8e419f3b274c8a86d41cf764386223d783d2bd4126927--compiler-gcc-5-release.tgz         last modified: 2016-08-04 15:00:32  size: 182.2 MiB
cache-linux-trusty-7e76f4c34f91b271042cb132b400fb5a33c84b0b7ace424f86a2ea6a8a7d315a--compiler-clang-3.8-debug.tgz       last modified: 2016-08-04 15:04:31  size: 38.58 MiB
cache-linux-trusty-b24ecea759992f6c301799c8c3f8b805e628b986a4a0fb6708408007fb23c82e--compiler-gcc-5-release-i686.tgz    last modified: 2016-08-04 15:04:23  size: 179.55 MiB
cache-osx-trusty-xcode7.3-326d0a261e2b63827a6817f892ea0f514e9a44920c8fe7086a1512ce37597137--compiler-clang.tgz          last modified: 2016-08-04 15:21:40  size: 42.29 MiB

On branch PR.2743:
cache-linux-trusty-32a2d0186778ea57ff9e06e2419ef5c589715dc864d001d648e641592afdd6ff--compiler-gcc-5-release-shared.tgz  last modified: 2016-08-04 17:32:35  size: 121.96 MiB
cache-linux-trusty-59fe2cf4dcd64e5f6be6cbf3414297b1dbcd9e68dad7fea1e4c6d2d45b307983--compiler-gcc-5-debug.tgz           last modified: 2016-08-04 17:34:08  size: 1.74 MiB
cache-linux-trusty-7c7a9aaa96f099cd97d8e419f3b274c8a86d41cf764386223d783d2bd4126927--compiler-gcc-5-release.tgz         last modified: 2016-08-04 17:37:55  size: 121.98 MiB
cache-linux-trusty-7e76f4c34f91b271042cb132b400fb5a33c84b0b7ace424f86a2ea6a8a7d315a--compiler-clang-3.8-debug.tgz       last modified: 2016-08-04 17:30:03  size: 38.53 MiB
cache-linux-trusty-b24ecea759992f6c301799c8c3f8b805e628b986a4a0fb6708408007fb23c82e--compiler-gcc-5-release-i686.tgz    last modified: 2016-08-04 17:33:53  size: 120.66 MiB
cache-osx-trusty-xcode7.3-326d0a261e2b63827a6817f892ea0f514e9a44920c8fe7086a1512ce37597137--compiler-clang.tgz          last modified: 2016-08-04 17:46:03  size: 42.22 MiB

On branch master:
cache-linux-trusty-32a2d0186778ea57ff9e06e2419ef5c589715dc864d001d648e641592afdd6ff--compiler-gcc-5-release-shared.tgz  last modified: 2016-08-05 10:17:19  size: 168.38 MiB
cache-linux-trusty-59fe2cf4dcd64e5f6be6cbf3414297b1dbcd9e68dad7fea1e4c6d2d45b307983--compiler-gcc-5-debug.tgz           last modified: 2016-08-05 09:54:33  size: 6.45 MiB
cache-linux-trusty-7c7a9aaa96f099cd97d8e419f3b274c8a86d41cf764386223d783d2bd4126927--compiler-gcc-5-release.tgz         last modified: 2016-08-05 09:59:55  size: 168.38 MiB
cache-linux-trusty-7e76f4c34f91b271042cb132b400fb5a33c84b0b7ace424f86a2ea6a8a7d315a--compiler-clang-3.8-debug.tgz       last modified: 2016-08-05 10:16:18  size: 55.52 MiB
cache-linux-trusty-b24ecea759992f6c301799c8c3f8b805e628b986a4a0fb6708408007fb23c82e--compiler-gcc-5-release-i686.tgz    last modified: 2016-08-05 10:12:49  size: 166.86 MiB
cache-osx-trusty-xcode7.3-326d0a261e2b63827a6817f892ea0f514e9a44920c8fe7086a1512ce37597137--compiler-clang.tgz          last modified: 2016-08-05 10:12:31  size: 60.11 MiB

Overall size of above caches: 2.21 GiB

[springmeyer]

The main problem is not the timeout or the caching per se, the main problem is there is no way to clear the cache other than going into the UI and doing it manually.

Focusing on that issue over at -> #2745