Measure local build cache size growth #477

lptr · 2023-02-06T14:07:58Z

When tests use the local cache, its size can grow. We want to capture that information in the benchmarks.

src/main/resources/org/gradle/profiler/report/report-template.html

-    <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/7.2.0/math.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.7.14/vue.min.js"></script>
+    <script src='https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js'></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/11.5.1/math.min.js"></script>


src/main/resources/org/gradle/profiler/report/report-template.html

-    <script src='https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.0.0-beta/chart.min.js'></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/7.2.0/math.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.7.14/vue.min.js"></script>
+    <script src='https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.9.1/chart.min.js'></script>


src/main/resources/org/gradle/profiler/report/report-template.html

-    <script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.6.12/vue.min.js"></script>
-    <script src='https://cdnjs.cloudflare.com/ajax/libs/Chart.js/3.0.0-beta/chart.min.js'></script>
-    <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjs/7.2.0/math.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.7.14/vue.min.js"></script>


…size

asodja

LGTM, some comments and tests need attention I see

asodja · 2023-02-07T16:35:36Z

subprojects/build-operations/src/main/java/org/gradle/trace/buildops/BuildOperationTrace.java

+                    ? Stream.empty()
+                    : Stream.of(cacheDirs);
+            }
+            cacheDirectories.forEach(cacheDirectory -> {


💭 Maybe just with streams:

cacheDirectories .map(File::listFiles) .filter(Objects::nonNull) .flatMap(Stream::of) .forEach(file -> { fileCount.incrementAndGet(); size.addAndGet(file.length()); });

Alright, I'll bite :)

asodja · 2023-02-07T16:36:51Z

src/main/java/org/gradle/profiler/report/JsonResultWriter.java

-import com.google.gson.JsonPrimitive;
-import com.google.gson.JsonSerializationContext;
-import com.google.gson.JsonSerializer;
+import com.google.gson.*;


🧐 Star import that needs attention?

asodja · 2023-02-07T17:00:12Z

src/main/java/org/gradle/profiler/mutations/ShowBuildCacheSizeMutator.java

+        if (cacheFiles == null) {
+            System.out.println("> Cannot list cache directory " + cacheDir);
+        } else {
+            long size = 0;


💭 Maybe this could be:

long size = Stream.of(cacheFiles).mapToLong(File::length).sum()

asodja

❓ I noticed build cache size is always reported zero, is that just for me?

* Running measured build #25
Execution time 1112 ms
> Build cache size: 0 bytes in 0 file(s) (build-cache-1)
> Build cache size: 0 bytes in 0 file(s) (build-cache-2)

lptr · 2023-02-07T20:53:50Z

Are you running a build without the build cache being enabled? I get it reported fine:

* Running cleanup for measured build #1
> Build cache size: 17 bytes in 2 file(s) (build-cache-1)
> Build cache size: 48,480,256 bytes in 1 file(s) (build-cache-2)

lptr · 2023-02-08T08:31:55Z

I added a test for show-build-cache-size.

asodja

LGTM

asodja · 2023-02-09T09:56:32Z

src/main/java/org/gradle/profiler/result/DurationSample.java

+
+    @Override
+    public double extractValue(T result) {
+        return extractTotalDurationFrom(result).toNanos() / 1_000_000d;


🧐 Does Duration has .toMillis(), to just do extractTotalDurationFrom(result).toMillis()?

This is to capture fractional milliseconds.

Otherwise format can contain a comma instead of a dot on some systems.

lptr · 2023-02-11T21:09:00Z

Well spotted fixes, thanks!

Introduce sample units

02b8d22

lptr added the @execution label Feb 6, 2023

lptr self-assigned this Feb 6, 2023

lptr force-pushed the lptr/measure-cache-size branch from 1ac94a1 to ab9f7c6 Compare February 6, 2023 14:15

Show separate axes for each unit

0381cef

lptr force-pushed the lptr/measure-cache-size branch from ab9f7c6 to 0381cef Compare February 6, 2023 14:16

Fix integration test

e16ca95

lptr force-pushed the lptr/measure-cache-size branch from 7640820 to e16ca95 Compare February 6, 2023 14:48

lptr added 3 commits February 6, 2023 17:04

Benchmark local cache size

420ec07

Format numbers on chart

8c04f50

Upgrade JS libraries used

76459db

github-advanced-security bot found potential problems Feb 7, 2023

View reviewed changes

lptr added 9 commits February 7, 2023 11:14

Show all cache directory sizes

0880b47

Count all local cache directories

bb9f7de

Capture cache size in MiB instead of KiB

1596aca

Use responsive chart

6ff465b

Improve chart rendering

d5c5452

Fix tests

adc4b60

Merge remote-tracking branch 'origin/master' into lptr/measure-cache-…

5d346e9

…size

Fix more tests

0730e35

Consider all files in the build cache when calculating size

ca80b02

lptr changed the title ~~Measure local cache size growth~~ Measure local build cache size growth Feb 7, 2023

Use local build cache everywhere

feab46e

asodja approved these changes Feb 7, 2023

View reviewed changes

asodja requested changes Feb 7, 2023

View reviewed changes

Improve functional code

b36bc2a

Improve lambdas more

2b9e0f4

lptr requested a review from asodja February 7, 2023 21:01

Revert star imports

c72ffe9

Fix test

41fcf1d

lptr added the enhancement label Feb 8, 2023

lptr marked this pull request as ready for review February 8, 2023 08:04

Add test for show-build-cache-size

d5fcd4a

asodja approved these changes Feb 9, 2023

View reviewed changes

asodja added 2 commits February 11, 2023 18:03

Fix benchmark local build cache size integration test

8bbc2d1

Use Locale.US for DOUBLE_FORMAT in CsvGenerator

735db1b

Otherwise format can contain a comma instead of a dot on some systems.

asodja force-pushed the lptr/measure-cache-size branch from 7f742e1 to 735db1b Compare February 11, 2023 17:18

lptr merged commit 836ce78 into master Feb 11, 2023

lptr deleted the lptr/measure-cache-size branch February 11, 2023 21:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Measure local build cache size growth #477

Measure local build cache size growth #477

lptr commented Feb 6, 2023

asodja left a comment •

edited

asodja Feb 7, 2023

lptr Feb 7, 2023

asodja Feb 7, 2023

asodja Feb 7, 2023

lptr Feb 7, 2023

asodja left a comment •

edited

lptr commented Feb 7, 2023 •

edited

lptr commented Feb 8, 2023

asodja left a comment

asodja Feb 9, 2023

lptr Feb 9, 2023

lptr commented Feb 11, 2023

Measure local build cache size growth #477

Measure local build cache size growth #477

Conversation

lptr commented Feb 6, 2023

asodja left a comment • edited

Choose a reason for hiding this comment

asodja Feb 7, 2023

Choose a reason for hiding this comment

lptr Feb 7, 2023

Choose a reason for hiding this comment

asodja Feb 7, 2023

Choose a reason for hiding this comment

asodja Feb 7, 2023

Choose a reason for hiding this comment

lptr Feb 7, 2023

Choose a reason for hiding this comment

asodja left a comment • edited

Choose a reason for hiding this comment

lptr commented Feb 7, 2023 • edited

lptr commented Feb 8, 2023

asodja left a comment

Choose a reason for hiding this comment

asodja Feb 9, 2023

Choose a reason for hiding this comment

lptr Feb 9, 2023

Choose a reason for hiding this comment

lptr commented Feb 11, 2023

asodja left a comment •

edited

asodja left a comment •

edited

lptr commented Feb 7, 2023 •

edited