Cache json db #2642

rtbo · 2023-05-18T08:28:29Z

As discussed in #2640, proposal to write Json files in the cache for 3rd party software like Meson to assess compatibility of artifacts in the Dub cache.
I've made the choice to write small Json files per package version rather than a gigantic one for the whole cache.
This comes at the price of stabilizing the directory structure within the cache (up to where the json file is written).

The PR add the following files in the cache

~/.dub/cache/vibe-d/0.9.6/+http/build-db.json
~/.dub/cache/vibe-d/0.9.6/+http/build-db.lock

To ensure interoperability, the following stabilization apply:

~/.dub/cache/vibe-d/0.9.6/+http/build-db.json
~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      1             2

$DUB_HOME/cache default location is stable (env-vars and settings customization still apply)
Json file path relative to $DUB_HOME/cache is stable

The final artifact location remains up to Dub developers.

fixes #2640

github-actions · 2023-05-18T08:28:47Z

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 15

Total warnings: 0

Build statistics:

 statistics (-before, +after)
-executable size=5280736 bin/dub
-rough build time=98s
+executable size=5284832 bin/dub
+rough build time=94s

Full build output

DUB version 1.31.1, built on May 12 2023
LDC - the LLVM D compiler (1.32.2):
  based on DMD v2.102.2 and LLVM 15.0.7
  built with LDC - the LLVM D compiler (1.32.2)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: skylake-avx512
  http://dlang.org - http://wiki.dlang.org/LDC

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc32le    - PowerPC 32 LE
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    ve         - VE
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.33.0-beta.1+commit.13.g88f50e80: building configuration [application]
source/dub/internal/dyaml/node.d(2513,9): Deprecation: scope variable `this` assigned to non-scope parameter `_param_0` calling `match`
source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp`
source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp`
source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.32.2/x64/ldc2-1.32.2-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5284832 bin/dub
STAT:rough build time=94s

WebFreak001 · 2023-05-18T09:26:05Z

since you want to ensure that the output doesn't change, do you want to add a test in the tests folder that checks the file to match exactly using diff?

source/dub/generators/build.d

rtbo · 2023-05-18T10:20:12Z

since you want to ensure that the output doesn't change, do you want to add a test in the tests folder that checks the file to match exactly using diff?

The Json file contains the build ID, which varies with compiler version. So I can't predict in a test what the exact content is.
At best I can check if all the Json fields are there and the value of the immutable ones.

source/dub/generators/build.d

test/pr2642-cache-db/source/test_cache_db.d

cache hits are more likely in the most recent builds

WebFreak001 · 2023-05-18T15:33:49Z

thanks, seems to be a well designed and useful feature!

rtbo · 2023-05-18T16:59:38Z

Thanks for the quick review !

Geod24 · 2023-05-21T16:05:44Z

I've missed the discussion here, but why are we adding JSON files for other tools to depend on ? dub describe was meant as the way to do this, why is it not enough ?

Geod24 · 2023-05-21T16:07:25Z

I'm not sure what "stabilization" means in this context, either. The commit just adds comments.

rtbo · 2023-05-21T16:17:34Z

dub describe doesn't give any info about cached builds.
e.g. on my system

$ tree ~/.dub/cache/urld
/home/remi/.dub/cache/urld
└── 2.1.1
    ├── build
    │   ├── library-debug-cu9jbmcFOeHiBceMMo2VbA
    │   │   └── liburld.a
    │   └── library-debug-Z7qINYX4IxM8muBSlyNGrw
    │       └── liburld.a
    ├── db.json
    └── db.lock

Meson needs to know what these 2 archives are (compiler, compiler version etc.) in order to use them properly.
Meson uses dub describe for everything else (import paths, sub-dependencies etc.), but the cache info is not available.
I have to say that enriching dub describe output would have been a viable option as well (e.g. add the buildId).

rtbo · 2023-05-21T16:19:25Z

I'm not sure what "stabilization" means in this context, either. The commit just adds comments.

What is stable is what should not be changed in future Dub versions.

Geod24 · 2023-05-21T16:55:43Z

What is stable is what should not be changed in future Dub versions.

The only thing that should not be changed in future dub version is whatever is part of the API. And the API to access dub's internal data is dub describe, hence my comment.

I have to say that enriching dub describe output would have been a viable option as well (e.g. add the buildId).

I think that might be a better option. But I need to do some reading on the issue(s).

rtbo added 2 commits May 18, 2023 08:43

stabilize the cache directory structure

5bf1f2d

write cache database Json file

9b63683

WebFreak001 requested changes May 18, 2023

View reviewed changes

source/dub/generators/build.d Show resolved Hide resolved

avoid duplicates in cache DB

b6a2b67

rtbo added 2 commits May 18, 2023 16:04

add "version" to the cache database

e752171

change db filename to "db.json"

06e237f

WebFreak001 reviewed May 18, 2023

View reviewed changes

source/dub/generators/build.d Outdated Show resolved Hide resolved

add test for the cache database

4328c63

rtbo force-pushed the cache-json-db branch from a9b23e8 to 4328c63 Compare May 18, 2023 14:23

WebFreak001 reviewed May 18, 2023

View reviewed changes

test/pr2642-cache-db/source/test_cache_db.d Outdated Show resolved Hide resolved

rtbo added 2 commits May 18, 2023 16:34

cache db test uses gitcompatibledubpackage

8323d1f

checking for duplicates in reverse order

18834c0

cache hits are more likely in the most recent builds

WebFreak001 approved these changes May 18, 2023

View reviewed changes

WebFreak001 merged commit 6dd0508 into dlang:master May 18, 2023
22 of 27 checks passed

rtbo deleted the cache-json-db branch May 18, 2023 16:59

rtbo mentioned this pull request May 18, 2023

Fix DUB dependencies mesonbuild/meson#11798

Open

rtbo mentioned this pull request May 21, 2023

add the build ID in TargetDescription #2644

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cache json db #2642

Cache json db #2642

rtbo commented May 18, 2023

github-actions bot commented May 18, 2023 •

edited

WebFreak001 commented May 18, 2023

rtbo commented May 18, 2023

WebFreak001 commented May 18, 2023

rtbo commented May 18, 2023

Geod24 commented May 21, 2023

Geod24 commented May 21, 2023

rtbo commented May 21, 2023 •

edited

rtbo commented May 21, 2023

Geod24 commented May 21, 2023 •

edited

Cache json db #2642

Cache json db #2642

Conversation

rtbo commented May 18, 2023

github-actions bot commented May 18, 2023 • edited

WebFreak001 commented May 18, 2023

rtbo commented May 18, 2023

WebFreak001 commented May 18, 2023

rtbo commented May 18, 2023

Geod24 commented May 21, 2023

Geod24 commented May 21, 2023

rtbo commented May 21, 2023 • edited

rtbo commented May 21, 2023

Geod24 commented May 21, 2023 • edited

github-actions bot commented May 18, 2023 •

edited

rtbo commented May 21, 2023 •

edited

Geod24 commented May 21, 2023 •

edited