test: optimize caching of downloaded assets

[gilbertsoft]

The Issue

Download of assets for tests fails regularly and tests have to be restarted.

How This PR Solves The Issue

This patch avoids the cleanup of assets between the test runs and introduces a unique hash for every assets which are now properly shared between the test runs and should minimize failures.

On GH the cache is always loaded from the master branch and updated once a week. This avoids we use more than the 10 GB available for caching and makes caching more stable. At the time of writing this patch we used almost 40 GB. Because big caches I have avoided the usage of hashFiles() function and implemented a key based on the date instead.

Caches can be removed manually at https://github.com/ddev/ddev/actions/caches.

Manual Testing Instructions

Caching behavior must be observed once this PR is merged. Important points are:

• cache on master is updated once a week
• branches are using the master cache
• asset download errors do not longer occur.

Automated Testing Overview

The changes in the internal asset caching is covered by the existing tests.

Related Issue Link(s)

Release/Deployment Notes

[github-actions]

Download the artifacts for this pull request:

• all-ddev-executables.zip
• ddev-linux-amd64.zip
• ddev-linux-arm64.zip
• ddev-macos-amd64.zip
• ddev-macos-arm64.zip
• ddev-windows-amd64-installer.zip

See Testing a PR

[rfay]

We don't have that much trouble with the caching of CMS tarballs, but improving it is fine. Only one test does the full set of these (Github Pull/TestDdevFullSiteSetup).

The most common failure is the node test, which absolutely needs improvement and caching.

The real approach to caching the CMS tarballs though is to cache them in the github test.

[gilbertsoft]

We don't have that much trouble with the caching of CMS tarballs, but improving it is fine. Only one test does the full set of these (Github Pull/TestDdevFullSiteSetup).

The most common failure is the node test, which absolutely needs improvement and caching.

The real approach to caching the CMS tarballs though is to cache them in the github test.

About 90% of the failures I have is because of failed downloads!

[gilbertsoft]

Happened again in this patch

[rfay]

I see more of the nodejs stuff, but every improvement is welcome.

However, improving the caching in the go code won't help much here unless we actually cache the testcache in the github tests.

[gilbertsoft]

I see more of the nodejs stuff, but every improvement is welcome.

However, improving the caching in the go code won't help much here unless we actually cache the testcache in the github tests.

Will check for more improvments e.g. GH once the tests are green again...

[rfay]

This seems to dump the tarball into two places (root of testcache and also inside tarballs directory):

$ ls -l . tarballs/
.:
total 0
drwxr-xr-x  21 rfay  staff  672 Jul  3 10:04 82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz
drwxr-xr-x   3 rfay  staff   96 Jul  3 10:04 tarballs

tarballs/:
total 43056
-rw-r--r--  1 rfay  staff  21171263 Jul  3 10:04 82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz

The one in the root is the extraction of the one in the tarballs directory I guess. It's pretty confusing.

However, the output shows otherwise, I guess it shows the old destination?

Downloading https://wordpress.org/wordpress-6.0.1.tar.gz
Downloaded https://wordpress.org/wordpress-6.0.1.tar.gz into /Users/rfay/.ddev/testcache/tarballs/82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz
Extracted /Users/rfay/.ddev/testcache/tarballs/82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz into /Users/rfay/.ddev/testcache/82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz
Copying directory /Users/rfay/.ddev/testcache/82f32c881c851f923c3442367116fccc8617c7bcc4efa35ea2e649363a3d1b27.tar.gz to /Users/rfay/tmp/ddevtest/TestPkgWordpress517092105

Please make the tarball have a name that can be studied and understood. For example, this one is for wordpress. Can we keep a name and date on it? I'm ok with it having a hash in the name of course.

My results of the testcache directory after running TestDdevStart with DDEV_DEBUG=true: https://gist.github.com/rfay/585cccdcb0a8814c371ea2655e862df2

[rfay]

I sure like it better now! I'm very willing to give this a try. Should be a win for us.

New testcache:

testcache
├── 00fbfea3_drupal-9.4.2.tar.gz
│   ├── INSTALL.txt
│   ├── LICENSE.txt
│   ├── README.md
│   ├── autoload.php
│   ├── composer.json
│   ├── composer.lock
│   ├── core
│   ├── example.gitignore
│   ├── index.php
│   ├── modules
│   ├── profiles
│   ├── robots.txt
│   ├── sites
│   ├── themes
│   ├── update.php
│   ├── vendor
│   └── web.config
├── 82f32c88_wordpress-6.0.1.tar.gz
│   ├── index.php
│   ├── license.txt
│   ├── readme.html
│   ├── wp-activate.php
│   ├── wp-admin
│   ├── wp-blog-header.php
│   ├── wp-comments-post.php
│   ├── wp-config-sample.php
│   ├── wp-content
│   ├── wp-cron.php
│   ├── wp-includes
│   ├── wp-links-opml.php
│   ├── wp-load.php
│   ├── wp-login.php
│   ├── wp-mail.php
│   ├── wp-settings.php
│   ├── wp-signup.php
│   ├── wp-trackback.php
│   └── xmlrpc.php
└── tarballs
    ├── 00fbfea3_drupal-9.4.2.tar.gz
    └── 82f32c88_wordpress-6.0.1.tar.gz

13 directories, 29 files