CI: Caching

[zeshanziya]

No description provided.

[hussainweb]

Thanks! I ran the pipeline again to see the level of improvement.

This is the run: https://github.com/contrib-tracker/backend/actions/runs/9128381842/attempts/1. The task where it pulls images took 1m33s. Caching took 1m50s (it was 1.6 GB) but that will probably not happen every time.

[hussainweb]

As the new run is progressing, I see the "images pull" task is taking 20s. But the problem is that the task that restores the cache takes 1m31s. This beats the point. I don't know if this is going to improve over time.

[hussainweb]

@zeshanziya, you can run actions again without doing a dummy commit. :)

[hussainweb]

Not a lot of improvement here. I think the transfer and decompressing the images offsets the saving we see. The overall time is improved only by about 30s or so...

[zeshanziya]

@zeshanziya, you can run actions again without doing a dummy commit. :)

I wanted to run a separate one so that I can compare with the previous one.

[zeshanziya]

@zeshanziya, you can run actions again without doing a dummy commit. :)

I wanted to run a separate one so that I can compare with the previous one.

I just saw it on github, it runs a separate one. Thank you.

[zeshanziya]

@hussainweb

I have added cache steps for npm and composer. Please see below

I have added cache steps for npm and Composer. Please see below:

NPM:

• Cache the npm cache directory and node_modules.
• Currently, we use npm ci, which always performs a clean install, so there's not much improvement.
• I tried npm install --prefer-offline, which improves performance by 10 to 15 seconds. However, the cache restore takes 4-5 seconds, so on average, we save around 10 seconds. But there are posts suggesting not to use this as it may modify package-lock.json and install new versions. I need your input on whether we should use npm install or npm ci.
• In contrib-tracker, we don't have many packages installed, so caching is not very effective. However, in some projects, we have many packages installed, and caching will be beneficial there.
• We can try moving to yarn to improve the performance.

Composer:

• I have cached all Drupal directories, reducing the time for composer install to 5 seconds. Without caching, it takes around 12/15 seconds.

Currently, caching is not making a significant impact, but it will certainly help on other projects where we have many modules/packages installed.

I would also suggest keeping the Docker cache too, as we have a plan to include Solr and Varnish in DDEV, which will increase the number of images, and caching those may help.

[hussainweb]

Please see my comments on Docker caching.

[hussainweb]

My concern with this key is that if there is a new DDEV version, what would be the impact here? I don't think the pipeline will fail but I am curious if the cache size will keep increasing indefinitely.

For example, we have DDEV 1.23.0 in cache and 1.23.1 is out. The key will still match and it will restore the images. But DDEV will download new images anyway.

Now, the question is what happens in the post action step. Will it see the new images and update the cache? If it updates the cache, will it remove the older images? If it doesn't remove the old images, then the cache size will increase significantly and it will take more time to restore the images than downloading fresh.

[hussainweb]

My suggestion is to figure out if we can bring the DDEV version within this key. If that takes some time, then we should remove this into a separate PR.

[zeshanziya]

@hussainweb Thanks for noticing out concern with cache key. I have added ddev version to cache key. Please review.