Introduce a transient Docker image (SPOC-250).

[danolivo]

To avoid repeating the process of refreshing the OS image required for Spock testing, prepare it once, store it at ghcr.io, and reuse it to build a Docker image with a patched Postgres and Spock extension.

No automatic check of the cached entry is necessary; re-run the workflow manually in GitHub Actions if the image appears slightly stale.

TODO:
It makes sense to cache a Docker image with a patched Postgres branch for supported versions on top of a basic Docker image, as we don't frequently change our patches. There will be some checks necessary to ensure that the patch set hasn't been changed since the last image creation, as well as the 'install.py' script.

[jason-lynch]

The overall idea makes sense to me! I've left a couple of minor suggestions, but, apart from making Codacy happy, it's nothing that I would block on.

[jason-lynch]

Are we going to use this image across multiple jobs? If not, you could keep skip publishing the image and use the gha cache to cut down on rebuilds.

[danolivo]

My idea was to re-use it between all the products, no only in Spock. Saving of computational resources is my addiction ;).

[pct960]

LGTM. I think the eventual goal to make this pipeline efficient would be to have A) cached base Docker image for dev/testing --> B) base image A + cached image with pgedge installed --> C) base image B + cached image with pg patches applied and compiled --> D) base image C + final image with latest spock changes.

Image A gets updated manually. Image B gets updated when there is a new release at https://downloads.pgedge.com/platform/repos/download/, and image C gets updated when the patch files are modified in a PR. Image D is always built.

Some of this might be overkill for our needs, but separating these out will help us save on actions storage and minutes, with, of course, the nice side effect of fast test runs.

[danolivo]

Some of this might be overkill for our needs, but separating these out will help us save on actions storage and minutes, with, of course, the nice side effect of fast test runs.

1. In our paradigm, we test not a commit, but the PR. Inventing image B means updating it on each commit to each PRed branch. Not sure we want it.
2. I tried to write a script to create image C, but it should be triggered on each commit to check that the patch set (or install.py) has changed. GitHub Actions, at the moment, is a little limited in guaranteeing that this image will be created before the next testing phase, so I just ended up with image A. It resolves our current problem.

[pct960]

@danolivo

In our paradigm, we test not a commit, but the PR. Inventing image B means updating it on each commit to each PRed branch. Not sure we want it.

Not really. We only rebuild the last image to compile and install Spock. All other images are cached. In fact, we could combine the pgEdge and (patched) Postgres images and call it pgedge-postgres. The base devel image and the other image are updated only as needed. In concrete terms:

• Image test-base, pre-built and cached: Contains development-related packages. You already have this with Dockerfile-base.el9
• Image pgedge-postgres, pre-built and cached: uses test-base. Contains installed pgEdge binaries along with compiled and installed patched Postgres. We could have three flavours of this image for different Postgres versions. This is basically what your TODO comment was referring to.
• Image spock, not cached: Uses pgedge-postgres. Built every time the GitHub Actions pipeline runs.

If we're not worried about the test-base image being reusable across other repositories, we could just have one cached image with devel binaries, pgedge binaries, and patched Postgres. The correct flavour of this cached image (based on the Postgres version) is pulled, Spock is compiled and installed, and the Spockbench tests are executed. This is the simplest option IMO.

I tried to write a script to create image C, but it should be triggered on each commit to check that the patch set (or install.py) has changed. GitHub Actions, at the moment, is a little limited in guaranteeing that this image will be created before the next testing phase, so I just ended up with image A. It resolves our current problem.

We can configure github actions to selectively run a job if the patches/ directory is changed. This job could rebuild the cached image if our patches change.

[danolivo]

We can configure github actions to selectively run a job if the patches/ directory is changed

That is the key to why I have stopped here. We have no guarantee this job will be executed before any other testing job that may cause failure. It is just not flexible enough to design a proper YAML script. I personally don't like such false positives because they waste hours of scrutiny.
Maybe you propose to add to each script something like:

on:
  push:
    paths:
      - 'src/**'

Unfortunately, it must be checked at each script file, and each job in these scripts should verify that, at least, while we don't want to test each commit.

Anyway, I'm no welder when it comes to this stuff. So, if you have ideas, I would be glad to discover the schema with code samples - Miro diagrams are a good place for such things.

[mason-sharp]

One last thing- I think we should delete Dockerfile.el9 if it is no longer being used.

[danolivo]

One last thing- I think we should delete Dockerfile.el9 if it is no longer being used.

Done