ansible-test - add a --docker-pull-only option that pulls down the required images but doesn't execute the tests

[briantist]

Summary

What

Looking for an option to ansible-test that will pull down the docker image(s) required, based on the options you've given, but not actually run the tests. Examples:

ansible-test integration --docker default --python 3.8 --docker-pull-only
ansible-test integration --docker default --docker-pull-only
ansible-test integration --docker ubuntu1604 --python 2.6 --docker-pull-only
ansible-test units --docker default --docker-pull-only
# etc.

Why?

My primary reason

In CI, I'm trying to separate the time it takes for tests to run (which includes other setup or preparation that ansible-test does), from the time taken to pull the test images, which is quite significant.

On a local machine you're re-using the pulled image but in CI you're bearing the cost every time, and I want to see those separately.

This isn't easily achievable outside of ansible-test because the exact image tags pulled are part of the ansible-test code, and in some cases depend on the options chosen (like whether or not to pull the pypi proxy image).

Secondary reason

For those on slow connections, being surprised by a new image after a pull of devel or something can result in painfully long times to pull, and it would be nice to be allow those on slow connections to schedule pull-only runs to keep their local images up-to-date.

Workarounds

I'm adding this into community.hashi_vault's CI by way of a Python script step in a custom GitHub Action that imports some private stuff. It doesn't handle pypi proxy either, though that's less of a concern due to its size at the moment: https://github.com/ansible-collections/community.hashi_vault/blob/main/.github/actions/pull-ansible-test-images/action.yml

Also here's the current state of the python for easier viewing (since it's embedded in YAML in the action):

        import os

        nwd = r'${{ inputs.working-directory }}'
        if nwd:
          os.chdir(nwd)

        try:
            from ansible_test._internal.util_common import get_docker_completion
        except ImportError:
            # 2.9
            from ansible_test._internal.util import get_docker_completion

        context = 'collection'
        wanted = ['default']

        dockers = get_docker_completion()

        for alias, data in dockers.items():
            if alias in wanted:
                if 'context' not in data or data['context'] == context:
                    image = data['name']
                    print('pulling %s' % image)
                    os.system('docker pull %s' % image)

Issue Type

Feature Idea

Component Name

ansible-test

Additional Information

see description

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• bin/ansible-test

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

[gundalow]

ansible-test (sanity,unit,integration) --docker-no-pull do not explicitly pull the latest docker images
This can be useful if you are on a slow connection.

[gundalow]

GitHub Actions also apparently support Docker Caching, though I'm not sure if that's only per run, or per repo.
Have you had the chance to look into that?

You can see how often the Docker images are updated by looking at the change history of https://github.com/ansible/ansible/commits/devel/test/lib/ansible_test/_data/completion/docker.txt

[briantist]

GitHub Actions also apparently support Docker Caching, though I'm not sure if that's only per run, or per repo.
Have you had the chance to look into that?

You can see how often the Docker images are updated by looking at the change history of https://github.com/ansible/ansible/commits/devel/test/lib/ansible_test/_data/completion/docker.txt

@gundalow
Is there a specific "docker caching" support you're referencing?

I have recently looked at using the "Docker Layer Caching" action: https://github.com/marketplace/actions/docker-layer-caching

It's a neat idea, and in theory, exactly what I'd want to do. Unfortunately in practice, it was a complete failure. Both restoring and populating the cache took WAY longer than just pulling. Where an Ansible 2.9 pull of the default image (which is much bigger in 2.9) could take up to 2 minutes, the caching thing replaced that with about 8 minutes of time, much of which was concurrency blocking, overloading the cache service, and retries, and sometimes it timed out.

The available generic caching action is something I am considering trying myself for this use case (using docker save and docker load), to see if trying to limit it to specific images as opposed to every layer on the system.

In fact, that experiment is what made me look at separating out the pull, so that I can measure the differences and ROI of any specific thing that's aimed at reducing the pull time (I have other ideas, at different levels, which I could go into in this ticket or via a separate conversation).

---

So this request is currently more scoped down to separating the pull. There's an internal function for performing the pull; it's based on the arguments sent in, which is great. The reasons I'm not using it in my workaround script above is because I didn't see it until later, and because I didn't (yet) want to duplicate the args, pass them into my action, and figure out how to get them parsed to pass into that function.

Since my workaround uses internal code, it'll be prone to break, and it'd be much nicer as an CLI argument, and I think easy to implement as it introduces almost nothing new internally except mapping the CLI option to that existing pull function (I may be missing important context though).

---

Also I probably need my workaround for a while longer, depending on whether this would get backported. to 2.11/2.10 (and I'd need a workaround for 2.9 still anyway). But it'd be nice going forward!