Due to rate limits of Docker we copy imaages for CI to apache/*

[potiuk]

Docker introduced rate limits for image pulls:

https://www.docker.com/increase-rate-limits

We are using a number of images which we do not have to build
ourselves and used them from official images such as mongo or
rabbit or cassndra, postgres, mysql, but we started to hit the limits
where a lot of builds are run in quick succession (with #14531 it will
become even worse because our builds will run faster).

Therefore we pushed the images to apache/airflow-* prefixed
images - apache/* images are exempted from pull rate limit.

Example failed build:

https://github.com/apache/airflow/runs/2117788455?check_suite_focus=true#step:6:378

Pulling postgres (postgres:13)...
toomanyrequests: You have reached your pull rate limit. You may
increase the limit by authenticating and upgrading:
https://www.docker.com/increase-rate-limit

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[ashb]

Hmmm something is going on here as the self-hosted runners were(are?) configured to be logged in as a airflowcirunners user which is a paid-for docker hub account, so so shouldn't be subject to pull limits.

Let me take a look at that and see what has gone on there first.

Curious, something is removing it.

On a failed job:

  Docker config after change:
  {
    "auths": {
      "docker.pkg.github.com": {
        "auth": "cG90aXVrOnYx***"
      }
    },
    "HttpHeaders": {
      "User-Agent": "Docker-Client/19.03.8 (linux)"
    },
    "experimental": "enabled"
  }

On a sucessful job:

   Docker config after change:
  {
    "auths": {
      "docker.pkg.github.com": {
        "auth": "cG90aXVrOnYx***"
      },
      "https://index.docker.io/v1/": {
        "auth": "YWlyZmxvd2NpcnV***"
      }
    },
    "HttpHeaders": {
      "User-Agent": "Docker-Client/19.03.8 (linux)"
    },
    "experimental": "enabled"
  }

@potiuk Can you think of anything in the scripts/ that might remove this login from .docker/config.json entirely? (Nothing springs to mind, just double checking)

[potiuk]

We have docker login for GitHub Registry and reconfiguring docker to enable the experimental features (see above) but none of that should be destructive.

[ashb]

Yeah -- the thing I can't work out is how some runners have the login still, but others don't.

[potiuk]

I wil take a look during my testing and see if I can figure it out.

[potiuk]

I could not find any reason in the scripts. I see that it is initialized when the runner starts from ssm. It is a bit puzzling indeed why it has disappeared. There is only one potential place that something could hapen with ${HOME}/.docker/ folder. There is a "forward-credentials.yaml" docker compose file that maps the .docker credentials inside the container but only if we set FORWARD_CREDENTIALS variable to "true". And it does not look like we are doing it anywhere in the CI.

This is very puzzling. I will add some more diagnostics soon. For now I think we can close it.