[CI][Packaging] Packaging almalinux 9 and centos 9 fail installing arrow due to missing libre2

[asfimport]

+underlined text+The following nightly packaging jobs have been failing:

almalinux-9-amd64
almalinux-9-arm64
centos-9-stream-amd64
centos-9-stream-arm64

It errors when installing with dnf. It seems due to not finding libre2:

 + dnf install -y --enablerepo=crb --enablerepo=epel arrow-devel-9.0.0.dev405-1.el9
Apache Arrow for AlmaLinux 9 - aarch64          2.6 MB/s |  25 kB     00:00    
Extra Packages for Enterprise Linux 9 - aarch64  15 MB/s | 8.3 MB     00:00    
Error: 
 Problem: package arrow-devel-9.0.0.dev405-1.el9.aarch64 requires libarrow.so.900()(64bit), but none of the providers can be installed
  - package arrow-devel-9.0.0.dev405-1.el9.aarch64 requires arrow9-libs = 9.0.0.dev405-1.el9, but none of the providers can be installed
  - conflicting requests
  - nothing provides libre2.so.0a()(64bit) needed by arrow9-libs-9.0.0.dev405-1.el9.aarch64
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
rake aborted!
Command failed with status (1): [docker run --log-driver none --rm --securi...]

We have lately upgraded some vendored versions like RE2: #13570
or rapidjson: #13608

The jobs started failing since the rapidjson one was merged. In case is related:

Reporter: Raúl Cumplido / @raulcd
Assignee: Kouhei Sutou / @kou

_{Note: This issue was originally created as ARROW-17111. Please see the migration documentation for further details.}

[asfimport]

Raúl Cumplido / @raulcd:
cc ~ @kou  

[asfimport]

Kouhei Sutou / @kou:
It's caused by AlmaLinux 9/CentOS 9 Stream update. I've removed cached Docker images on DockerHub and restarted jobs.

[asfimport]

Raúl Cumplido / @raulcd:
Thanks, that seems to be solved on the retried jobs. Where are the cached images hosted? I am trying to follow the workflows but I can't seem to find where are we caching these docker images from upstream. I probably don't have permissions to do what you did but just want to understand a little more.

[asfimport]

Kouhei Sutou / @kou:

Where are the cached images hosted?

DockerHub as I mentioned. :-)

You can find images for AlmaLinux 9 and CentOS 9 Stream at the following URLs:

• https://hub.docker.com/r/apache/arrow-dev/tags?page=1&name=almalinux-9

• https://hub.docker.com/r/apache/arrow-dev/tags?page=1&name=centos-9-stream

  But we can't delete images from these URLs.

  We need to use the following management URLs:

• https://hub.docker.com/repository/docker/apache/arrow-dev/tags?page=1&ordering=last_updated&name=almalinux-9

• https://hub.docker.com/repository/docker/apache/arrow-dev/tags?page=1&ordering=last_updated&name=centos-9-stream

  We can see image URL by build log. For example:

  https://github.com/ursacomputing/crossbow/runs/7385779728?check_suite_focus=true#step:6:34

  
  docker pull apache/arrow-dev:amd64-almalinux-9-package-apache-arrow