[Dev][Archery] Push ancestor docker images

[asfimport]

The "debian-c-glib" and "ubuntu-c-glib" docker-compose configurations fail with the following message:

CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find utf8proc (missing: UTF8PROC_LIB UTF8PROC_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/Findutf8proc.cmake:41 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:159 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:2096 (resolve_dependency)
  CMakeLists.txt:467 (include)

 

Reporter: Antoine Pitrou / @pitrou
Assignee: Krisztian Szucs / @kszucs

PRs and other links:

• GitHub Pull Request #7681

_{Note: This issue was originally created as ARROW-9334. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
cc @kou @kszucs

[asfimport]

Antoine Pitrou / @pitrou:
Rebuilding the image fixes the issue, so it seems the uploaded images are not up-to-date?

[asfimport]

Kouhei Sutou / @kou:
Ah, they are old because archery docker push ubuntu-ruby pushes only ubuntu-ruby image.
https://github.com/apache/arrow/runs/842945967?check_suite_focus=true#step:9:2

@kszucs Could you update Archery to push ancestor images of the specified image too? The following patch may work but I didn't run it:

diff --git a/dev/archery/archery/docker.py b/dev/archery/archery/docker.py
index 23448e6da..65fac3a12 100644
--- a/dev/archery/archery/docker.py
+++ b/dev/archery/archery/docker.py
@@ -271,6 +271,8 @@ class DockerCompose(Command):
                    .format(image))
             raise RuntimeError(msg) from None
         else:
+            for ancestor in self.nodes[image]:
+                self._execute_compose('push', ancestor)
             self._execute_compose('push', image)
 
     def images(self):

[asfimport]

Antoine Pitrou / @pitrou:
It seems that similarly, conda-python is not up-to-date (at least with PYTHON=3.6).

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 7681
#7681

[asfimport]

Antoine Pitrou / @pitrou:
Are you sure this is actually fixed?

[asfimport]

Antoine Pitrou / @pitrou:
I got the following:

$ archery docker run --no-build debian-c-glib
Pulling debian-cpp ... done
Pulling debian-c-glib ... done
WARNING: Some service image(s) must be built from source by running:
    docker-compose build debian-c-glib

[...]

CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find utf8proc (missing: UTF8PROC_LIB UTF8PROC_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/Findutf8proc.cmake:41 (find_package_handle_standard_args)
  cmake_modules/ThirdpartyToolchain.cmake:159 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:2096 (resolve_dependency)
  CMakeLists.txt:484 (include)


-- Configuring incomplete, errors occurred!

[asfimport]

Krisztian Szucs / @kszucs:
By passing --no-build you prevent it from rebuilding the image. Try with archery docker run debian-c-glib

[asfimport]

Antoine Pitrou / @pitrou:
That's the whole point. The upstream image should be up-to-date now.

[asfimport]

Krisztian Szucs / @kszucs:
Well, the push has happened https://github.com/apache/arrow/runs/856279754#step:9:20

Additionally archery docker run --no-build debian-c-glib works for me locally, so it may be caused by other docker/docker-compose cache issues.

I'm trying to reproduce it locally.

[asfimport]

Krisztian Szucs / @kszucs:
We're running the ubuntu-ruby builds on the CI not the debian ones, so debian-c-glib and debian-ruby are not up to date, only the ubuntu images.

Executing archery docker run --no-build ubuntu-c-glib works as expected.

[asfimport]

Antoine Pitrou / @pitrou:
Hmm, thank you. It's a bit confusing that we have those Debian images which are not exercised nor updated...

[asfimport]

Krisztian Szucs / @kszucs:
We're exercising them as nightlies, but not pushing them to the apache docker account. I'm creating an issue about pushing from the nightly builds with our own credentials.

https://issues.apache.org/jira/browse/ARROW-9435