snapshoter.py error when running "buildah commit" on centos 8

[hhenkel]

On centos 8 with python 3.6.8, buildah 1.5, ansible-bender (current head from master) and ansible 2.9.0rc2 as "vagrant" user, I get the following error when I try to build a container with ansble-bender build simple-playbook.yaml:

Writing manifest to image destination
Storing signatures
9571314a1c4b084b25183fd0a3355c13966eed6c633d0ba633d6f13cef2a9e55

PLAY [Demonstration of ansible-bender functionality] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************************************
ok: [a-very-nice-image-20191017-111321187300-cont]

TASK [Run a sample command] ****************************************************************************************************************************************************************************************************
changed: [a-very-nice-image-20191017-111321187300-cont]
11:14:04.433 snapshoter.py     ERROR  error while running the build: Command '['buildah', 'commit', '-q', '--iidfile', '/tmp/tmpamg4g2q0', 'a-very-nice-image-20191017-111321187300-cont']' returned non-zero exit status 1.

TASK [Stat a file] *************************************************************************************************************************************************************************************************************
skipping: [a-very-nice-image-20191017-111321187300-cont]

PLAY RECAP *********************************************************************************************************************************************************************************************************************
a-very-nice-image-20191017-111321187300-cont : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

Getting image source signatures
Skipping fetch of repeat blob sha256:03901b4a2ea88eeaad62dbe59b072b28b6efa00491962b8741081c5df50c65e0
Skipping fetch of repeat blob sha256:c5728f2d608b440a199ad90adec2648d4b7930e1c03198810c4c3a22f0e79522
Skipping fetch of repeat blob sha256:e66c3e5b679d5c2973342563778a9ce60a16e13b46ec52d9e48f076361bca3f1
Skipping fetch of repeat blob sha256:27d48c45ea8617a3d8b8590f9f770ee6992f00641d977ef9ef393c86c784329b
Skipping fetch of repeat blob sha256:1bab2a1920e5e25964a6a1ebd8164ae4355ced6d85d6d811d7ab2b09a5f928f1
Copying blob sha256:69d71db6b36fe815c43f0d8d9f2aa1423d6f75e77f6a10250c3856fb10619cbb

 0 B / 455.50 KiB [------------------------------------------------------------]
 455.50 KiB / 455.50 KiB [==================================================] 0s
Copying config sha256:9f1a68934463586371f0f1f7e34891e194c8f20e53fc92def0384eded98760da

 0 B / 5.80 KiB [--------------------------------------------------------------]
 5.80 KiB / 5.80 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
9f1a68934463586371f0f1f7e34891e194c8f20e53fc92def0384eded98760da
Image 'a-very-nice-image' was built successfully \o/

It's worth to mention that even though there is an error, the image gets written. If I try to run a playbook with multiple tasks, it seems like all the other tasks are skipped due to the error.

[hhenkel]

Same error appears when I try to build the container as user "root", therefore it seems like it is not related to rootless stuff.

[TomasTomecek]

Does buildah function properly in that environment? Could you try buildah from centos and buildah run ...?

[hhenkel]

So I did a quick test and it seems to work fine. If you need to see something different, let me know.

(venv) [vagrant@centos8 buildah]$ container=$(buildah from docker.io/library/centos:7)
(venv) [vagrant@centos8 buildah]$ buildah run $container -- yum -y update
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.init7.net
 * extras: mirror.init7.net
 * updates: mirror.init7.net
base                                                                                                                                                                                                     | 3.6 kB  00:00:00
extras                                                                                                                                                                                                   | 2.9 kB  00:00:00
updates                                                                                                                                                                                                  | 2.9 kB  00:00:00
(1/4): base/7/x86_64/group_gz                                                                                                                                                                            | 165 kB  00:00:00
(2/4): extras/7/x86_64/primary_db                                                                                                                                                                        | 152 kB  00:00:00
(3/4): updates/7/x86_64/primary_db                                                                                                                                                                       | 1.9 MB  00:00:02
(4/4): base/7/x86_64/primary_db                                                                                                                                                                          | 6.0 MB  00:00:05
Resolving Dependencies
--> Running transaction check
---> Package audit-libs.x86_64 0:2.8.4-4.el7 will be updated
---> Package audit-libs.x86_64 0:2.8.5-4.el7 will be an update
...

[hhenkel]

I looked a bit more in the issue with "buildah commit" and it seems ansible-bender ist missing a parameter. Following the test done above, I executed the following:

buildah commit --iidfile /tmp/containerid $container

That gave me:

an image name must be specified
ERRO[0000] exit status 1

When I use the command like this, it works fine:

buildah commit --iidfile /tmp/containerid $container new_commited_container

The intial error message from ansible-bender looks like this:

11:14:04.433 snapshoter.py     ERROR  error while running the build: Command '['buildah', 'commit', '-q', '--iidfile', '/tmp/tmpamg4g2q0', 'a-very-nice-image-20191017-111321187300-cont']' returned non-zero exit status 1.

Therefore it seems like there is one parameter at the end missing (in the rhel/centos 8 version of buildah)?

[TomasTomecek]

Thanks for such a thorough investigation!

I recall I was playing with buildah commit not so long ago and ran into a similar issue: AFAIK there was a change in the CLI behaviour of buildah commit that image name was a required argument and they removed the requirement in a recent buildah release. So all I need to do is to make bender work with older buildah interface.

[TomasTomecek]

For the reference, the change happened in buildah 1.7.3: https://github.com/containers/buildah/releases/tag/v1.7.3

[TomasTomecek]

Another thing which should be fixed here is that the build should fail once buildah commit exits with RC > 0.