Boxbuild stuck at zypper & runs into an interactive shell within VM

[ellcs]

I am trying to build an appliance, using the boxbuild feature. The box has been pulled from the internet and everything seems supercool! :) 👍 Sadly, i am not able to build an image:

• when building as root, the box is actually booting and starting the build process. But it gets stuck on installing packages with zypper. Something weird i experienced on every build: I actually get a command-prompt in the VM.
• as an unprivileged user. It seems it has some issues with tar. This happened, because i first ran the boxbuild as root and the boxes belonged to root in /var/tmp/kiwi. After deleting /var/lib/kiwi, i could run it with low privileges. The images were downloaded again. But i ran into the same issue as root.

Problem description

I am using the following command to trigger the build. Note: Instead of using --type i use --profile, changing it made no difference for me.

kiwi-ng --profile Live system boxbuild --box leap -- \
    --description /home/${USER}/kiwi-descriptions/suse/x86_64/suse-leap-15.2 \
    --target-dir /tmp/myimage

In the same CLI i triggerd the build, i receive the command prompt on the VM and the process is stuck in Processing:.

[ INFO    ]: 15:02:24 | --> package: glibc-locale
[ INFO    ]: 15:02:24 | --> package: openSUSE-release
[ INFO    ]: 15:02:24 | --> package: udev
[ INFO    ]: 15:02:24 | --> package: zypper
[ INFO    ]: Processing: [                                        ] 0%
localhost:~ # 

End of the result.log:

INFO: 14:02:52 | --> package: zypper
DEBUG: 14:02:52 | EXEC: [zypper --non-interactive --pkg-cache-dir /result/build/image-root/var/cache/kiwi/packages --reposd-dir /result/build/image-root/var/cache/kiwi/zypper/repos --solv-cache-dir /result/build/image-root/var/cache/kiwi/zypper/solv --cache-dir /result/build/image-root/var/cache/kiwi/zypper --raw-cache-dir /result/build/image-root/var/cache/kiwi/zypper/raw --config /result/build/image-root/tmplzno8jm3 --root /result/build/image-root install --auto-agree-with-licenses --no-recommends -- ca-certificates ca-certificates-mozilla cracklib-dict-full filesystem glibc-locale openSUSE-release udev zypper]

Expected behaviour

• The build runs successfully.
  • It runs zypper successfully.
• The boxbuild documentation some insights to the local boxes:
  • E.g. where they are located /var/tmp/kiwi/boxes

Steps to reproduce

1. Run an Opensuse 15.2 as a host.
2. Add the appliance-builder repostiory (https://osinside.github.io/kiwi/installation.html#installation-from-obs)
3. Install kiwi-ng from the vendor repo. (https://osinside.github.io/kiwi/installation.html#installation-from-obs)
4. Install python3-kiwi_boxed_plugin from the vendor repo: (https://osinside.github.io/kiwi/self_contained.html)
5. Fetch the example kiwi-descriptions: git clone https://github.com/OSInside/kiwi-descriptions
6. Trigger the build:

   kiwi-ng --profile Live system boxbuild --box leap -- \
   --description /home/${USER}/kiwi-descriptions/suse/x86_64/suse-leap-15.2 \
   --target-dir /tmp/myimage

OS and Software information

• KIWI version:

  Name           : python3-kiwi
  Version        : 9.23.25-lp152.1.1
  Arch           : x86_64
  Vendor         : obs://build.opensuse.org/Virtualization:Appliances
  

• KIWI Boxed plugin version:

  Name           : python3-kiwi_boxed_plugin
  Version        : 0.2.8-lp152.1.1
  Arch           : noarch
  Vendor         : obs://build.opensuse.org/Virtualization:Appliances
  

• Operating system: openSUSE Leap 15.2
• OBS version: ?

[ellcs]

I am not sure if it is helpful, but i paste the beautified qemu command, fetched from ps aux | qemu on the host. Note one path contains a variable ($user):

qemu-system-x86_64 \
 -m 8096 \
 -machine accel=kvm \
 -cpu host \
 -nographic \
 -nodefaults \
 -snapshot \
 -kernel /var/tmp/kiwi/boxes/leap/kernel.x86_64 \
 -append console=hvc0 root=/dev/vda1 rd.plymouth=0 kiwi="--profile Live system build" sharing-backend=_9p_ \
 -drive file=/var/tmp/kiwi/boxes/leap/Leap-Box.x86_64-1.42.1-System-BuildBox.qcow2,if=virtio,driver=qcow2,cache=off,snapshot=on \
 -netdev user,id=user0 \
 -device virtio-net-pci,netdev=user0 \
 -device virtio-serial \
 -chardev stdio,id=virtiocon0 \
 -device virtconsole,chardev=virtiocon0 \
 -fsdev local,security_model=mapped,id=fsdev0,path=/home/$user/repos/kiwi-descriptions/suse/x86_64/suse-leap-15.2 \
 -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=kiwidescription \
 -fsdev local,security_model=mapped,id=fsdev1,path=/tmp/myimage5 \
 -device virtio-9p-pci,id=fs1,fsdev=fsdev1,mount_tag=kiwibundle \
 -initrd /var/tmp/kiwi/boxes/leap/initrd.x86_64 \
 -smp 4

[schaefi]

Hi, I try to reproduce this today. My first guess is on the network. We run kvm using the virtio net emulation which allows a pass through of the network traffic from the box to the guest and kvm manages the connection on the host without requiring the user to configure any network host/guest configuration. virtio net has its limitations and depending on how complex your host network we might run into issues there... just a guess though

as an unprivileged user. It seems it has some issues with tar. This happened, because i first ran the boxbuild as root and the boxes belonged to root in /var/tmp/kiwi. After deleting /var/lib/kiwi, i could run it with low privileges. The images were downloaded again. But i ran into the same issue as root.

yes you are right, if you run it as root first the box images will be downloaded and stored as root. If you run it as user and a new box exists the existing one cannot be moved due to the permissions issue. Maybe we should switch the download location into the calling user's home directory. Could you open an extra issue for this one ? Thanks much

Something weird i experienced on every build: I actually get a command-prompt in the VM.

This behavior is intentional. We boot the box with a box console such that you always have access to this environment as long as the build runs in the box. It's actually an opportunity to get access to the virtual machine. The VM is by default called in snapshot mode, thus any changes are not persistent. If you run with --box-debug the environment stays open (the box does not reboot) and you can work/debug/develop inside of the box. As I did not see a security issue with this console, it's always at your hand when you build.

[schaefi]

In the same CLI i triggerd the build, i receive the command prompt on the VM and the process is stuck in Processing:

Are you sure it's stuck and not just downloading the packages. There is a log file which should tell you more:

tail -f /tmp/mytest/result.log

Let me know what that tells you

My build is currently running and looks good so far

[schaefi]

My build has finished

kiwi-ng --profile Live system boxbuild --box leap -- --description ~ms/Project/kiwi-descriptions/suse/x86_64/suse-leap-15.2/ --target-dir /tmp/mytest/

so I can't reproduce it so far

[ellcs]

Hello @schaefi,

thank you for the reply.

Hi, I try to reproduce this today. My first guess is on the network. We run kvm using the virtio net emulation which allows a pass through of the network traffic from the box to the guest and kvm manages the connection on the host without requiring the user to configure any network host/guest configuration. virtio net has its limitations and depending on how complex your host network we might run into issues there... just a guess though+

I will have a look if it is a network issue. I am just a bit curious, because i could actually build other distributions (e.g. debian) successfully. But you might be right, zypper might not be stuck, but just be able to download.

• But shouldn't i receive a timeout after some time? I left it at zypper for ~15min.
• When i do curl google.com within the box-VM i actually get a response.
• When running zypper repos wihtin the box-VM it says: Warning: No repositories defined. or is that no issue because of the cache? It's odd because the log says that another zypper install could actually complete. See result.log at buttom.

yes you are right, if you run it as root first the box images will be downloaded and stored as root. If you run it as user and a new box exists the existing one cannot be moved due to the permissions issue. Maybe we should switch the download location into the calling user's home directory. Could you open an extra issue for this one ? Thanks much

Sure, I will open another ticket.

This behavior is intentional. We boot the box with a box console such that you always have access to this environment as long as the build runs in the box. It's actually an opportunity to get access to the virtual machine. The VM is by default called in snapshot mode, thus any changes are not persistent. If you run with --box-debug the environment stays open (the box does not reboot) and you can work/debug/develop inside of the box. As I did not see a security issue with this console, it's always at your hand when you build.

Haha, it's not a bug, but a feature! I am totally fine with it.

Are you sure it's stuck and not just downloading the packages. There is a log file which should tell you more:

tail -f /tmp/mytest/result.log

Sure, here is the result.

host $ tail -fn 30 result.log
DEBUG: 14:12:46 | EXEC: [mkdir -p /result/build/image-root/var/lib]
DEBUG: 14:12:46 | EXEC: [ln -s --no-target-directory ../../usr/lib/sysimage/rpm /result/build/image-root/var/lib/rpm]
INFO: 14:12:46 | Setting up repository obs://Virtualization:Appliances:Builder/openSUSE_Leap_15.2
INFO: 14:12:46 | --> Type: rpm-md
INFO: 14:12:46 | --> Priority: 1
INFO: 14:12:46 | --> Translated: http://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/openSUSE_Leap_15.2/
INFO: 14:12:46 | --> Alias: kiwi
DEBUG: 14:12:46 | EXEC: [mv -f /var/cache/kiwi/packages /var/cache/kiwi/packages.moved]
DEBUG: 14:12:46 | EXEC: [zypper --non-interactive --pkg-cache-dir /result/build/image-root/var/cache/kiwi/packages --reposd-dir /result/build/image-root/var/cache/kiwi/zypper/repos --solv-cache-dir /result/build/image-root/var/cache/kiwi/zypper/solv --cache-dir /result/build/image-root/var/cache/kiwi/zypper --raw-cache-dir /result/build/image-root/var/cache/kiwi/zypper/raw --config /result/build/image-root/tmpspmbq7v_ --root /result/build/image-root addrepo --refresh --keep-packages --no-check http://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/openSUSE_Leap_15.2/ kiwi]
DEBUG: 14:12:47 | EXEC: [mv -f /var/cache/kiwi/packages.moved /var/cache/kiwi/packages]
INFO: 14:12:47 | Setting up repository obs://openSUSE:Leap:15.2/standard
INFO: 14:12:47 | --> Type: rpm-md
INFO: 14:12:47 | --> Translated: http://download.opensuse.org/distribution/leap/15.2/repo/oss/
INFO: 14:12:47 | --> Alias: Leap_15_2
DEBUG: 14:12:47 | EXEC: [mv -f /var/cache/kiwi/packages /var/cache/kiwi/packages.moved]
DEBUG: 14:12:47 | EXEC: [zypper --non-interactive --pkg-cache-dir /result/build/image-root/var/cache/kiwi/packages --reposd-dir /result/build/image-root/var/cache/kiwi/zypper/repos --solv-cache-dir /result/build/image-root/var/cache/kiwi/zypper/solv --cache-dir /result/build/image-root/var/cache/kiwi/zypper --raw-cache-dir /result/build/image-root/var/cache/kiwi/zypper/raw --config /result/build/image-root/tmpspmbq7v_ --root /result/build/image-root addrepo --refresh --keep-packages --no-check http://download.opensuse.org/distribution/leap/15.2/repo/oss/ Leap_15_2]
DEBUG: 14:12:47 | EXEC: [mv -f /var/cache/kiwi/packages.moved /var/cache/kiwi/packages]
DEBUG: 14:12:47 | EXEC: [rm -r -f /result/build/image-root/var/cache/kiwi/zypper/solv/@System]
INFO: 14:12:47 | Using package manager backend: zypper
INFO: 14:12:47 | Installing bootstrap packages
INFO: 14:12:47 | --> collection type: onlyRequired
INFO: 14:12:47 | --> package: ca-certificates
INFO: 14:12:47 | --> package: ca-certificates-mozilla
INFO: 14:12:47 | --> package: cracklib-dict-full
INFO: 14:12:47 | --> package: filesystem
INFO: 14:12:47 | --> package: glibc-locale
INFO: 14:12:47 | --> package: openSUSE-release
INFO: 14:12:47 | --> package: udev
INFO: 14:12:47 | --> package: zypper
DEBUG: 14:12:47 | EXEC: [zypper --non-interactive --pkg-cache-dir /result/build/image-root/var/cache/kiwi/packages --reposd-dir /result/build/image-root/var/cache/kiwi/zypper/repos --solv-cache-dir /result/build/image-root/var/cache/kiwi/zypper/solv --cache-dir /result/build/image-root/var/cache/kiwi/zypper --raw-cache-dir /result/build/image-root/var/cache/kiwi/zypper/raw --config /result/build/image-root/tmpspmbq7v_ --root /result/build/image-root install --auto-agree-with-licenses --no-recommends -- ca-certificates ca-certificates-mozilla cracklib-dict-full filesystem glibc-locale openSUSE-release udev zypper]

My build has finished

kiwi-ng --profile Live system boxbuild --box leap -- --description ~ms/Project/kiwi-descriptions/suse/x86_64/suse-leap-15.2/ --target-dir /tmp/mytest/

so I can't reproduce it so far

How long takes your boxed build to complete?

[schaefi]

Hmm, indeed that's a strange behavior from zypper and I haven't seen this so far. You said you killed this after 20min and nothing happens ? really weird

But shouldn't i receive a timeout after some time? I left it at zypper for ~15min.

I would have expected that yes

When i do curl google.com within the box-VM i actually get a response.

ok this means the basic network functionality is there and working.

When running zypper repos wihtin the box-VM it says: Warning: No repositories defined. or is that no issue because of the cache? It's odd because the log says that another zypper install could actually complete. See result.log at buttom.

This is expected. The kiwi build uses non-standard zypper paths to avoid conflicts. If you just call "zypper repos" it would lookup repos defined in the standard path on the box VM and there are none. So at least this is working as expected :-))

How long takes your boxed build to complete?

It was done in under 2minutes. So on normal circumstances I would have expected your build to be done in less than 5minutes. And you should always see progress in the log

Hmm

Can you try the following:

• run the boxbuild with --box-debug
• inside of the box when things are stuck kill the zypper process "kill PID"
• this will also exit the kiwi process, now re-run the kiwi build in the box by just calling /usr/local/bin/run_kiwi

This just restarts the build in the box. Does it get stuck at the same stage again ?

Please note: There is no control character support in the console. So if you hit Ctrl-C or something this will exit your VM process not the process running inside the VM blocking your console ;)

Thanks for your patience. So far I have no idea

[ellcs]

Well, it seems i was not patient enough. The build completed after ~35min. I guess zypper in just took ages on my computer.

Start time: 16:12:32.
End time: 16:47:42.

Solution: Ask the boss for better hardware.

Thank you @schaefi!