Issue with guestmount building on WSL2

[matsbror]

Background Work

• Yes, I searched the mailing list
• Yes, I searched prior issues
• Yes, I searched the documentation

Chipyard Version and Hash

Release: 1.10.0
Hash: commit b7644b2 (HEAD, tag: 1.10.0, origin/stable)

OS Setup

Trying to build under WSL2, Ubuntu 22.04. According to a closed issue, this should work.

$ uname -a
Linux BRONSON 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ conda list
# packages in environment at /home/mats/hpca/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
annotated-types           0.5.0              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hc6cd4ac_0    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachecontrol              0.13.1             pyhd8ed1ab_0    conda-forge
cachecontrol-with-filecache 0.13.1             pyhd8ed1ab_0    conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h2fee648_5    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
click-default-group       1.2.4              pyhd8ed1ab_0    conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.7.4          py310hff52083_0    conda-forge
conda-libmamba-solver     23.7.0             pyhd8ed1ab_0    conda-forge
conda-lock                1.4.0              pyhd8ed1ab_2    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              41.0.4          py310h75e40e8_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
ensureconda               1.4.3              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
filelock                  3.12.4             pyhd8ed1ab_0    conda-forge
fmt                       10.1.1               h00ab1b0_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
jaraco.classes            3.3.0              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py310hff52083_3    conda-forge
keyring                   24.2.0          py310hff52083_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libcurl                   8.3.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libglib                   2.78.0               hebfc3b9_0    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libmamba                  1.5.1                h744094f_0    conda-forge
libmambapy                1.5.1           py310h39ff949_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.24               hfc55251_4    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.5               h232c23b_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.5.1           py310h51d5547_0    conda-forge
markupsafe                2.1.3           py310h2372a71_1    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.6           py310hd41b1e2_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5764c6d_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  2.3.0              pyhd8ed1ab_0    conda-forge
pydantic-core             2.6.3           py310hcb5633a_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
secretstorage             3.3.3           py310hff52083_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
tk                        8.6.13               h2797004_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.12.1             pyha770c72_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
virtualenv                20.24.4            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.21.0          py310h1275a96_1    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

$ printenv
SHELL=/bin/bash
NVM_INC=/home/mats/.nvm/versions/node/v19.6.1/include/node
WSL2_GUI_APPS_ENABLED=1
CONDA_EXE=/home/mats/hpca/conda/bin/conda
_CE_M=
WSL_DISTRO_NAME=U22
WT_SESSION=d88c8258-5b36-49f5-933f-c2fa130d879a
LIBGUESTFS_TRACE=1
XILINX_VIVADO=/home/mats/tools/Xilinx/Vivado/2022.2
SSH_AUTH_SOCK=/tmp/ssh-XXXXXXPvLB8x/agent.415
SSH_AGENT_PID=429
XML_CATALOG_FILES=file:///home/mats/hpca/conda/etc/xml/catalog file:///etc/xml/catalog
NAME=BRONSON
PWD=/home/mats/hpca/chipyard
GSETTINGS_SCHEMA_DIR=/home/mats/hpca/conda/share/glib-2.0/schemas
LOGNAME=mats
CONDA_PREFIX=/home/mats/hpca/conda
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
GIT_PS1_SHOWCOLORHINTS=true
WASI_VERSION_FULL=20.0
HOME=/home/mats
WASI_SDK_PATH=/opt/wasi-sdk-20.0
LANG=C.UTF-8
WSL_INTEROP=/run/WSL/378_interop
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
WASMTIME_HOME=/home/mats/.wasmtime
WASMCLOUD_OCI_ALLOWED_INSECURE=localhost:5000
WAYLAND_DISPLAY=wayland-0
WASI_VERSION=20
XILINX_HLS=/home/mats/tools/Xilinx/Vitis_HLS/2022.2
CONDA_PROMPT_MODIFIER=(base)
PROMPT_COMMAND=set_bash_prompt
LIBGUESTFS_DEBUG=1
NVM_DIR=/home/mats/.nvm
WASI_SYSROOT=/opt/wasi-sdk/share/wasi-sysroot
LESSCLOSE=/usr/bin/lesspipe %s %s
TERM=xterm-256color
_CE_CONDA=
LESSOPEN=| /usr/bin/lesspipe %s
USER=mats
CONDA_SHLVL=1
DISPLAY=:0
SHLVL=1
NVM_CD_FLAGS=
CONDA_PYTHON_EXE=/home/mats/hpca/conda/bin/python
LD_LIBRARY_PATH=:/home/mats/lib
ENABLE_SBT_THIN_CLIENT=1
XDG_RUNTIME_DIR=/run/user/1000/
CONDA_DEFAULT_ENV=base
WSLENV=WT_SESSION::WT_PROFILE_ID
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/home/mats/.local/bin:/home/mats/bin:/home/mats/hpca/conda/bin:/home/mats/hpca/conda/condabin:/home/mats/.wasmer/bin:/home/mats/.wasmtime/bin:/home/mats/tools/Xilinx/Vitis_HLS/2022.2/bin:/home/mats/tools/Xilinx/Model_Composer/2022.2/bin:/home/mats/tools/Xilinx/DocNav:/home/mats/tools/Xilinx/Vivado/2022.2/bin:/home/mats/.nvm/versions/node/v19.6.1/bin:/home/mats/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/libnvvp:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/NVIDIA Corporation/Nsight Compute 2022.2.1:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Program Files/Git/cmd:/mnt/d/flutter/bin:/mnt/c/Program Files/PuTTY:/mnt/c/Program Files (x86)/Calibre2:/mnt/c/Program Files/usbipd-win:/mnt/c/Users/mbror/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/mbror/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/ProgramData/Microsoft/Windows/Start Menu/Programs/NVM for Windows:/snap/bin:/home/mats/riscv/bin:/home/mats/bin:/home/mats/.wasmer/globals/wapm_packages/.bin
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
NVM_BIN=/home/mats/.nvm/versions/node/v19.6.1/bin
HOSTTYPE=x86_64
WASMER_CACHE_DIR=/home/mats/.wasmer/cache
PULSE_SERVER=unix:/mnt/wslg/PulseServer
WT_PROFILE_ID={884b5260-6c0c-544e-bd40-07aee13d445e}
WASMER_DIR=/home/mats/.wasmer
OLDPWD=/home/mats/hpca
_=/usr/bin/printenv

Other Setup

I have followed the documentation here: https://chipyard.readthedocs.io/en/stable/Chipyard-Basics/Initial-Repo-Setup.html

Also installed guestmount like this:
sudo apt-get install -y guestmount libguestfs-tools

Current Behavior

Command ./build-setup.sh riscv-tools fails when the following command is called (see attached log file):

subprocess.CalledProcessError: Command 'guestmount --pid-file guestmount.pid -a /home/mats/hpca/chipyard/software/firemarshal/images/firechip/br-base/br-base.img -m /dev/sda /home/mats/hpca/chipyard/software/firemarshal/disk-mount' returned non-zero exit status 1.

Detailed trace output of the guestmount command:

$ guestmount --pid-file guestmount.pid -a /home/mats/hpca/chipyard/software/firemarshal/images/firechip/br-base/br-base.img -m /dev/sda /home/mats/hpca/chipyard/software/firemarshal/disk-mount
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: create: flags = 0, handle = 0x55e4054e1770, program = guestmount
libguestfs: trace: set_recovery_proc false
libguestfs: trace: set_recovery_proc = 0
libguestfs: trace: add_drive "/home/mats/hpca/chipyard/software/firemarshal/images/firechip/br-base/br-base.img"
libguestfs: trace: add_drive = 0
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: trace: version
libguestfs: trace: version = <struct guestfs_version = major: 1, minor: 46, release: 2, extra: , >
libguestfs: trace: get_backend
libguestfs: trace: get_backend = "direct"
libguestfs: launch: program=guestmount
libguestfs: launch: version=1.46.2
libguestfs: launch: backend registered: unix
libguestfs: launch: backend registered: uml
libguestfs: launch: backend registered: libvirt
libguestfs: launch: backend registered: direct
libguestfs: launch: backend=direct
libguestfs: launch: tmpdir=/tmp/libguestfsOWrNLq
libguestfs: launch: umask=0022
libguestfs: launch: euid=1000
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: begin building supermin appliance
libguestfs: run supermin
libguestfs: command: run: /usr/bin/supermin
libguestfs: command: run: \ --build
libguestfs: command: run: \ --verbose
libguestfs: command: run: \ --if-newer
libguestfs: command: run: \ --lock /var/tmp/.guestfs-1000/lock
libguestfs: command: run: \ --copy-kernel
libguestfs: command: run: \ -f ext2
libguestfs: command: run: \ --host-cpu x86_64
libguestfs: command: run: \ /usr/lib/x86_64-linux-gnu/guestfs/supermin.d
libguestfs: command: run: \ -o /var/tmp/.guestfs-1000/appliance.d
supermin: version: 5.2.1
supermin: package handler: debian/dpkg
supermin: acquiring lock on /var/tmp/.guestfs-1000/lock
supermin: build: /usr/lib/x86_64-linux-gnu/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/excludefiles type uncompressed excludefiles
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/hostfiles type uncompressed hostfiles
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/init.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages type uncompressed packages
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-hfsplus type uncompressed packages
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-reiserfs type uncompressed packages
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/packages-xfs type uncompressed packages
supermin: build: visiting /usr/lib/x86_64-linux-gnu/guestfs/supermin.d/udev-rules.tar.gz type gzip base image (tar)
supermin: mapping package names to installed packages
supermin: resolving full list of package dependencies
supermin: build: 218 packages, including dependencies
supermin: build: 8444 files
supermin: build: 4972 files, after matching excludefiles
supermin: build: 4975 files, after adding hostfiles
supermin: build: 4972 files, after removing unreadable files
supermin: build: 4978 files, after munging
supermin: kernel: looking for kernel using environment variables ...
supermin: kernel: looking for kernels in /lib/modules/*/vmlinuz ...
supermin: kernel: looking for kernels in /boot ...
supermin: failed to find a suitable kernel (host_cpu=x86_64).

I looked for kernels in /boot and modules in /lib/modules.

If this is a Xen guest, and you only have Xen domU kernels
installed, try installing a fullvirt kernel (only for
supermin use, you shouldn't boot the Xen guest with it).
libguestfs: error: /usr/bin/supermin exited with error status 1, see debug messages above
libguestfs: trace: launch = -1 (error)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x55e4054e1770 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsOWrNLq

Expected Behavior

The build setup should work.

Other Information

Log file:
build-setup.log

[jerryz123]

Hmm, this will take some time to debug.
For now, a workaround would be to skip the firemarhshal initialization step using a flag to build-setup (use -h to see the flags).

This only prevents you from building firemarshal linux images

[matsbror]

Thanks @jerryz123 , I also tried to perform the build setup in an Ubuntu 22.04 docker container and it failed in the same step.

[jerryz123]

Could this be the same issue? steigr/docker-hipchat-server#1

[matsbror]

Could this be the same issue? steigr/docker-hipchat-server#1

I will try

@jerryz123 sorry to say it did not change the outcome.

[matsbror]

Have you had a chance to look at this?

[barrettsmits]

When using virt-customize on a ubuntu 22.04 WSL2 instance I was getting the same error, I had to get a Linux image installed to address the above error.

While searching I stumbled up on this issue: Github Issue Link. As my WSL env is fully ephemeral, I went ahead and installed the kernel, and got virt-customize working as expected.

Hopefully this helps someone in need.