aarch64 mock shell failed with error: /usr/sbin/ldconfig: No such file or directory

[junaruga]

Short description of the problem

mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell to run aarch64 environment is failed with the below error on my x86_64 Fedora 30 host environment.
Could you give me some advices to fix the issue?
Thanks.

$ uname -m
x86_64

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 --scrub=all

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell

error: lua script failed: [string "%post(glibc-2.29.9000-28.fc31.aarch64)"]:6: /usr/sbin/ldconfig: No such file or directory
error: No exec() called after fork() in lua scriptlet
error: lua script failed: [string "%post(glibc-2.29.9000-28.fc31.aarch64)"]:6: /usr/sbin/iconvconfig: No such file or directory
error: No exec() called after fork() in lua scriptlet
...
error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %post(libsepol-2.9-1.fc31.aarch64) scriptlet failed, exit status 127
...
Error in POSTIN scriptlet in rpm package libsepol
...
Failed:
  ca-certificates-2019.2.32-2.fc31.noarch   libutempter-1.1.6-16.fc30.aarch64  

Error: Transaction failed

Output of rpm -q mock

$ rpm -q mock
mock-1.4.16-1.fc30.noarch

$ rpm -q qemu-user-static
qemu-user-static-3.1.0-8.fc30.x86_64

$ which python3
/usr/local/python-3.6.5/bin/python3

Steps to reproduce issue

Run below command with /etc/mock/fedora-rawhide-aarch64.cfg following this document.

1. mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell

Do not forget to mention full commandline with the mock command you executed.

The outputted full log is here.

Any additional notes

Consider adding output of mock --debug-config this can help developers to reproduce the issue.

$ mock --debug-config
INFO: mock.py version 1.4.16 starting (python version = 3.7.3)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
config_opts['backup_base_dir'] = '/var/lib/mock/backup'
config_opts['backup_on_clean'] = False
config_opts['basedir'] = '/var/lib/mock'
config_opts['bootstrap_chroot_additional_packages'] = []
config_opts['bootstrap_module_enable'] = []
config_opts['bootstrap_module_install'] = []
config_opts['build_log_fmt_name'] = 'unadorned'
config_opts['build_log_fmt_str'] = '%(message)s'
config_opts['cache_alterations'] = False
config_opts['cache_topdir'] = '/var/cache/mock'
config_opts['check'] = True
config_opts['chroot_name'] = 'default'
config_opts['chroot_setup_cmd'] = 'install @buildsys-build'
config_opts['chrootgid'] = 135
config_opts['chrootgroup'] = 'mock'
config_opts['chroothome'] = '/builddir'
config_opts['chrootuid'] = 28707
config_opts['chrootuser'] = 'mockbuild'
config_opts['clean'] = True
config_opts['cleanup_on_failure'] = False
config_opts['cleanup_on_success'] = False
config_opts['config_file'] = '/etc/mock/default.cfg'
config_opts['config_paths'] = ['/etc/mock/site-defaults.cfg',
 '/etc/mock/site-defaults.cfg',
 '/etc/mock/default.cfg',
 '/etc/mock/fedora-rawhide-x86_64.cfg',
 '/home/jaruga/.config/mock.cfg',
 '/home/jaruga/.config/mock.cfg']
config_opts['createrepo_command'] = '/usr/bin/createrepo_c -d -q -x *.src.rpm'
config_opts['createrepo_on_rpms'] = False
config_opts['dist'] = 'rawhide'
config_opts['dnf_command'] = '/usr/bin/dnf'
config_opts['dnf_disable_plugins'] = ['local', 'spacewalk']
config_opts['dnf_install_command'] = 'install dnf dnf-plugins-core distribution-gpg-keys'
config_opts['dynamic_buildrequires'] = False
config_opts['enable_disable_repos'] = []
config_opts['environment'] = {'HOME': '/builddir',
 'HOSTNAME': 'mock',
 'LANG': 'en_US.UTF-8',
 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin',
 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"',
 'PS1': '<mock-chroot> \\s-\\v\\$ ',
 'SHELL': '/bin/bash',
 'TERM': 'vt100'}
config_opts['exclude_from_homedir_cleanup'] = ['build/SOURCES', '.bash_history', '.bashrc']
config_opts['extra_chroot_dirs'] = ['/run/lock']
config_opts['files'] = {}
config_opts['forcearch'] = None
config_opts['hostname'] = None
config_opts['internal_dev_setup'] = True
config_opts['legal_host_arches'] = ('x86_64',)
config_opts['log_config_file'] = 'logging.ini'
config_opts['macros'] = {'%_buildhost': 'unused-4-164.brq.redhat.com',
 '%_rpmfilename': '%%{NAME}-%%{VERSION}-%%{RELEASE}.%%{ARCH}.rpm',
 '%_topdir': '/builddir/build'}
config_opts['microdnf_builddep_command'] = '/usr/bin/dnf'
config_opts['microdnf_builddep_opts'] = []
config_opts['microdnf_command'] = '/usr/bin/microdnf'
config_opts['microdnf_common_opts'] = []
config_opts['microdnf_install_command'] = 'dnf-install microdnf dnf dnf-plugins-core distribution-gpg-keys'
config_opts['module_enable'] = []
config_opts['module_install'] = []
config_opts['more_buildreqs'] = {}
config_opts['no_root_shells'] = False
config_opts['nosync'] = True
config_opts['nosync_force'] = False
config_opts['nspawn_args'] = ['--capability=cap_ipc_lock',
 '--bind=/tmp/mock-resolv.vga0gwdm:/etc/resolv.conf']
config_opts['online'] = True
config_opts['opstimeout'] = 0
config_opts['package_manager'] = 'dnf'
config_opts['plugin_conf'] = {'bind_mount_enable': True,
 'bind_mount_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                     'cache_topdir': '/var/cache/mock',
                     'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                     'create_dirs': False,
                     'dirs': [('/home/jaruga/rh/tests/ruby', '/mnt/tests_ruby'),
                              ('/home/jaruga/git/cucumber/multi_test',
                               '/mnt/multi_test')],
                     'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                     'root': 'fedora-rawhide-x86_64'},
 'ccache_enable': False,
 'ccache_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                 'cache_topdir': '/var/cache/mock',
                 'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                 'compress': None,
                 'dir': '%(cache_topdir)s/%(root)s/ccache/u%(chrootuid)s/',
                 'max_cache_size': '4G',
                 'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                 'root': 'fedora-rawhide-x86_64'},
 'chroot_scan_enable': False,
 'chroot_scan_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                      'cache_topdir': '/var/cache/mock',
                      'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                      'only_failed': True,
                      'regexes': ['^[^k]?core(\\.\\d+)?$', '\\.log$'],
                      'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                      'root': 'fedora-rawhide-x86_64'},
 'hw_info_enable': True,
 'hw_info_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                  'cache_topdir': '/var/cache/mock',
                  'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                  'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                  'root': 'fedora-rawhide-x86_64'},
 'lvm_root_enable': False,
 'lvm_root_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                   'cache_topdir': '/var/cache/mock',
                   'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                   'pool_name': 'mockbuild',
                   'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                   'root': 'fedora-rawhide-x86_64'},
 'mount_enable': False,
 'mount_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                'cache_topdir': '/var/cache/mock',
                'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                'root': 'fedora-rawhide-x86_64'},
 'overlayfs_enable': False,
 'overlayfs_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                    'cache_topdir': '/var/cache/mock',
                    'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                    'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                    'root': 'fedora-rawhide-x86_64'},
 'package_state_enable': True,
 'package_state_opts': {'available_pkgs': False,
                        'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                        'cache_topdir': '/var/cache/mock',
                        'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                        'installed_pkgs': True,
                        'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                        'root': 'fedora-rawhide-x86_64'},
 'pm_request_enable': False,
 'pm_request_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                     'cache_topdir': '/var/cache/mock',
                     'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                     'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                     'root': 'fedora-rawhide-x86_64'},
 'root_cache_enable': True,
 'root_cache_opts': {'age_check': True,
                     'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                     'cache_topdir': '/var/cache/mock',
                     'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                     'compress_program': 'pigz',
                     'decompress_program': None,
                     'dir': '%(cache_topdir)s/%(root)s/root_cache/',
                     'exclude_dirs': ['./proc',
                                      './sys',
                                      './dev',
                                      './tmp/ccache',
                                      './var/cache/yum',
                                      './var/cache/dnf'],
                     'extension': '.gz',
                     'max_age_days': 15,
                     'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                     'root': 'fedora-rawhide-x86_64',
                     'tar': 'gnutar'},
 'selinux_enable': True,
 'selinux_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                  'cache_topdir': '/var/cache/mock',
                  'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                  'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                  'root': 'fedora-rawhide-x86_64'},
 'sign_enable': False,
 'sign_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
               'cache_topdir': '/var/cache/mock',
               'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
               'cmd': 'rpmsign',
               'opts': '--addsign %(rpms)s',
               'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
               'root': 'fedora-rawhide-x86_64'},
 'tmpfs_enable': False,
 'tmpfs_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                'cache_topdir': '/var/cache/mock',
                'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                'keep_mounted': False,
                'max_fs_size': None,
                'mode': '0755',
                'required_ram_mb': 900,
                'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                'root': 'fedora-rawhide-x86_64'},
 'yum_cache_enable': True,
 'yum_cache_opts': {'basedir': '/var/lib/mock/fedora-rawhide-x86_64',
                    'cache_topdir': '/var/cache/mock',
                    'cachedir': '/var/cache/mock/fedora-rawhide-x86_64',
                    'dir': '%(cache_topdir)s/%(root)s/%(package_manager)s_cache/',
                    'max_age_days': 30,
                    'max_metadata_age_days': 30,
                    'online': True,
                    'package_manager': 'dnf',
                    'resultdir': '/var/lib/mock/fedora-rawhide-x86_64/result',
                    'root': 'fedora-rawhide-x86_64',
                    'target_dir': '/var/cache/%(package_manager)s/'}}
config_opts['plugin_dir'] = '/usr/lib/python3.7/site-packages/mockbuild/plugins'
config_opts['plugins'] = ['tmpfs',
 'root_cache',
 'yum_cache',
 'bind_mount',
 'ccache',
 'selinux',
 'package_state',
 'chroot_scan',
 'lvm_root',
 'compress_logs',
 'sign',
 'pm_request',
 'hw_info',
 'mount',
 'overlayfs']
config_opts['post_install'] = False
config_opts['print_main_output'] = True
config_opts['priorities.conf'] = '\n[main]\nenabled=0'
config_opts['releasever'] = '28'
config_opts['resultdir'] = '%(basedir)s/%(root)s/result'
config_opts['rhnplugin.conf'] = '\n[main]\nenabled=0'
config_opts['root'] = 'fedora-rawhide-x86_64'
config_opts['root_log_fmt_name'] = 'detailed'
config_opts['root_log_fmt_str'] = '%(levelname)s %(filename)s:%(lineno)d:  %(message)s'
config_opts['rpm_command'] = '/bin/rpm'
config_opts['rpmbuild_arch'] = 'x86_64'
config_opts['rpmbuild_command'] = '/usr/bin/rpmbuild'
config_opts['rpmbuild_networking'] = True
config_opts['rpmbuild_timeout'] = 0
config_opts['scm'] = False
config_opts['scm_opts'] = {'cvs_get': 'cvs -d /srv/cvs co SCM_BRN SCM_PKG',
 'distgit_get': 'rpkg clone -a --branch SCM_BRN SCM_PKG SCM_PKG',
 'distgit_src_get': 'rpkg sources',
 'exclude_vcs': True,
 'ext_src_dir': '/dev/null',
 'git_get': 'git clone SCM_BRN git://localhost/SCM_PKG.git SCM_PKG',
 'git_timestamps': False,
 'method': 'git',
 'spec': 'SCM_PKG.spec',
 'svn_get': 'svn co file:///srv/svn/SCM_PKG/SCM_BRN SCM_PKG',
 'write_tar': False}
config_opts['state_log_fmt_name'] = 'state'
config_opts['state_log_fmt_str'] = '%(asctime)s - %(message)s'
config_opts['subscription-manager.conf'] = ''
config_opts['system_dnf_command'] = '/usr/bin/dnf'
config_opts['system_yum_command'] = '/usr/bin/yum'
config_opts['tar'] = 'gnutar'
config_opts['target_arch'] = 'x86_64'
config_opts['update_before_build'] = True
config_opts['use_bootstrap_container'] = False
config_opts['use_container_host_hostname'] = True
config_opts['use_host_resolv'] = True
config_opts['use_nspawn'] = True
config_opts['useradd'] = '/usr/sbin/useradd -o -m -u %(uid)s -g %(gid)s -d %(home)s -n %(user)s'
config_opts['verbose'] = 1
config_opts['version'] = '1.4.16'
config_opts['yum.conf'] = ('\n'
 '[main]\n'
 'keepcache=1\n'
 'debuglevel=2\n'
 'reposdir=/dev/null\n'
 'logfile=/var/log/yum.log\n'
 'retries=20\n'
 'obsoletes=1\n'
 'gpgcheck=0\n'
 'assumeyes=1\n'
 'syslog_ident=mock\n'
 'syslog_device=\n'
 '# default\n'
 '#install_weak_deps=0\n'
 '# Same for actual dnf.\n'
 'install_weak_deps=1\n'
 'metadata_expire=0\n'
 'best=1\n'
 '\n'
 '# repos\n'
 '\n'
 '[fedora]\n'
 'name=fedora\n'
 'metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=$basearch\n'
 'failovermethod=priority\n'
 'enabled=1\n'
 '\n'
 '[local]\n'
 'name=local\n'
 'baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/\n'
 'cost=2000\n'
 '# Disable usunally for performance. Enable to get latest version.\n'
 'enabled=0\n'
 '\n'
 '[result]\n'
 'name=result\n'
 'baseurl=file:///var/lib/mock/fedora-rawhide-x86_64/result\n'
 '# Disable if use copr repo.\n'
 'enabled=0\n'
 'metadata_expire=0\n'
 '\n'
 '[jaruga-staging]\n'
 'name=Copr repo for staging owned by jaruga\n'
 'baseurl=https://copr-be.cloud.fedoraproject.org/results/jaruga/staging/fedora-rawhide-$basearch/\n'
 'type=rpm-md\n'
 'skip_if_unavailable=True\n'
 'gpgcheck=1\n'
 'gpgkey=https://copr-be.cloud.fedoraproject.org/results/jaruga/staging/pubkey.gpg\n'
 'repo_gpgcheck=0\n'
 'enabled=0\n'
 'enabled_metadata=1\n'
 '\n'
 '[pvalena-ruby-on-rails]\n'
 'name=Copr repo for ruby-on-rails owned by pvalena\n'
 'baseurl=https://copr-be.cloud.fedoraproject.org/results/pvalena/ruby-on-rails/fedora-rawhide-$basearch/\n'
 'type=rpm-md\n'
 'skip_if_unavailable=True\n'
 'gpgcheck=1\n'
 'gpgkey=https://copr-be.cloud.fedoraproject.org/results/pvalena/ruby-on-rails/pubkey.gpg\n'
 'repo_gpgcheck=0\n'
 'enabled=0\n'
 'enabled_metadata=1\n'
 '\n'
 '[fedora-debuginfo]\n'
 'name=Fedora Rawhide - x86_64 - Debug\n'
 'failovermethod=priority\n'
 'metalink=https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=$basearch\n'
 'enabled=1\n'
 '\n'
 '# bazel\n'
 '[vbatts-bazel]\n'
 'name=Copr repo for bazel owned by vbatts\n'
 'baseurl=https://copr-be.cloud.fedoraproject.org/results/vbatts/bazel/fedora-rawhide-$basearch/\n'
 'type=rpm-md\n'
 'skip_if_unavailable=True\n'
 'gpgcheck=1\n'
 'gpgkey=https://copr-be.cloud.fedoraproject.org/results/vbatts/bazel/pubkey.gpg\n'
 'repo_gpgcheck=0\n'
 'enabled=0\n'
 'enabled_metadata=1')
config_opts['yum_builddep_command'] = '/usr/bin/yum-builddep'
config_opts['yum_builddep_opts'] = []
config_opts['yum_command'] = '/usr/bin/yum'
config_opts['yum_common_opts'] = []
config_opts['yum_install_command'] = 'install yum yum-utils shadow-utils distribution-gpg-keys'
Finish: run

Thanks for your help!

[xsuchy]

I cannot reproduce. :(
You can try the same with --bootstrap-chroot

[junaruga]

Thanks for the advice. I tried the --bootstrap-chroot. But I got similar error.

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 --scrub=all
$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 --bootstrap-chroot shell

I also checked with debug options, but I could not find the cause of the error. :<

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 --trace -v shell 2>&1 | tee mock.log

[voxik]

I would be surprised if this worked. It takes system, i.e. x86_64 DNF, and tries to execute scriptlets which are aarch64 native code ...

[xsuchy]

I just sit together with @junaruga and we found no differences. We come as deep as that:

 sudo chroot /var/lib/mock/fedora-30-aarch64/root /usr/sbin/groupadd

on my system execute groupadd while on @junaruga it prints No such file or directory.

And another strange thing that while dnf claims that glibc is installed (and is trying postintall scriptlet) there is missing /var/lib/mock/fedora-30-aarch64/root/sbin/glibc_post_upgrade.aarch64 file on @junaruga machine. Dunno why.

[junaruga]

I remembered I did run below program related to qemu.

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

That's to execute below scripts to register "binfmt-conf" strings to /proc/sys/fs/binfmt_misc/register and emulate a different arch's container image.

I suspect this is related to the issue.

https://github.com/multiarch/qemu-user-static/blob/master/register/Dockerfile
https://github.com/multiarch/qemu-user-static/blob/master/register/register.sh
https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh

@xsuchy do you have below file and mount entry on your environment?

$ ls -l /proc/sys/fs/binfmt_misc/register
--w------- 1 root root 0 Jun 11 15:27 /proc/sys/fs/binfmt_misc/register

$ mount | grep binfmt
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=48,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17167)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

On Fedora there is qemu-user-binfmt package that conflicts with qemu-user-static.
https://src.fedoraproject.org/rpms/qemu/blob/master/f/qemu.spec#_567

[xsuchy]

$ ls -l /proc/sys/fs/binfmt_misc/register
--w-------. 1 root root 0 25. čen 13.08 /proc/sys/fs/binfmt_misc/register
$ mount | grep binfmt
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16005)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)

[junaruga]

Today I did run below commands sitting together with @xsuchy .
Interestingly today it worked.

$ mock -r fedora-30-aarch64 --forcearch=aarch64 --scrub=all
$ mock -r fedora-30-aarch64 --forcearch=aarch64 shell
...
  Running scriptlet: glibc-2.29-15.fc30.aarch64                                        31/148 
ERROR: ld.so: object '/var/tmp/tmp.mock.l9e8dfjh/$LIB/nosync.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

  Installing       : bash-5.0.7-1.fc30.aarch64                                         32/148 
  Running scriptlet: bash-5.0.7-1.fc30.aarch64                                         32/148 
  Installing       : libsepol-2.9-1.fc30.aarch64                                       33/148 
  Running scriptlet: libsepol-2.9-1.fc30.aarch64                                       33/148 
ERROR: ld.so: object '/var/tmp/tmp.mock.l9e8dfjh/$LIB/nosync.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

  Installing       : zlib-1.2.11-15.fc30.aarch64                                       34/148 
...
Complete!
Finish: dnf install
Start: creating root cache
Finish: creating root cache
Finish: chroot init
INFO: Installed packages:
Start: shell

<mock-chroot> sh-5.0# uname -m
aarch64

What I did during last time before running the commands are to run below commands.
Because I suspected above manual setting of binfmt might be related to this issue.

$ sudo dnf remove qemu-user-static
$ sudo dnf install qemu-user-binfmt

$ sudo dnf remove qemu-user-binfmt
$ sudo dnf install qemu-user-static

When I faced the issue again, I let you know.
Thanks.

[praiskup]

So we can close this now, right?

[junaruga]

@praiskup I think we can reopen this ticket now. Because I succeeded to reproduce the issue now.

Run below command (detail: https://github.com/multiarch/qemu-user-static)

$ sudo podman run --rm --privileged multiarch/qemu-user-static:register --reset

Then below mock ... shell outputs the error.

$ mock -r fedora-30-aarch64 --forcearch=aarch64 --scrub=all
$ mock -r fedora-30-aarch64 --forcearch=aarch64 shell
...
  Running scriptlet: glibc-2.29-15.fc30.aarch64                                        31/148-
error: failed to exec scriptlet interpreter /usr/sbin/glibc_post_upgrade.aarch64: No such file or directory
warning: %post(glibc-2.29-15.fc30.aarch64) scriptlet failed, exit status 127

Error in POSTIN scriptlet in rpm package glibc
  Installing       : bash-5.0.7-1.fc30.aarch64                                         32/148-
  Running scriptlet: bash-5.0.7-1.fc30.aarch64                                         32/148-
  Installing       : libsepol-2.9-1.fc30.aarch64                                       33/148- 
  Running scriptlet: libsepol-2.9-1.fc30.aarch64                                       33/148-
error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %post(libsepol-2.9-1.fc30.aarch64) scriptlet failed, exit status 127
...

But after running below commands,

$ sudo dnf remove qemu-user-static
$ sudo dnf install qemu-user-static

below mock commands are succeeded.

$ mock -r fedora-30-aarch64 --forcearch=aarch64 --scrub=all
$ mock -r fedora-30-aarch64 --forcearch=aarch64 shell
...
<mock-chroot> sh-5.0# uname -m
aarch64

multiarch/qemu-user-static:register

What sudo podman run --rm --privileged multiarch/qemu-user-static:register --reset is doing is like below command, to register settings (text) to binfmt_misc.

echo ":qemu-$cpu:M::$magic:$mask:$qemu:$flags" > /proc/sys/fs/binfmt_misc/register

Here are the actual messages for the command.

$ sudo podman run --rm --privileged multiarch/qemu-user-static:register --reset
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k

[junaruga]

I talked with @xsuchy, We agreed for keeping closing the ticket for now.
Because it's more than mock's issue. At least I need to simplify the reproducer script.

[junaruga]

I found the essential reason of this issue. I share it for someone who faced this issue.

Feodra's qemu-user-static RPMs install /proc/sys/fs/binfmt_misc/qemu-$cpu files as dnf install qemu-user-static's post scriptlet.

But when other programs update the files from flags: F to flags: (empty flag), this issue happens.

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

The meaning of the flag: F is to keep the qemu setting for persistent on memory.
https://www.kernel.org/doc/html/latest/admin-guide/binfmt-misc.html
But if the flag F is not set, interpreter /usr/bin/qemu-aarch64-static needs to be exist on the chroot environment when executing aarch64 binaries on the chroot in my understanding.

Initial condition.

$ sudo dnf install qemu-user-static

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

Below is ok case.

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell

<mock-chroot> sh-5.0# uname -m
aarch64
<mock-chroot> sh-5.0# exit
logout

Now update the file from flags: F to flags: .
Open 2 terminals for root user and normal user. sudo does not work for "# ..." command.

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: F
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
cat: /proc/sys/fs/binfmt_misc/qemu-aarch64: No such file or directory

# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

$ cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags: 
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

Then

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell
...
Start: cleaning dnf metadata
Finish: cleaning dnf metadata
INFO: enabled HW Info plugin
ERROR: Command failed: 
 # /usr/bin/systemd-nspawn -q -M e2e411dedf644ceca57cd8e6ca2d6c0e -D /var/lib/mock/fedora-rawhide-aarch64/root -a --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/builddir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=en_US.UTF-8 --setenv=LD_PRELOAD=/var/tmp/tmp.mock.h2ikb3kx/$LIB/nosync.so /usr/sbin/usermod -u 28707 --non-unique mockbuild

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 --scrub=all

$ mock -r fedora-rawhide-aarch64 --forcearch=aarch64 shell
error: lua script failed: [string "%post(glibc-2.29.9000-34.fc31.aarch64)"]:6: /usr/sbin/ldconfig: No such file or directory
error: No exec() called after fork() in lua scriptlet
error: lua script failed: [string "%post(glibc-2.29.9000-34.fc31.aarch64)"]:6: /usr/sbin/iconvconfig: No such file or directory
error: No exec() called after fork() in lua scriptlet
...
Failed:
  ca-certificates-2019.2.32-2.fc31.noarch          libutempter-1.1.6-16.fc30.aarch64         

Error: Transaction failed

The solution is as I mentioned, below command.
sudo dnf remove qemu-user-static do nothing for /proc/sys/fs/binfmt_misc/qemu-$cpu files, But sudo dnf install qemu-user-static recreate the files.

$ sudo dnf remove qemu-user-static
$ sudo dnf install qemu-user-static

or restart systemd-binfmt service. The service recreates /proc/sys/fs/binfmt_misc/qemu-aarch64 file from /usr/lib/binfmt.d/qemu-aarch64-static.conf file that is included in qemu-user-static RPM.
sudo systemctl start systemd-binfmt recreates those files.

$ sudo systemctl restart systemd-binfmt

I opened the ticket to discuss the qemu-user-static's behavior on Fedora qemu-user-static.
https://bugzilla.redhat.com/show_bug.cgi?id=1732178

[junaruga]

But if the flag F is not set, interpreter /usr/bin/qemu-aarch64-static needs to be exist on the chroot environment when executing aarch64 binaries on the chroot in my understanding.

I experimented above using mock's chroot.

First, okay case with default /proc/sys/fs/binfmt_misc/qemu-aarch64 .

$ sudo chroot /var/lib/mock/fedora-rawhide-aarch64/root /usr/bin/uname -m
aarch64

Now update the file from flags: F to flags: (empty flag).

# echo -1 > /proc/sys/fs/binfmt_misc/qemu-aarch64
# echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:" > /proc/sys/fs/binfmt_misc/register

Below command is failed, because when the flag F is not set, "interpreter /usr/bin/qemu-aarch64-static" is need inside of the chroot to run the emulation.

$ sudo chroot /var/lib/mock/fedora-rawhide-aarch64/root /usr/bin/uname -m
chroot: failed to run command ‘/usr/bin/uname’: No such file or directory

qemu-aarch64-static does not exist.

$ ls /var/lib/mock/fedora-rawhide-aarch64/root/usr/bin/qemu-aarch64-static
ls: cannot access '/var/lib/mock/fedora-rawhide-aarch64/root/usr/bin/qemu-aarch64-static': No such file or directory

But after copying qemu-aarch64-static binary to chroot, below sudo chroot ... works.

$ sudo cp -p /usr/bin/qemu-aarch64-static /var/lib/mock/fedora-rawhide-aarch64/root/usr/bin/

$ ls /var/lib/mock/fedora-rawhide-aarch64/root/usr/bin/qemu-aarch64-static
/var/lib/mock/fedora-rawhide-aarch64/root/usr/bin/qemu-aarch64-static*

$ sudo chroot /var/lib/mock/fedora-rawhide-aarch64/root /usr/bin/uname -m
aarch64

[xsuchy]

Wow. Nice findings. Thank you.

[surfzoid]

Hi, i'm running Mageia9, none above workaround worked, because to recreate /proc/sys/fs/binfmt_misc/qemu* i need to reboot the computer :-D