Alpine Linux Template #7323

ayakael · 2022-03-05T00:49:01Z

description

QubeOS should have an Alpine Linux template. This has been requested in the past, and I've finally decided to tackle it, having packaging experience with this distro. This space is for tracking issues.

current state

xorg works
memory ballooning works
copy / paste works + qvm-copy
OpenRC scripts generally working, but things lack finesse
tempate builder not implemented yet
automatic assignment of ip address broken (now fixed)
apk does not work on templates without networking

steps

Create qubes-vm packages (repo)
Test packages within Alpine HVM with the goal of creating a functional TemplateVM
Create builder-alpine (repo)
Test within qubes-builder environment with the goal of creating a tar.gz file that matches TemplateVM on test system

hurdles

Alpine uses OpenRC as its init system, while afaik every other template uses Systemd. A major part of the work will be updating existing initd scripts, and creating them where they don't exist.
Alpine uses musl rather than libc, but this doesn't seem to be a major issue.

ayakael · 2022-03-05T00:56:22Z

So far, all of the packages have been implemented as an APKBUILD. I'm currently working on initd scripts, with the following, as far as I can tell, mostly working:

xendriverdomain
qubes-db
qubes-meminfo-writer
qubes-sysinit
qubes-qrexec-agent

My current roadblock is qubes-gui-agent. It is stuck at Waiting on /var/run/xf86-qubes-socket socket. Anyone can give me any pointers on how I should debug this? The initd script is as is:

#!/sbin/openrc-run
  
name=$RC_SVCNAME
cfgfile="/etc/qubes/$RC_SVCNAME.conf"
command="/usr/bin/qubes-gui"
pidfile="/run/qubes/$RC_SVCNAME.pid"
command_background="yes"
error_log=/var/log/qubes/$RC_SVCNAME.log

depend() {
        need qubes-db
}

start_pre() {
	checkpath --directory --owner $command_user:$command_user --mode 0775 \
                /run/qubes /var/log/qubes /var/run/console/user
	# start console-kit-daemon
	/usr/bin/ck-list-sessions > /dev/null 2>&1
	# pretend tha user is at local console
	touch /var/run/console/user
	/bin/sh -c /usr/lib/qubes/qubes-gui-agent-pre.sh
	. /var/run/qubes-service-environment
	command_args="$command_args $GUI_OPTS"
	export DISPLAY=:0
}

moiselazarus · 2022-03-05T14:41:06Z

Thank you for working on this. I can't help you with this issue, but maybe this link, how Void Linux done this, can help you.
https://github.com/Nexolight/void-tainted-pkgs/blob/qubes/srcpkgs/qubes-gui-agent-linux/files/qubes-gui-agent/run
https://github.com/Nexolight/void-tainted-pkgs/tree/qubes/QubesOS

ayakael · 2022-03-08T20:34:53Z

Thanks @moiselazarus! I threw an afternoon on it with no success, but that void has suceeded in packaging for Qubes before helps a lot! I havn't implemented the one-shot scripts adequately, so that's where I'm at.

ayakael · 2022-06-20T17:46:56Z

Back on trying to figure this out. Despite implementing the one-shot scripts, I still am facing the Waiting on /var/run/xf86-qubes-socket socket issue. Anyone could fill me in to exactly what qubes-gui-agent is waiting on, or could give me pointers on where to go to debug this next. Of note, I am running off of the Alpine kernel, as the qubes kernels don't work

ayakael · 2022-06-20T22:50:25Z

I've made more progress: hvm mode with qubes kernel works using additional kernelopts modules=ext4 rootfstype=ex4 after introducting udev in the initramfs and thus fixing /dev/mapper/dmroot generation. pvh mode is broken, hanging after boot. Still stuck on waiting qubes-socket error ...

ayakael · 2022-06-21T07:18:40Z

@Nexolight Would you have encountered the issue of Waiting on /var/run/xf86-qubes-socket socket when creating the void template?

marmarek · 2022-06-21T09:30:59Z

Sounds like either Xorg not starting (or crashing on start?) or not loading qubes driver (generic config file instead of the one from qubes-gui-agent?).

ayakael · 2022-06-21T19:20:52Z

The /etc/X11/xorg-qubes.conf is the same as my Fedora template, so it can't be the generic config file. Driver are also available at /usr/lib/xorg/modules. At every start, the config file is updated, as well. It just seems like Xorg is never started, as there are no Xorg log (or error) files. In command line qubes-gui-runuser is in sleep (Z) mode.

ayakael · 2022-06-22T17:08:54Z

Of note, /usr/bin/qubes-gui-runuser /usr/bin/xinit /etc/X11/xinit/xinitrc -- /usr/libexec/Xorg :0 -nolisten tcp vt07 -wr -config /etc/X11/xorg-qubes.conf fails with exit code 255 when executed from shell.

ayakael · 2022-06-28T14:29:11Z

Good news! Tracked down the issue - it isn't a problem with Xorg, but rather with pam. There needs to be an Alpine version of /etc/pam.d/qubes-gui-agent. I've been trying for the last hour to throw spaguetti at the wall, see if it stick, but alas I do not know anything about pam. I see that they are copies of /etc/pam.d/su, so I went ahead and copied that, but now I get a execve cmd: Permission denied error.

ayakael · 2022-06-28T14:42:48Z

Doesn't seem to have to do with pam anymore (edit: or maybe it does??)

strace:

execve("/usr/bin/qubes-gui-runuser", ["/usr/bin/qubes-gui-runuser", "user", "/etc/X11", "/bin/sh", "-l", "-c", "exec /usr/bin/xinit /etc/X11/xin"...], 0x7ffdc547b8d0 /* 23 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x77e0c74a5b48) = 0
set_tid_address(0x77e0c74a5f90)         = 2011
brk(NULL)                               = 0x64cf75ec6000
brk(0x64cf75ec8000)                     = 0x64cf75ec8000
mmap(0x64cf75ec6000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x64cf75ec6000
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libpam.so.0", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=59288, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\3004\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 65536, PROT_READ, MAP_PRIVATE, 3, 0) = 0x77e0c73ff000
mmap(0x77e0c7402000, 28672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x77e0c7402000
mmap(0x77e0c7409000, 12288, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0xa000) = 0x77e0c7409000
mmap(0x77e0c740d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0xd000) = 0x77e0c740d000
close(3)                                = 0
mprotect(0x77e0c740d000, 4096, PROT_READ) = 0
mprotect(0x77e0c74a2000, 4096, PROT_READ) = 0
mprotect(0x64cf7503c000, 4096, PROT_READ) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1 RT_2], NULL, 8) = 0
rt_sigaction(SIGTERM, {sa_handler=0x64cf7503a506, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x77e0c7455c8a}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGHUP, {sa_handler=0x64cf7503a506, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x77e0c7455c8a}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:0:root:/root:/bin/ash\nb"..., 1024) = 1024
read(3, "gin\nsmmsp:x:209:209:smmsp:/var/s"..., 1024) = 579
lseek(3, -325, SEEK_CUR)                = 1278
close(3)                                = 0
munmap(0x77e0c73fe000, 4096)            = 0
stat("/etc/pam.d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/etc/pam.d/qubes-gui-agent", O_RDONLY|O_LARGEFILE) = 3
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
read(3, "# basic PAM configuration for Al"..., 1024) = 203
open("/lib/security/pam_rootok.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 4
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
fstat(4, {st_mode=S_IFREG|0755, st_size=14152, ...}) = 0
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\20\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 4, 0) = 0x77e0c73f9000
mmap(0x77e0c73fa000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 4, 0x1000) = 0x77e0c73fa000
mmap(0x77e0c73fb000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 4, 0x2000) = 0x77e0c73fb000
mmap(0x77e0c73fc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x2000) = 0x77e0c73fc000
close(4)                                = 0
mprotect(0x77e0c73fc000, 4096, PROT_READ) = 0
open("/etc/pam.d/base-auth", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 437
open("/lib/security/pam_env.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=18248, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\21\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 24576, PROT_READ, MAP_PRIVATE, 5, 0) = 0x77e0c73f3000
mmap(0x77e0c73f4000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 5, 0x1000) = 0x77e0c73f4000
mmap(0x77e0c73f6000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 5, 0x3000) = 0x77e0c73f6000
mmap(0x77e0c73f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x3000) = 0x77e0c73f7000
close(5)                                = 0
mprotect(0x77e0c73f7000, 4096, PROT_READ) = 0
open("/lib/security/pam_unix.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=47000, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0%\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 102400, PROT_READ, MAP_PRIVATE, 5, 0) = 0x77e0c73da000
mmap(0x77e0c73dc000, 24576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x77e0c73dc000
mmap(0x77e0c73e2000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED, 5, 0x8000) = 0x77e0c73e2000
mmap(0x77e0c73e5000, 57344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0xa000) = 0x77e0c73e5000
mmap(0x77e0c73e7000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x77e0c73e7000
close(5)                                = 0
mprotect(0x77e0c73e5000, 4096, PROT_READ) = 0
open("/lib/security/pam_nologin.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=14152, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\21\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 5, 0) = 0x77e0c73d5000
mmap(0x77e0c73d6000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 5, 0x1000) = 0x77e0c73d6000
mmap(0x77e0c73d7000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x77e0c73d7000
mmap(0x77e0c73d8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x77e0c73d8000
close(5)                                = 0
mprotect(0x77e0c73d8000, 4096, PROT_READ) = 0
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-account", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 69
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-password", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 87
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-session-noninteractive", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 101
open("/lib/security/pam_limits.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFREG|0755, st_size=22424, ...}) = 0
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\"\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 28672, PROT_READ, MAP_PRIVATE, 5, 0) = 0x77e0c73ce000
mmap(0x77e0c73d0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 5, 0x2000) = 0x77e0c73d0000
mmap(0x77e0c73d2000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 5, 0x4000) = 0x77e0c73d2000
mmap(0x77e0c73d3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 5, 0x4000) = 0x77e0c73d3000
close(5)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73cd000
mprotect(0x77e0c73d3000, 4096, PROT_READ) = 0
read(4, "", 1024)                       = 0
close(4)                                = 0
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x77e0c73fe000, 4096)            = 0
open("/etc/pam.d/other", O_RDONLY|O_LARGEFILE) = 3
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
read(3, "# basic PAM configuration for Al"..., 1024) = 173
open("/etc/pam.d/base-auth", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 437
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73cc000
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-account", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 69
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-password", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 87
read(4, "", 1024)                       = 0
close(4)                                = 0
open("/etc/pam.d/base-session-noninteractive", O_RDONLY|O_LARGEFILE) = 4
read(4, "# basic PAM configuration for Al"..., 1024) = 101
read(4, "", 1024)                       = 0
close(4)                                = 0
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x77e0c73fe000, 4096)            = 0
ioctl(0, TIOCGWINSZ, {ws_row=25, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
ioctl(0, TIOCGWINSZ, {ws_row=25, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0
readlink("/proc/self/fd/0", "/dev/tty1", 32) = 9
stat("/dev/tty1", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x4, 0x1), ...}) = 0
fstat(0, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x4, 0x1), ...}) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1656427127, tv_nsec=903365906}) = 0
getuid()                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory)
close(3)                                = 0
munmap(0x77e0c73fe000, 4096)            = 0
open("/etc/group", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:root\nbin:x:1:root,bin,d"..., 1024) = 919
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x77e0c73fe000, 4096)            = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[], NULL, 8) = 0
setgroups(1, [1000])                    = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73fe000
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:0:root:/root:/bin/ash\nb"..., 1024) = 1024
read(3, "gin\nsmmsp:x:209:209:smmsp:/var/s"..., 1024) = 579
lseek(3, -325, SEEK_CUR)                = 1278
close(3)                                = 0
prlimit64(0, RLIMIT_CPU, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_FSIZE, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_DATA, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_CORE, NULL, {rlim_cur=0, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_RSS, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_NPROC, NULL, {rlim_cur=3625, rlim_max=3625}) = 0
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=4*1024}) = 0
prlimit64(0, RLIMIT_MEMLOCK, NULL, {rlim_cur=8192*1024, rlim_max=8192*1024}) = 0
prlimit64(0, RLIMIT_AS, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_LOCKS, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
prlimit64(0, RLIMIT_SIGPENDING, NULL, {rlim_cur=3625, rlim_max=3625}) = 0
prlimit64(0, RLIMIT_MSGQUEUE, NULL, {rlim_cur=800*1024, rlim_max=800*1024}) = 0
prlimit64(0, RLIMIT_NICE, NULL, {rlim_cur=0, rlim_max=0}) = 0
prlimit64(0, RLIMIT_RTPRIO, NULL, {rlim_cur=0, rlim_max=0}) = 0
prlimit64(0, RLIMIT_RTTIME, NULL, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
getpriority(PRIO_PROCESS, 0)            = 20
open("/etc/security/limits.conf", O_RDONLY|O_LARGEFILE) = 3
read(3, "# /etc/security/limits.conf\n#\n#T"..., 1024) = 1024
read(3, " the soft limits\n#        - \"har"..., 1024) = 1024
read(3, "ity\n#\n#<domain>      <type>  <it"..., 1024) = 378
read(3, "", 1024)                       = 0
close(3)                                = 0
open("/etc/security/limits.d/", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e0c73ca000
getdents64(3, 0x77e0c73ca0b8 /* 3 entries */, 2048) = 88
getdents64(3, 0x77e0c73ca0b8 /* 0 entries */, 2048) = 0
close(3)                                = 0
munmap(0x77e0c73ca000, 8192)            = 0
open("/etc/security/limits.d/90-qubes-gui.conf", O_RDONLY|O_LARGEFILE) = 3
read(3, "# Qubes GUI agent needs to mlock"..., 1024) = 158
read(3, "", 1024)                       = 0
close(3)                                = 0
prlimit64(0, RLIMIT_MEMLOCK, {rlim_cur=131072*1024, rlim_max=131072*1024}, NULL) = 0
setpriority(PRIO_PROCESS, 0, 0)         = 0
getuid()                                = 0
open("/etc/login.defs", O_RDONLY|O_LARGEFILE) = 3
read(3, "USERGROUPS_ENAB yes\n", 1024)  = 20
read(3, "", 1024)                       = 0
close(3)                                = 0
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
read(3, "root:x:0:0:root:/root:/bin/ash\nb"..., 1024) = 1024
read(3, "gin\nsmmsp:x:209:209:smmsp:/var/s"..., 1024) = 579
lseek(3, -325, SEEK_CUR)                = 1278
close(3)                                = 0
getuid()                                = 0
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/dev/log"}, 12) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1656427127, tv_nsec=906160763}) = 0
sendto(3, "<86>Jun 28 14:38:47 : pam_unix(q"..., 107, 0, NULL, 0) = 107
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
fork()                                  = 2012
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
wait4(2012, [{WIFEXITED(s) && WEXITSTATUS(s) == 127}], 0, NULL) = 2012
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2012, si_uid=1000, si_status=127, si_utime=0, si_stime=0} ---
getuid()                                = 0
open("/etc/login.defs", O_RDONLY|O_LARGEFILE) = 4
read(4, "USERGROUPS_ENAB yes\n", 1024)  = 20
read(4, "", 1024)                       = 0
close(4)                                = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1656427127, tv_nsec=909603199}) = 0
sendto(3, "<86>Jun 28 14:38:47 : pam_unix(q"..., 86, 0, NULL, 0) = 86
munmap(0x77e0c73fe000, 4096)            = 0
munmap(0x77e0c73cc000, 4096)            = 0
exit_group(127)                         = ?
+++ exited with 127 +++

moiselazarus · 2022-06-30T11:25:27Z

Can you push the openrc files to your repo. I would like to test, maybe some hints or steps to build the template.

ayakael · 2022-07-06T17:00:16Z

@moiselazarus Apologies for the delay, I was without my laptop for a little while. You will find the repo updated with latest openrc files. Following this guide to setting up an Alpine HVM proved useful: https://github.com/Nexolight/void-tainted-pkgs/tree/qubes/QubesOS. A few notes:

kernelopts should be set via qvm-prefs -s $vmname "kernelopts modules=ext4 rootfstype=ext4"
There should be no need for linux-qubes, as qvm-vm-core now sets CONFIG_TRANSPARENT_HUGEPAGE_MADVISE as Y at startup
qubes-mkinitfs is needed to allow inclusion of udev in initramfs. Without this, /dev/mapper/dm_mapper is never created in standalone VMs.
Make sure created partition is GPT with following partition table:
xvda1 200M EFI System
xvda2 2048K BIOS boot partition
xvda3 (whatever) Root filesystem
(note the name for xvda3 as "Root filesystem")
To create getty@hvc0 (xen tty), add to /etc/inittab: hvc0::respawn:/sbin/getty -L hvc0 115200 vt220
make sure to rc-update add udev + udev-trigger
make sure to add "eudev" to /etc/mkinitfs/mkinitfs.conf features line and regenerate initramfs with mkinitfs

By memory, that should get your VM to a similar state as mine. Don't desitate in case of questions

ayakael · 2022-07-11T21:13:53Z

Current state of qubes-gui-runuser issue:

fixed error connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory) by installing musl-nscd, executing echo 'hosts: files dns' > /etc/nsswitch.conf, and executing nscd
fixed error open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory) by executing echo '/lib:/usr/local/lib:/usr/lib' > /etc/ld-musl-x86_64.path

That the issue still persists suggests that these error were non-fatal.

@marmarek Having created qubes-gui-runuser binary, would you have any pointers as to the source of the wait4(1994, execve cmd: Permission denied error in the strace?

ayakael · 2022-07-14T14:27:20Z

Debugged the issue: the qubes-gui-runuser issue had to do with two things: one was an error of mine in misunderstanding the command syntax, and the other was that qubes-run-xorg was trying to start xorg with ash (via /bin/sh link) rather than with bash. Of course, it'd be too easy for it to just work, as Xorg runs now but then crashes shortly after. See log below:

[    19.332] 
X.Org X Server 1.21.1.3
X Protocol Version 11, Revision 0
[    19.332] Current Operating System: Linux alpine 5.16.18-2.fc32.qubes.x86_64 #1 SMP PREEMPT Fri Apr 1 22:28:01 CEST 2022 x86_64
[    19.332] Kernel command line: root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 xen_scrub_pages=0 modules=ext4 rootfstype=ext4
[    19.332]  
[    19.332] Current version of pixman: 0.40.0
[    19.332] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[    19.332] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[    19.332] (==) Log file: "/home/user/.local/share/xorg/Xorg.0.log", Time: Thu Jul 14 16:15:09 2022
[    19.332] (++) Using config file: "/etc/X11/xorg-qubes.conf"
[    19.332] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[    19.332] (==) ServerLayout "Default Layout"
[    19.332] (**) |-->Screen "Screen0" (0)
[    19.332] (**) |   |-->Monitor "Monitor0"
[    19.332] (**) |   |-->Device "Videocard0"
[    19.332] (**) |-->Input Device "qubesdev"
[    19.332] (==) Automatically adding devices
[    19.332] (==) Automatically enabling devices
[    19.332] (==) Automatically adding GPU devices
[    19.332] (==) Automatically binding GPU devices
[    19.332] (==) Max clients allowed: 256, resource mask: 0x1fffff
[    19.333] (WW) The directory "/usr/share/fonts/TTF" does not exist.
[    19.333] 	Entry deleted from font path.
[    19.333] (==) FontPath set to:
	/usr/share/fonts/misc,
	/usr/share/fonts/100dpi:unscaled,
	/usr/share/fonts/75dpi:unscaled,
	/usr/share/fonts/Type1
[    19.333] (==) ModulePath set to "/usr/lib/xorg/modules"
[    19.333] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[    19.333] (II) Module ABI versions:
[    19.333] 	X.Org ANSI C Emulation: 0.4
[    19.333] 	X.Org Video Driver: 25.2
[    19.333] 	X.Org XInput driver : 24.4
[    19.333] 	X.Org Server Extension : 10.0
[    19.334] (--) PCI:*(0@0:4:0) 1234:1111:1af4:1100 rev 2, Mem @ 0xf1000000/16777216, 0xf2012000/4096, BIOS @ 0x????????/131072
[    19.334] (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
[    19.334] (II) "glx" will be loaded by default.
[    19.334] (II) LoadModule: "fb"
[    19.334] (II) Module "fb" already built-in
[    19.334] (II) LoadModule: "glx"
[    19.335] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[    19.345] (II) Module glx: vendor="X.Org Foundation"
[    19.345] 	compiled for 1.21.1.3, module version = 1.0.0
[    19.345] 	ABI class: X.Org Server Extension, version 10.0
[    19.345] (II) LoadModule: "dummyqbs"
[    19.345] (II) Loading /usr/lib/xorg/modules/drivers/dummyqbs_drv.so
[    19.345] (II) Module dummyqbs: vendor="X.Org Foundation"
[    19.345] 	compiled for 1.21.1.2, module version = 0.3.6
[    19.345] 	Module class: X.Org Video Driver
[    19.345] 	ABI class: X.Org Video Driver, version 25.2
[    19.345] (II) LoadModule: "qubes"
[    19.345] (II) Loading /usr/lib/xorg/modules/drivers/qubes_drv.so
[    19.345] (II) Module qubes: vendor="X.Org Foundation"
[    19.345] 	compiled for 1.21.1.2, module version = 0.0.1
[    19.345] 	Module class: X.Org XInput Driver
[    19.345] 	ABI class: X.Org XInput driver, version 24.4
[    19.345] (II) DUMMYQBS: Driver for Dummy chipsets: dummy
[    19.345] (WW) Falling back to old probe method for dummyqbs
[    19.345] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support
[    19.345] (II) DUMMYQBS(0): Chipset is a DUMMY
[    19.346] (**) DUMMYQBS(0): Depth 24, (--) framebuffer bpp 32
[    19.346] (==) DUMMYQBS(0): RGB weight 888
[    19.346] (==) DUMMYQBS(0): Default visual is TrueColor
[    19.346] (==) DUMMYQBS(0): Using gamma correction (1.0, 1.0, 1.0)
[    19.346] (DB) xf86MergeOutputClassOptions unsupported bus type 0
[    19.346] (**) DUMMYQBS(0): Option "GUIDomID" "0"
[    19.346] (**) DUMMYQBS(0): VideoRAM: 28000 kByte
[    19.346] (--) DUMMYQBS(0): Max Clock: 300000 kHz
[    19.346] (II) DUMMYQBS(0): Monitor0: Using hsync range of 49.00-50.00 kHz
[    19.346] (II) DUMMYQBS(0): Monitor0: Using vrefresh range of 46.00-47.00 Hz
[    19.346] (II) DUMMYQBS(0): Clock range:  11.00 to 300.00 MHz
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x350" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x175" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x400" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "720x400" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "360x200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x240" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x240" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x240" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x240" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "400x300" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "400x300" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "400x300" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "400x300" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "400x300" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768i" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x384i" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1152x864" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "576x432" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x960" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x960" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x480" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x1024" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x512" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x1024" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x512" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x1024" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x512" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1792x1344" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "896x672" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1792x1344" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "896x672" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1856x1392" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "928x696" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1856x1392" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "928x696" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1440" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1440" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "832x624" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "416x312" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1400x1050" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "700x525" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1400x1050" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "700x525" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1440" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1536" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1536" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1536" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x360" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x180" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x360" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "320x180" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "720x405" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "360x202" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "720x405" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "360x202" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "864x486" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "432x243" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "864x486" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "432x243" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x540" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "480x270" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x540" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "480x270" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x576" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x288" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x576" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "512x288" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x360" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x360" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1368x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "684x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1368x768" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "684x384" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x450" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "800x450" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1080" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x540" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1080" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x540" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1152" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x576" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1152" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1024x576" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1440" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1440" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x720" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2880x1620" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1440x810" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2880x1620" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1440x810" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3200x1800" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3200x1800" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1600x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3840x2160" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1080" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3840x2160" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1080" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "4096x2304" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1152" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "4096x2304" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2048x1152" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "5120x2880" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1440" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "5120x2880" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1440" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "7680x4320" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3840x2160" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "7680x4320" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "3840x2160" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "15360x8640" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "7680x4320" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "15360x8640" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "7680x4320" (insufficient memory for mode)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x800" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x400" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x800" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "640x400" (vrefresh out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1400x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "700x450" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1400x900" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "700x450" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1680x1050" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "840x525" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1680x1050" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "840x525" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1920x1200" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "960x600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1600" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x800" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Not using default mode "2560x1600" (bad mode clock/interlace/doublescan)
[    19.346] (II) DUMMYQBS(0): Not using default mode "1280x800" (hsync out of range)
[    19.346] (II) DUMMYQBS(0): Virtual size is 1920x1080 (pitch 1920)
[    19.346] (**) DUMMYQBS(0): *Mode "QB1920x1080": 96.0 MHz, 49.9 kHz, 46.1 Hz
[    19.346] (II) DUMMYQBS(0): Modeline "QB1920x1080"x46.1   96.00  1920 1921 1922 1923  1080 1081 1082 1083 (49.9 kHz z)
[    19.346] (==) DUMMYQBS(0): DPI set to (96, 96)
[    19.346] (II) Loading sub module "fb"
[    19.346] (II) LoadModule: "fb"
[    19.346] (II) Module "fb" already built-in
[    19.346] (II) Loading sub module "ramdac"
[    19.346] (II) LoadModule: "ramdac"
[    19.346] (II) Module "ramdac" already built-in
[    19.346] (EE) DUMMYQBS(0): Failed to open xengntshr: Permission denied!
[    19.346] (EE) 
Fatal server error:
[    19.346] (EE) AddScreen/ScreenInit failed for driver 0
[    19.346] (EE) 
[    19.346] (EE) 
Please consult the The X.Org Foundation support 
	 at http://wiki.x.org
 for help. 
[    19.346] (EE) Please also check the log file at "/home/user/.local/share/xorg/Xorg.0.log" for additional information.
[    19.347] (EE) 
[    19.347] (EE) Server terminated with error (1). Closing log file.

Failed to open xengntshr: Permission denied! relates to the following bit of code in DUMMYQBS:

    dPtr->xgs = xengntshr_open(NULL, 0);
    if (dPtr->xgs == NULL) {
        xf86DrvMsg(pScrn->scrnIndex, X_ERROR, "Failed to open xengntshr: %s!\n", strerror(errno));
        return FALSE;
    }

Unfortunately, my debugging hits a new wall...

marmarek · 2022-07-14T14:37:04Z

Failed to open xengntshr: Permission denied!

Missing udev rules from https://github.com/QubesOS/qubes-linux-utils/blob/master/udev/udev-qubes-misc.rules ? Or maybe user isn't a member of qubes group?

marmarek · 2022-07-14T14:39:53Z

fixed error connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory) by installing musl-nscd, executing echo 'hosts: files dns' > /etc/nsswitch.conf, and executing nscd

fixed error open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory) by executing echo '/lib:/usr/local/lib:/usr/lib' > /etc/ld-musl-x86_64.path

I don't think any of those require "fixing" - both are optional features and everything should work just fine without them (and indeed the process continues to run past those attempts).

ayakael · 2022-07-14T14:52:16Z

Failed to open xengntshr: Permission denied!

Missing udev rules from https://github.com/QubesOS/qubes-linux-utils/blob/master/udev/udev-qubes-misc.rules ? Or maybe user isn't a member of qubes group?

Adding user to qubes group did the trick, Xorg now works, although qvm-run alpine urxvt does not work. I'm able to execute sudo -u user urxvt from within the VM via xl console alpine, which opens it fortunately.

Another issue is that while qvm-copy to alpine works, it sends file to /root/QubesIncoming.

fixed error connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 24) = -1 ENOENT (No such file or directory) by installing musl-nscd, executing echo 'hosts: files dns' > /etc/nsswitch.conf, and executing nscd

fixed error open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory) by executing echo '/lib:/usr/local/lib:/usr/lib' > /etc/ld-musl-x86_64.path

I don't think any of those require "fixing" - both are optional features and everything should work just fine without them (and indeed the process continues to run past those attempts).

Indeed, this was desperation. I was barking up the wrong tree.

marmarek · 2022-07-14T15:03:30Z

qubes-session process should run as user, and should have that user environment (including HOME=/home/user).

qvm-run alpine urxvt does not work

You can add -p option to get the command output, hopefully that will help. If it just hangs, it probably waits for qubes.WaitForSession service - that would be related to the qubes-session process above, and especially qrexec-fork-server started from there (it should also run as user).

ayakael · 2022-07-14T15:36:04Z

qubes-session process should run as user, and should have that user environment (including HOME=/home/user).

qvm-run alpine urxvt does not work

You can add -p option to get the command output, hopefully that will help. If it just hangs, it probably waits for qubes.WaitForSession service - that would be related to the qubes-session process above, and especially qrexec-fork-server started from there (it should also run as user).

initrd starts qubes-session under user, but it stalls. htop shows xfsettingsd --replace subprocess under qubes-session. XFCE4 is installed. Log:

(gsd-xsettings:2617): Gdk-CRITICAL **: 17:31:26.664: gdk_atom_intern: assertion 'atom_name != NULL' failed

(gsd-xsettings:2617): Gdk-CRITICAL **: 17:31:26.664: gdk_atom_intern: assertion 'atom_name != NULL' failed

(gsd-xsettings:2617): xsettings-plugin-WARNING **: 17:31:26.672: You can only run one xsettings manager at a time; exiting
Failed to start: Could not initialize xsettings manager.
2022-07-14 17:31:26,952 icon-sender: running: ['qrexec-client-vm', 'dom0', 'qubes.WindowIconUpdater']

(evolution-alarm-notify:2568): Gdk-CRITICAL **: 17:31:27.142: gdk_atom_intern: assertion 'atom_name != NULL' failed

(evolution-alarm-notify:2568): Gdk-CRITICAL **: 17:31:27.142: gdk_atom_intern: assertion 'atom_name != NULL' failed

(evolution-alarm-notify:2568): Gdk-WARNING **: 17:31:28.268: ../gdk/x11/gdkproperty-x11.c:224 invalid X atom: 1701998435

(evolution-alarm-notify:2568): Gdk-WARNING **: 17:31:28.268: ../gdk/x11/gdkproperty-x11.c:224 invalid X atom: 1852403568

(xfsettingsd:2526): Gdk-WARNING **: 17:31:28.268: ../gdk/x11/gdkproperty-x11.c:224 invalid X atom: 1701998435

(xfsettingsd:2526): Gdk-WARNING **: 17:31:28.268: ../gdk/x11/gdkproperty-x11.c:224 invalid X atom: 1852403568
xfsettingsd: No window manager registered on screen 0.

(xfsettingsd:2526): xfsettingsd-WARNING **: 17:31:31.629: Failed to get the _NET_NUMBER_OF_DESKTOPS property.

marmarek · 2022-07-14T15:42:35Z

starts qubes-session under user, but it stalls

It should remain running, with various children processes (mostly dependent on /etc/xdg/autostart content). The question is whether qrexec-fork-server is started as one of them.

ayakael · 2022-07-14T18:02:29Z

starts qubes-session under user, but it stalls

It should remain running, with various children processes (mostly dependent on /etc/xdg/autostart content). The question is whether qrexec-fork-server is started as one of them.

Fixed it, it was a permission issue in /var/run/qubes (group owner root, not qubes) making the start of qrexec-fork-server fail

ayakael · 2022-07-16T10:29:11Z

Memory ballooning doesn't work, any pointers that I could follow?

Note that no initcpio hooks have been adapted to Alpine. I suspect it may have something to do with that. Also note, AppVM does announce succesful start, so connection to qmemman should be restarted.

marmarek · 2022-07-16T12:36:04Z

Memory ballooning doesn't work, any pointers that I could follow?

That's about qubes-meminfo-writer service.

ayakael · 2022-07-17T19:41:25Z

Memory ballooning doesn't work, any pointers that I could follow?

That's about qubes-meminfo-writer service.

Indeed, /sbin/meminfo-writer is started on boot, but it doesn't do anything when memory is maxed out via tail /dev/zero, A workaround I've found is executing kill -SIGUSR1 $(cat /run/qubes/qubes-meminfo-writer.pid) as a start_post() command in the OpenRC init script, but that immediately allocates the maximum amount of memory, defeating the point of memory ballooning

ayakael · 2022-07-17T20:40:06Z

After converting to template, qvm-run appvm-alpine-315 urxvt doesn't work in AppVM. qrexec-fork-server gives following error:

2022-07-17 22:33:04.813 qrexec-fork-server[12043]: qrexec-agent-data.c:244:handle_new_process_common: executed: QUBESRPC qubes.VMShell dom0 (pid 12045)
2022-07-17 22:33:04.816 qrexec-fork-server[12043]: qrexec-agent-data.c:272:handle_new_process_common: pid 12045 exited with 2

dom0 error after qvm-run:

Running 'urxvt' on appvm-alpine-315
appvm-alpine-315: command failed with code 2

I can execute urxvt without issue via sudo -u user urxvt through xl console appvm-alpine-315.

ayakael · 2023-08-26T03:22:21Z

I got the template builder ported over. It is available here:
https://lab.ilot.io/ayakael/qubes-builder-alpine

It does not produce a working template yet. While I can build, install, and boot the template, something is still broken as qubes-db refuses to start (fails with xencall could not obtain handle on privileged command interface: No such file or directory).

I've also got the build infrastructure implemented here: https://lab.ilot.io/ayakael/qubes-aports The template builder pulls the packages from that infra and then sets up the image.

All in all, good progress! I'm sure a full day of troubleshooting will be enough to iron out whatever's the problem.

ayakael · 2023-08-26T15:38:59Z

The qubesdb problem is evading me still. I'd appreciate some pointers, here is an strace:

execve("/sbin/qubesdb-daemon", ["/sbin/qubesdb-daemon", "0"], 0x7ffe3b0d96c8 /* 15 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x71e856294b48) = 0
set_tid_address(0x71e856294fb8)         = 1438
brk(NULL)                               = 0x5cf2aa9e2000
brk(0x5cf2aa9e4000)                     = 0x5cf2aa9e4000
mmap(0x5cf2aa9e2000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x5cf2aa9e2000
open("/etc/ld-musl-x86_64.path", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/libvchan-xen.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libvchan-xen.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libvchan-xen.so", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=18336, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 24576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561f5000
mmap(0x71e8561f7000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561f7000
mmap(0x71e8561f8000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x71e8561f8000
mmap(0x71e8561f9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x71e8561f9000
close(3)                                = 0
open("/lib/libxenvchan.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxenvchan.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxenvchan.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=22440, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 28672, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561ee000
mmap(0x71e8561f0000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561f0000
mmap(0x71e8561f2000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x71e8561f2000
mmap(0x71e8561f3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x71e8561f3000
close(3)                                = 0
open("/lib/libxenctrl.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxenctrl.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxenctrl.so.4.17", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=169960, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 176128, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561c3000
mmap(0x71e8561cb000, 102400, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x8000) = 0x71e8561cb000
mmap(0x71e8561e4000, 32768, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x21000) = 0x71e8561e4000
mmap(0x71e8561ec000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x28000) = 0x71e8561ec000
close(3)                                = 0
open("/lib/libxenstore.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxenstore.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxenstore.so.4", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=34808, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 49152, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561b7000
mmap(0x71e8561b9000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561b9000
mmap(0x71e8561bd000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x6000) = 0x71e8561bd000
mmap(0x71e8561bf000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x7000) = 0x71e8561bf000
mmap(0x71e8561c1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x71e8561c1000
close(3)                                = 0
open("/lib/libxengnttab.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxengnttab.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxengnttab.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=18344, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 24576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561b1000
mmap(0x71e8561b2000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e8561b2000
mmap(0x71e8561b4000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x71e8561b4000
mmap(0x71e8561b5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x3000) = 0x71e8561b5000
close(3)                                = 0
open("/lib/libxenevtchn.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxenevtchn.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxenevtchn.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14248, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561ac000
mmap(0x71e8561ad000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e8561ad000
mmap(0x71e8561ae000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561ae000
mmap(0x71e8561af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561af000
close(3)                                = 0
open("/lib/libxentoollog.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxentoollog.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxentoollog.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14248, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561a7000
mmap(0x71e8561a8000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e8561a8000
mmap(0x71e8561a9000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561a9000
mmap(0x71e8561aa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561aa000
close(3)                                = 0
open("/lib/libxencall.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxencall.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxencall.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14248, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e8561a2000
mmap(0x71e8561a3000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e8561a3000
mmap(0x71e8561a4000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561a4000
mmap(0x71e8561a5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561a5000
close(3)                                = 0
open("/lib/libxenforeignmemory.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxenforeignmemory.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxenforeignmemory.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14248, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e85619d000
mmap(0x71e85619e000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e85619e000
mmap(0x71e85619f000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e85619f000
mmap(0x71e8561a0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e8561a0000
close(3)                                = 0
open("/lib/libxendevicemodel.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxendevicemodel.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxendevicemodel.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=22440, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 28672, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e856196000
mmap(0x71e856198000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e856198000
mmap(0x71e85619a000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x71e85619a000
mmap(0x71e85619b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x4000) = 0x71e85619b000
close(3)                                = 0
open("/lib/libxentoolcore.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/local/lib/libxentoolcore.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/libxentoolcore.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=14168, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 20480, PROT_READ, MAP_PRIVATE, 3, 0) = 0x71e856191000
mmap(0x71e856192000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0x1000) = 0x71e856192000
mmap(0x71e856193000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e856193000
mmap(0x71e856194000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x71e856194000
close(3)                                = 0
mprotect(0x71e8561f9000, 4096, PROT_READ) = 0
mprotect(0x71e856291000, 4096, PROT_READ) = 0
mprotect(0x71e8561f3000, 4096, PROT_READ) = 0
mprotect(0x71e8561ec000, 4096, PROT_READ) = 0
mprotect(0x71e8561bf000, 4096, PROT_READ) = 0
mprotect(0x71e8561b5000, 4096, PROT_READ) = 0
mprotect(0x71e8561af000, 4096, PROT_READ) = 0
mprotect(0x71e8561aa000, 4096, PROT_READ) = 0
mprotect(0x71e8561a5000, 4096, PROT_READ) = 0
mprotect(0x71e8561a0000, 4096, PROT_READ) = 0
mprotect(0x71e85619b000, 4096, PROT_READ) = 0
mprotect(0x71e856194000, 4096, PROT_READ) = 0
mprotect(0x5cf2a8d74000, 4096, PROT_READ) = 0
pipe([3, 4])                            = 0
rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1 RT_2], [], 8) = 0
rt_sigprocmask(SIG_BLOCK, ~[], ~[KILL STOP RTMIN RT_1 RT_2], 8) = 0
fork()                                  = 1439
rt_sigprocmask(SIG_SETMASK, ~[KILL STOP RTMIN RT_1 RT_2], NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
close(4)                                = 0
read(3, "", 6)                          = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1439, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
writev(2, [{iov_base="", iov_len=0}, {iov_base="startup failed\n", iov_len=15}], 2startup failed
) = 15
exit_group(1)                           = ?
+++ exited with 1 +++

lubellier · 2023-08-26T16:29:58Z

Your strace shows a lot of not found libraries, it could be normal (or not). Could you basically check that all the libraries are available with ldd ?

[user@qubes-dev ~]$ grep ^PRETTY_NAME /etc/os-release 
PRETTY_NAME="Fedora Linux 37 (Thirty Seven)"
[user@qubes-dev ~]$ file /sbin/qubesdb-daemon 
/sbin/qubesdb-daemon: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=00b7b6e43d310b1dd332d69f016d71c53e25bb2e, for GNU/Linux 3.2.0, stripped
[user@qubes-dev ~]$ ldd /sbin/qubesdb-daemon 
	linux-vdso.so.1 (0x00007ffea8fc6000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007ad2255fb000)
	libvchan-xen.so => /lib64/libvchan-xen.so (0x00007ad2255f4000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ad225417000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007ad22540d000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007ad2253d9000)
	libzstd.so.1 => /lib64/libzstd.so.1 (0x00007ad225320000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007ad2252fb000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ad2252db000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ad2256f1000)
	libxenvchan.so.4.16 => /lib64/libxenvchan.so.4.16 (0x00007ad2252d4000)
	libxenctrl.so.4.16 => /lib64/libxenctrl.so.4.16 (0x00007ad2252a8000)
	libxentoollog.so.1 => /lib64/libxentoollog.so.1 (0x00007ad2252a3000)
	libxenstore.so.4 => /lib64/libxenstore.so.4 (0x00007ad225295000)
	libxengnttab.so.1 => /lib64/libxengnttab.so.1 (0x00007ad22528f000)
	libxenevtchn.so.1 => /lib64/libxenevtchn.so.1 (0x00007ad22528a000)
	libxencall.so.1 => /lib64/libxencall.so.1 (0x00007ad225285000)
	libxenforeignmemory.so.1 => /lib64/libxenforeignmemory.so.1 (0x00007ad22527f000)
	libxendevicemodel.so.1 => /lib64/libxendevicemodel.so.1 (0x00007ad225278000)
	libxentoolcore.so.1 => /lib64/libxentoolcore.so.1 (0x00007ad225271000)

ayakael · 2023-08-26T16:39:32Z

@lubellier Thanks! Yes, everything is there. The missing libraries is actually qubesdb going through every possible library location. It eventually finds the libraries, and then it continues.

lubellier · 2023-08-26T17:22:53Z

Did you compare your strace output with a fedora template workable qubesdb-daemon strace output ?

ayakael · 2023-08-26T19:40:17Z

I went down the comparison rabbit hole and ended up comparing the working manually built Alpine template with the broken one. Indeed, it had nothing to do with qubesdb, but rather some missing low-level services that needed to be added to a runlevel. There were also groups + user issues.

I'm glad to say that the template builder now builds a functional template!

Next up is getting the template builder to be able to generate multiple versions of Alpine (right now, it can only do the latest stable), after I'll get some Gitlab pipelines up to make prebuilt RPMs available via Releases.

ayakael · 2023-08-26T23:16:44Z

I finished up the basic components of the infra. RPM are now available here: https://lab.ilot.io/ayakael/qubes-builder-alpine/-/releases

Feel free to test and come back to me on any issues. I only tested the ability to open xterm within the template.

TwinkleToes777 · 2023-10-01T19:01:27Z

Issue: I was not able to connect to any services running on 127.0.0.1 while using AppVM based on this template, because lo interface was not automatically turned on (sudo ifconfig lo up)

LindaFerum · 2023-10-08T16:20:05Z

@ayakael wow, this is just awesome! Thank you for this great work

ayakael · 2023-12-08T01:08:33Z

Template RPMs for v3.19 are up for both r4.1 and r4.2

@TwinkleToes777 Thank you for noticing this. I added the necessary setup steps in the builder

@LindaFerum With pleasure ^^

I have time to work on this more. Any idea on how I might want to improve accessibility of this project? @marmarek @andrewdavidwong What steps can I take to improve visibility so that it's comparable to Arch Linux?

andrewdavidwong · 2023-12-08T11:53:34Z

I have time to work on this more. Any idea on how I might want to improve accessibility of this project? @marmarek @andrewdavidwong What steps can I take to improve visibility so that it's comparable to Arch Linux?

That's a good question. There are two separate aspects:

What is the proper procedure when a member of the community wants to contribute a community-supported template to the Qubes OS Project? I am not sure whether it falls under the package contribution procedure. If so, then simply follow that procedure. If not, then I don't think there's a documented procedure. (Maybe it's been infrequent enough just to be handled on a case-by-case basis.) All of this is up to @marmarek to decide.
What's the best way to spread the word about the new template and get others from the community involved? This one is straightforward. You can simply post about it in an appropriate Qubes venue. I'd suggest the forum and maybe qubes-users or qubes-devel (depending on the nature of your post and your target audience).

lubellier · 2023-12-10T21:00:53Z

Today I installed the Antoine's 3.19 alpine template from the rpm release in Qubes-OS 4.1. And It works great, congratulation to Antoine ( @ayakael ) !

I tested:

template installation
app qube creation and usage (terminal, web browser)
disposable template creation and disposable usage (terminal, web browser)
qvm-copy, qvm-move, qvm-open-in-dvm usages

Presently, for my usage, the main limitation (already in the the-yet-to-be-implemented-list)) is :

apk proxying from within template (thus you must allow internet access to template to install packages)

Lately I saw that I installed the wrong template ( 4.2 rpm template in QubesOS 4.1 ) but It works :-) I will fix my mistake later...

ayakael · 2023-12-10T21:09:50Z

@lubellier Many thanks for checking that! If you find a way to get apk proxying working, please feel free to share. I can then integrate it in the template builder. It's low on my list of things to tackle, so outside help would be welcomed. I'd like to tackle systems VMs next.

As for your mistake, it should be as easy as changing the URL in /etc/apk/repositories from r4.2 to r4.1 and executing apk upgrade -a. If ever you want to go to edge, I also release the packages for edge, so requires just a change from v3.19 to edge like the rest of the repos. ^^

@andrewdavidwong Awesome, thank you! I'll wait for @marmarek's perspective, and then get on that. My building infra (available here) has all of the APKBUILD files in the same repo. I assume getting it to match the archlinux infra would involve integrating those APKBUILD files in their relevant repos.

lubellier · 2024-02-04T08:42:14Z

If you find a way to get apk proxying working, please feel free to share.

@ayakael I found a solution, I don't own a gitlab account on lab.ilot.io and I never did an APKBUILD, so I share here.

/etc/init.d/qubes-updates-proxy-forwarder creates a socket with netcat to the sys-net proxy controlled by the QubesOS RPC. @ayakael : in the qubes-vm-core-openrc apk package, you shoud add this service (rc-update add ?). For debugging, just add the -v to the nc command and see the output_log and error_log files. You will see the created socket with netstat -apn --inet --listen.

#!/sbin/openrc-run
# Updates proxy forwarder     Startup script for the updates proxy forwarder
# description: forwards connection to updates proxy over Qubes RPC
# The clients should use the below shell variable exports:
#   http_proxy="http://127.0.0.1:8082/"
#   https_proxy="http://127.0.0.1:8082/"
# For apk, see the /etc/profile.d/apk-proxy.sh alias

name=$RC_SVCNAME
cfgfile="/etc/qubes/$RC_SVCNAME.conf"
command="/bin/busybox"
command_args="nc -lk -s 127.0.0.1 -p 8082 -e /usr/bin/qrexec-client-vm @default qubes.UpdatesProxy"
command_user="root"
pidfile="/run/qubes/$RC_SVCNAME.pid"
command_background="yes"
output_log="/var/log/qubes/$RC_SVCNAME.log"
error_log="/var/log/qubes/$RC_SVCNAME.err"

depend() {
    need qubes-qrexec-agent
    need net
}

start_pre() {
    checkpath --directory --owner $command_user:qubes --mode 0775 \
              /run/qubes \
	      /var/log/qubes \
	      /var/run/qubes
    # TODO should fail if qubes-update-proxy is running
    # if qsvc qubes-updates-proxy ; then
    #     # updates proxy running here too, avoid looping traffic back to itself
    #     exit 0
    # fi
}

/etc/profile.d/apk-proxy.sh creates an apk alias with the proxy usage

# Use the update proxy over the QubesOS RPC for apk
# /etc/init.d/qubes-updates-proxy-forwarder creates the socket to the proxy
# root uses the ash shell and the users use the bash or other shells, 
#  so use /etc/profile.d/ and not /etc/bashrc/ for the alias
alias apk='https_proxy="http://127.0.0.1:8082/" apk'

/etc/bash/sudo-aliases.sh allows the above alias for sudo

# allow aliases with sudo
alias sudo='sudo '

Thanks to the 2 aliases, the usage is very basic:

as a user : sudo apk upgrade
as root : apk upgrade

ayakael · 2024-02-08T18:05:52Z

@lubellier Many thanks for this! I've just pushed an update to qubes-vm-core with your proposed changes, with one adjustment. I've consolidated sudo-aliases.sh and apk-proxy.sh and have them mirrored via a symbolic link on both /etc/bash and /etc/profile.d. There are still issues with non-interactive shells on sourcing the alias, but this seems to cover most cases.

I've also pushed a new image r4.2 and r4.1 image with the changes, along with a fix for the repository issue related to r4.2 pointing to r4.1.

lubellier · 2024-02-08T21:46:24Z

@ayakael Thanks for the package and repository update!
Is it the package or the user to do the rc-update add qubes-updates-proxy-forwarder ? Because I need to do it manually after the apk upgrade (+reboot), to get the init service started by default.

ayakael · 2024-02-09T22:09:03Z

@lubellier The new template RPM will have qubes-updates-proxy-forwarder added to boot. Old templates will have to apk upgrade + rc-update add qubes-updates-proxy-forwarder to enable the APK proxy.

jobforacowboy · 2024-02-12T13:22:04Z

@ayakael i have request refused in /var/log/qubes/qubes-updates-proxy-forwarder.err on alpine based AppVM or StandaloneVM. it seems like /etc/qubes/policy.d/90-default.policy need modifications.

marmarek · 2024-02-12T13:28:00Z

do not modify the 90-default.policy file, use earlier file instead. Is that R4.2? if so, there should be 50-config-updates.policy that is managed by the GUI "qubes global config" tool - you can configure updates proxy there (but defaults should work...)

is this standalone qube instead of template by a chance? those should not use updates proxy by default (but it's still okay to enable it, if you configure policy for it too)

jobforacowboy · 2024-02-12T13:54:31Z

@marmarek, R4.2

Denied qubes.UpdatesProxy from test-alpine to @default

 qvm-prefs test-alpine
audiovm               D  dom0
autostart             D  False
backup_timestamp      U
debug                 D  False
default_dispvm        D  default-dvm
default_user          D  user
dns                   D  10.139.1.1 10.139.1.2
gateway               D  
gateway6              D  
guivm                 D  dom0
icon                  D  appvm-red
include_in_backups    D  True
installed_by_rpm      D  False
ip                    D  10.137.0.27
ip6                   D  
kernel                D  6.6.14-1.fc37
kernelopts            D  swiotlb=2048
keyboard_layout       D  us++
klass                 D  AppVM
label                 -  red
mac                   D  00:16:3e:5e:6c:00
management_dispvm     D  default-mgmt-dvm
maxmem                D  4000
memory                D  400
name                  -  test-alpine
netvm                 D  sys-firewall
provides_network      -  False
qid                   -  27
qrexec_timeout        D  60
shutdown_timeout      D  60
start_time            D  1707744843.24
stubdom_mem           U
stubdom_xid           D  -1
template              -  alpine
template_for_dispvms  D  False
updateable            D  False
uuid                  -  03d4dc3d-e9b2-488d-b976-d13103c2475d
vcpus                 D  2
virt_mode             D  pvh
visible_gateway       D  10.138.38.213
visible_gateway6      D  
visible_ip            D  10.137.0.27
visible_ip6           D  
visible_netmask       D  255.255.255.255
xid                   D  17

default settings in my policy

/etc/qubes/policy.d/50-config-updates.policy 

# THIS IS AN AUTOMATICALLY GENERATED POLICY FILE.
# Any changes made manually may be overwritten by Qubes Configuration Tools.

qubes.UpdatesProxy	*	@tag:whonix-updatevm	@default	allow target=sys-whonix

marmarek · 2024-02-12T14:03:55Z

ah, so it isn't even standalone, just normal app qube; proxy shouldn't be enabled in that case, see https://github.com/QubesOS/qubes-core-agent-linux/blob/main/network/update-proxy-configs#L79 (updates-proxy-setup is enabled in templates only: https://github.com/QubesOS/qubes-core-agent-linux/blob/main/vm-systemd/qubes-sysinit.sh#L9-L13)

ayakael · 2024-02-12T16:30:54Z

ah, so it isn't even standalone, just normal app qube; proxy shouldn't be enabled in that case, see https://github.com/QubesOS/qubes-core-agent-linux/blob/main/network/update-proxy-configs#L79 (updates-proxy-setup is enabled in templates only: https://github.com/QubesOS/qubes-core-agent-linux/blob/main/vm-systemd/qubes-sysinit.sh#L9-L13)

I see, we're missing if qsvc yum-proxy-setup || qsvc updates-proxy-setup ; then logics for apk. I'll investigate this later this week. Anyways, the current alias approach for setting https_proxy env for apk is not excellent. We need a better approach.

lubellier · 2024-02-13T00:13:21Z

Anyways, the current alias approach for setting https_proxy env for apk is not excellent. We need a better approach.

I agree.

Apk doesn't provide a setting file for the proxy settings. So we should find a way to inject the https_proxy env for the apk command. Apk uses libfetch, libfetch get the proxy settings in http.c:780.

Ideas : a shell function, a shell script, sudo env https_proxy="http://127.0.0.1:8082/" apk , ...

ayakael · 2024-04-20T21:21:01Z

Of note, I've rebuilt all r4.1 and r4.2 packages to support current edge that now uses python 3.12. With r4.1 EOL in mid-june, this is likely the last update for r4.1 packages. I will sunset the r4.1 building infra around that time, as well.

EDIT Alpine 3.20, due for release in the beginning of may, will thus be the last release that will support r4.1, and that only till mid-june.

ayakael added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: enhancement Type: enhancement. A new feature that does not yet exist or improvement of existing functionality. labels Mar 5, 2022

andrewdavidwong added the C: Alpine Linux label Mar 6, 2022

andrewdavidwong added this to the Release TBD milestone Mar 6, 2022

This comment was marked as off-topic.

Sign in to view

Alpine Linux Template #7323

Alpine Linux Template #7323

Comments

ayakael commented Mar 5, 2022 • edited

description

current state

steps

hurdles

ayakael commented Mar 5, 2022 • edited

moiselazarus commented Mar 5, 2022

ayakael commented Mar 8, 2022

ayakael commented Jun 20, 2022 • edited

ayakael commented Jun 20, 2022

ayakael commented Jun 21, 2022

marmarek commented Jun 21, 2022

ayakael commented Jun 21, 2022

ayakael commented Jun 22, 2022

ayakael commented Jun 28, 2022

ayakael commented Jun 28, 2022 • edited

moiselazarus commented Jun 30, 2022

ayakael commented Jul 6, 2022

ayakael commented Jul 11, 2022 • edited

ayakael commented Jul 14, 2022 • edited

marmarek commented Jul 14, 2022

marmarek commented Jul 14, 2022

ayakael commented Jul 14, 2022

marmarek commented Jul 14, 2022

ayakael commented Jul 14, 2022

marmarek commented Jul 14, 2022

ayakael commented Jul 14, 2022

ayakael commented Jul 16, 2022

marmarek commented Jul 16, 2022

ayakael commented Jul 17, 2022

ayakael commented Jul 17, 2022

ayakael commented Aug 26, 2023

ayakael commented Aug 26, 2023

lubellier commented Aug 26, 2023

ayakael commented Aug 26, 2023

lubellier commented Aug 26, 2023

ayakael commented Aug 26, 2023 • edited

ayakael commented Aug 26, 2023

TwinkleToes777 commented Oct 1, 2023

LindaFerum commented Oct 8, 2023

ayakael commented Dec 8, 2023

andrewdavidwong commented Dec 8, 2023 • edited

lubellier commented Dec 10, 2023

ayakael commented Dec 10, 2023

lubellier commented Feb 4, 2024

ayakael commented Feb 8, 2024

lubellier commented Feb 8, 2024

ayakael commented Feb 9, 2024 • edited

jobforacowboy commented Feb 12, 2024

marmarek commented Feb 12, 2024

jobforacowboy commented Feb 12, 2024

marmarek commented Feb 12, 2024

ayakael commented Feb 12, 2024

lubellier commented Feb 13, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

ayakael commented Apr 20, 2024 • edited

ayakael commented Mar 5, 2022 •

edited

ayakael commented Mar 5, 2022 •

edited

ayakael commented Jun 20, 2022 •

edited

ayakael commented Jun 28, 2022 •

edited

ayakael commented Jul 11, 2022 •

edited

ayakael commented Jul 14, 2022 •

edited

ayakael commented Aug 26, 2023 •

edited

andrewdavidwong commented Dec 8, 2023 •

edited

ayakael commented Feb 9, 2024 •

edited

ayakael commented Apr 20, 2024 •

edited