FreeBSD 13 jail: Failed to create CoreCLR: HRESULT: 0x8007054f #99857
Comments
|
Tagging subscribers to this area: @mangod9 |
dotnet-policy-service
bot
added
the
untriaged
root@f13:~ # truss -f dotnet --help 2>&1 | grep ERR
75727: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
75727: open("/lib/casper/libthr.so.3",O_RDONLY|O_CLOEXEC|O_VERIFY,032266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libcxxrt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,013026041400) ERR#2 'No such file or directory'
75727: open("/lib/casper/libm.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such file or directory'
75727: open("/lib/casper/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,010320643400) ERR#2 'No such file or directory'
75727: open("/lib/casper/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,020443000) ERR#2 'No such file or directory'
75727: readlink("/etc/malloc.conf",0x7fffffffcc10,1024) ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/libcoreclr.so",F_OK) ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/dotnet.dll",F_OK) ERR#2 'No such file or directory'
75727: access("/root/global.json",F_OK) ERR#2 'No such file or directory'
75727: access("/global.json",F_OK) ERR#2 'No such file or directory'
75727: __realpathat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
75727: fstatat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x7fffffffd340,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
75727: __realpathat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
75727: fstatat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x7fffffffce50,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
75727: access("",F_OK) ERR#2 'No such file or directory'
75727: access("opt/coreservicing",F_OK) ERR#2 'No such file or directory'
75727: access("",F_OK) ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/store/x64/net8.0",F_OK) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/lib/casper/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/libz.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libz.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libmd.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such file or directory'
75727: statfs("/sys/fs/cgroup",0x7fffffffc958) ERR#2 'No such file or directory'
75727: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
75727: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
75727: mlock(0x883200000,4096) ERR#1 'Operation not permitted'Apparently, it requires I haven't tried .NET 6, but it would be interesting to know when this |
@am11 FYI, also I think that executable pages might be implicitly locked? |
|
Maybe, but with allow_mlock, the webapi started for me successfully in freebsd 13.3 jail: |
|
Mlock is already enabled, it was already a requirement for DotNet 6. I have one user who can run it successfully though, might be limited to my particular setup, trying to know more. I run TrueNAS Core as a VM inside TrueNAS scale, never had any issue with DotNet 6 under the same condition though. DotNet 8 also works fine on FreeBSD 13 and 14 also running as VMs on Scale. Could also be that my jail is broken, I will try to start fresh later today. |
|
My jail setup was failing and recreating with allow_mlock fixed it. Can you share the output of |
|
New information, I'm told it works in a 13.3 jail, but not in a 13.2 jail, didn't realize I didn't have the latest version since I did update it before testing. Will need to create a fresh one to get the latest version. I'm sure I'll find that it works there, will keep you posted. Thanks for the help. |
|
@am11 the @alucryd the pkg builds ( |
|
@Thefrank Yeah learned that the hard way, wasn't able to have a single package for both 13 and 14. I can confirm that dotnet 8 does work fine in a 13.3 jail though, so this particular issue can be closed. Thank you all for helping with my issue. |
dotnet-policy-service
bot
removed
the
untriaged
I created 13.2 jail with cbsd and dotnet 8 works there as well. If you are able to repro it again, the truss output will help understanding the cause. You can also run it under lldb inside the jail (with |
|
@am11 There you go, this outcome has been confirmed by 2 other TrueNAS Core users: |
ERR 22 is EINVAL, and per https://man.freebsd.org/cgi/man.cgi?query=cpuset_getaffinity&sektion=2&n=1, it happens when arg1 ( Ballpark: I guess the |
|
The jail is running a newer version of FreeBSD than the host? While that is something that you can do, it is not advised. This version of dotnet8 (sdk 8.0.100) was built under FreeBSD 12.4 and works under TrueNAS CORE 13.1 u6.1 jails dotnet8 (sdk 8.0.101) that was built under FreeBSD 13.2 does not work in either a FreeBSD 13.1 or FreeBSD 13.2 jail under TrueNAS CORE. The result is the same when using Atleast this lets me know I need to build THREE versions of my packages now :( |
Was that a bug in 13.2 jail or does building for 13.2 specifically fix the problem under TrueNAS Core? It sounds to me the former case that 13.2 jail (created with cpuset=# etc.) was buggy. |
|
TrueNAS CORE uses The config when using {
"allow_mlock": 1,
"bpf": 1,
"cloned_release": "13.2-RELEASE",
"defaultrouter": "192.168.0.1/24",
"dhcp": 1,
"host_hostname": "newjail",
"host_hostuuid": "newjail",
"jail_zfs_dataset": "iocage/jails/newjail/data",
"release": "13.2-RELEASE-p8",
"vnet": 1
}AFAIK Here is what I have been able to determine via testing on AMD64 systems with dotnet8
A cursory check of net7 and net6 yield similar results. Only versions built for 12.4 will work under TrueNAS CORE. I guess I need to build my packages for 13.1(?)/whatever TrueNAS is, 13.2, and 14.0. I might be able to get away with dropping 14.0 as 13.2 binaries seem to work fine post-OpenSSL patch |
|
The error
I think we should try to keep the runtime code robust against the point-in-time kernel/base-system bugs, as we do on linux; rather than expecting that many variant builds by the package maintainers. The next step would be to repro it with plain C and find out the issue with $ cat > repro.c <<EOF
#include <sched.h>
#include <unistd.h>
int main() {
cpu_set_t cpuSet;
return sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
}
EOF
$ cc repro.c
$ truss -f ./a.outUnfortunately, I couldn't reproduce the error in my jail environment so far. |
|
first test: result: second test: result: edit: full output |
|
I see. In my case it works fine with 13.2-RELEASE jain (sitting on 14.0 host): $ uname -a
FreeBSD f132.my.domain 13.2-RELEASE FreeBSD 14.0-RELEASE #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
$ freebsd-version
13.2-RELEASE
$ truss -f ./a.out
26962: mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 53112616255488 (0x304e3ea00000)
26962: mprotect(0x19e25b164000,4096,PROT_READ) = 0 (0x0)
26962: issetugid() = 0 (0x0)
26962: sigfastblock(0x1,0x19e25b166990) = 0 (0x0)
26962: open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,07650030030) = 3 (0x3)
26962: fstat(3,{ mode=-rw-r--r-- ,inode=50,size=47,blksize=4096 }) = 0 (0x0)
26962: read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
26962: close(3) = 0 (0x0)
26962: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
26962: open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010002363) = 3 (0x3)
26962: read(3,"Ehnt\^A\0\0\0\M^@\0\0\0G\0\0\0\0"...,128) = 128 (0x80)
26962: fstat(3,{ mode=-r--r--r-- ,inode=385077,size=199,blksize=4096 }) = 0 (0x0)
26962: pread(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,71,0x80) = 71 (0x47)
26962: close(3) = 0 (0x0)
26962: open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,012320443000) = 3 (0x3)
26962: fstat(3,{ mode=-r--r--r-- ,inode=26910,size=1958208,blksize=131072 }) = 0 (0x0)
26962: mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34941075456 (0x822a68000)
26962: mmap(0x0,4206592,PROT_NONE,MAP_GUARD,-1,0x0) = 34923323392 (0x82197a000)
26962: mmap(0x82197a000,540672,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) = 34923323392 (0x82197a000)
26962: mmap(0x8219fe000,1359872,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x83000) = 34923864064 (0x8219fe000)
26962: mmap(0x821b4a000,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ce000) = 34925223936 (0x821b4a000)
26962: mmap(0x821b54000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d7000) = 34925264896 (0x821b54000)
26962: mmap(0x821b5b000,2236416,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0) = 34925293568 (0x821b5b000)
26962: munmap(0x822a68000,4096) = 0 (0x0)
26962: close(3) = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ) = 0 (0x0)
26962: sysarch(AMD64_SET_FSBASE,0x8210fdfb0) = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ) = 0 (0x0)
26962: readlink("/etc/malloc.conf",0x8210fd6a0,1024) ERR#2 'No such file or directory'
26962: issetugid() = 0 (0x0)
26962: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 53112618352640 (0x304e3ec00000)
26962: clock_gettime(4,{ 146856.753862790 }) = 0 (0x0)
26962: clock_gettime(4,{ 146856.754531870 }) = 0 (0x0)
26962: clock_gettime(4,{ 146856.755200949 }) = 0 (0x0)
26962: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 53112620449792 (0x304e3ee00000)
26962: mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 53112622546944 (0x304e3f000000)
26962: mprotect(0x202000,4096,PROT_READ) = 0 (0x0)
26962: getpid() = 26962 (0x6952)
26962: cpuset_getaffinity(0x3,0x9,0x6952,0x20,0x8210fead8) = 0 (0x0)
26962: exit(0x0)
26962: process exit, rval = 0Could you give this a try: --- repro.c
+++ repro.c
int main() {
cpu_set_t cpuSet;
+ pid_t pid = getpid();
- return sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
+ return cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TIDPID, pid == 0 ? -1 : pid, sizeof(cpu_set_t), &cpuSet);
}if it works, we can add it as a fallback in PAL. |
|
Running under TrueNAS CORE host. first setup as earlier (13.1 jail on 13.1 host) second setup as earlier (13.2 jail on 13.1 host) From what I remember This appears to be a case where the jail is expecting a behavior different than what the host provides. The jail uses the host's kernel. FreeBSD's symbol versioning can work around this most of the time but that only works for older jail on newer host not the other way around. The symbols either don't exist on the older host or they do but are not marked as needing versioning. This is made a bit more complex as TrueNAS CORE is not patch-for-patch the same version as FreeBSD. I am unsure which upstream branch TrueNAS CORE tracks. uname from TrueNAS Core then FreeBSD 13.3-RELENG This is starting to look more like behavior from running newer os on older kernel but I am also willing to keep looking into this. |
|
|
Resolves the failure on the older one |
|
@Thefrank, that looks promising. If you have time, could you try applying this patch main...am11:feature/freebsd-port/getaffinity and test the runtime build? Otherwise, I'll try it over the weekend. Assuming this was the only issue, we can put up the PR. |
|
I am hitting this on native builds both with and without the patch when building from HEAD (for me it was: ffea258)
Will try again both from crossbuild from linux and from net9 preview2 tag and edit this post edit: crossbuild finishes but I don't recall if uses crossgen. Is cross-os supported? As a side note, the generated edit2: net9p2 runs into the same(?) issue at crossgen2 edit3: crossbuild taken from net9p2 + main...am11:feature/freebsd-port/getaffinity under 3 different conditions |
Description
It is impossible to run dotnet 8 inside a FreeBSD 13 jail, where it was previously possible to run dotnet 6.
Reproduction Steps
allow_mlock, which was required for dotnet 6)pkg install dotnetdotnet --helpExpected behavior
Dotnet 8 should run, as did dotnet 6.
Actual behavior
Dotnet fails to run with the following error:
Failed to create CoreCLR: HRESULT: 0x8007054f.Regression?
Was working fine with dotnet 6.
Known Workarounds
No response
Configuration
Dotnet 8.0
FreeBSD 13 jail on TrueNAS Core 13.0-U6.1
Other information
No response
The text was updated successfully, but these errors were encountered: