mkdwarfs always crashes with SIGABRT

[paisleyrob]

I grabbed this release: dwarfs-0.9.7-Linux-x86_64-clang and from the same directory I issued:

$ ./bin/mkdwarfs --input bin --output bin.dwarfs
I 16:18:51.803941 scanning "/home/redacted/dl/dwarfs-0.9.7-Linux-x86_64-clang/bin"
I 16:18:51.804988 assigning directory and link inodes...
I 16:18:51.805028 waiting for background scanners...
I 16:18:51.899182 scanning CPU time: 348.3ms
I 16:18:51.899210 finalizing file inodes...
I 16:18:51.899230 saved 0 B / 50.91 MiB in 0/4 duplicate files
I 16:18:51.899240 assigning device inodes...
I 16:18:51.899252 assigning pipe/socket inodes...
I 16:18:51.899264 building metadata...
I 16:18:51.899280 building blocks...
I 16:18:51.899693 saving names and symlinks...
I 16:18:51.899778 updating name and link indices...
I 16:18:51.900443 waiting for segmenting/blockifying to finish...
I 16:18:52.113039 total ordering CPU time: 665.7us
I 16:18:52.113091 total segmenting CPU time: 212ms
I 16:18:52.114382 saving chunks...
I 16:18:52.114439 saving directories...
I 16:18:52.114464 saving shared files table...
I 16:18:52.115385 saving names table... [901us]
I 16:18:52.115416 saving symlinks table... [1.785us]
I 16:18:52.115692 waiting for compression to finish...
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
1 dirs, 0/0 soft/hard links, 4/4 files, 0 other
original size: 50.91 MiB, hashed: 0 B (0 files, 0 B/s)
scanned: 50.91 MiB (4 files, 155.4 MiB/s), categorizing: 0 B/s
saved by deduplication: 0 B (0 files), saved by segmenting: 1.912 MiB
filesystem: 49 MiB in 4 blocks (274 chunks, 4/4 fragments, 4 inodes)
compressed filesystem: 0 blocks/0 B written
██████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                                                    ▏ 50% 🌖
[compressing] compressed 1023 KiB to 228.9 KiB (ratio 22.37%)                                                                                                                                           0 B/s
Assertion `::EVP_DigestInit(context_.get(), evp)` failed in /home/ubuntu/dwarfs/src/dwarfs/checksum.cpp(70): EVP_DigestInit() failed
terminate called without an active exception
*** Aborted at 1712938737 (Unix time, try 'date -d @1712938737') ***
*** Signal 6 (SIGABRT) (0x1e0a8e80006019f) received by PID 393631 (pthread TID 0x7f95065f76c0) (linux TID 393641) (maybe from PID 393631, UID 31500520) (code: -6), stack trace: ***
    @ 0000000000c11d29 (unknown)
    @ 0000000000dd1d0f (unknown)
    @ 0000000000dff31c (unknown)
    @ 0000000000dd1c8d (unknown)
    @ 0000000000420d87 (unknown)
    @ 00000000004191f1 (unknown)
    @ 0000000000cde35b (unknown)
    @ 000000000041914d (unknown)
    @ 00000000004937a5 (unknown)
    @ 0000000000493840 (unknown)
    @ 0000000000484093 (unknown)
    @ 0000000000483b50 (unknown)
    @ 00000000004af2a1 (unknown)
    @ 00000000004aedcc (unknown)
    @ 00000000004ab72b (unknown)
    @ 00000000004a94c9 (unknown)
    @ 0000000000d6b123 (unknown)
    @ 0000000000dfda4b (unknown)
    @ 0000000000e87763 (unknown)
Aborted (core dumped)

Kernel 4.18.0-513.18.1.el8_9.x86_64 on RHEL 8.9 if that helps.

My only guess is machine has FIPS enabled on it.

[mhx]

Interesting! Can you do me a favour and run:

$ mkdwarfs -H

The output should contain something like this:

  --file-hash arg (=xxh3-128)           choice of file hashing function (none, 
                                        blake2b512, blake2s256, md5, md5-sha1, 
                                        ripemd160, sha1, sha224, sha256, 
                                        sha3-224, sha3-256, sha3-384, sha3-512,
                                        sha384, sha512, sha512-224, sha512-256,
                                        shake128, shake256, sm3, xxh3-128, 
                                        xxh3-64)

DwarFS uses EVP_sha512_256() to compute digests for each file system block and I assume this is what triggers the assertion.

Unfortunately, I have no experience with FIPS mode; from the OpenSSL documentation I would assume that SHA512/256 would even work in FIPS mode as it states "CONFORMING TO NIST FIPS 180-4".

[paisleyrob]

This is equally bizarre output. I grabbed just the --file-hash section:

  --file-hash arg (=xxh3-128)           choice of file hashing function (none,
                                        xxh3-128, xxh3-64)

[paisleyrob]

The top of -H looks like the following:

mkdwarfs (v0.9.7 [2024-04-10])
built for x86_64, Linux-5.15.0-101-generic, Clang 18.1.0

using: FLAC++-1.4.3, boost-1.83.0, brotlidec-1.1.0, brotlienc-1.1.0,
       crypto-3.0.13, fmt-10.2.1, jemalloc-5.3.0, lz4-1.9.4, lzma-5.4.5,
       xxhash-0.8.2, zstd-1.5.5

Which seems to be missing openssl 🤔

[mhx]

This is equally bizarre output. I grabbed just the --file-hash section:

  --file-hash arg (=xxh3-128)           choice of file hashing function (none,
                                        xxh3-128, xxh3-64)

Yeah, I kinda expected something like this. Didn't quite expect that none of the OpenSSL algorithms would be present, though.

The top of -H looks like the following:

mkdwarfs (v0.9.7 [2024-04-10])
built for x86_64, Linux-5.15.0-101-generic, Clang 18.1.0

using: FLAC++-1.4.3, boost-1.83.0, brotlidec-1.1.0, brotlienc-1.1.0,
       crypto-3.0.13, fmt-10.2.1, jemalloc-5.3.0, lz4-1.9.4, lzma-5.4.5,
       xxhash-0.8.2, zstd-1.5.5

Which seems to be missing openssl 🤔

That's actually expected; the OpenSSL is not (yet) part of this list. It's nonetheless linked into the binary.

I wonder if OpenSSL has some dependency on kernel functions and the (rather old) kernel and OpenSSL aren't getting along to well.

[mhx]

I wonder if OpenSSL has some dependency on kernel functions and the (rather old) kernel and OpenSSL aren't getting along to well.

I don't think it's that. I just installed Fedora 22 and tried:

[root@localhost ~]# uname -a
Linux localhost.localdomain 4.4.14-200.fc22.x86_64 #1 SMP Fri Jun 24 21:19:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -H | grep -A 10 file-hash
warning: failed to set user default locale: locale::facet::_S_create_c_locale name not valid
  --file-hash arg (=xxh3-128)           choice of file hashing function (none, 
                                        blake2b512, blake2s256, md5, md5-sha1, 
                                        ripemd160, sha1, sha224, sha256, 
                                        sha3-224, sha3-256, sha3-384, sha3-512,
                                        sha384, sha512, sha512-224, sha512-256,
                                        shake128, shake256, sm3, xxh3-128, 
                                        xxh3-64)

That kernel is even older. The binary seems to work just fine:

[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -i /usr/ -o /dev/null --force -l4
warning: failed to set user default locale: locale::facet::_S_create_c_locale name not valid
I 19:01:53.486009 scanning "/usr"
I 19:01:54.263798 assigning directory and link inodes...
I 19:01:54.267666 waiting for background scanners...
I 19:01:55.724926 scanning CPU time: 4.714s
I 19:01:55.724941 finalizing file inodes...
I 19:01:55.734015 saved 15.55 MiB / 990.2 MiB in 2504/42710 duplicate files
I 19:01:55.734257 assigning device inodes...
I 19:01:55.734591 assigning pipe/socket inodes...
I 19:01:55.734846 building metadata...
I 19:01:55.734862 building blocks...
I 19:01:55.734915 saving names and symlinks...
I 19:01:55.735250 waiting for segmenting/blockifying to finish...
I 19:01:55.748615 updating name and link indices...
I 19:02:00.024608 total ordering CPU time: 256.7us
I 19:02:00.024765 total segmenting CPU time: 4.212s
I 19:02:00.027036 saving chunks...
I 19:02:00.034627 saving directories...
I 19:02:00.039707 saving shared files table...
I 19:02:00.054576 saving names table... [9.325ms]
I 19:02:00.056605 saving symlinks table... [1.809ms]
I 19:02:00.080138 waiting for compression to finish...
I 19:02:00.082730 compressed 990.2 MiB to 316.3 MiB (ratio=0.319461)
I 19:02:00.087192 compression CPU time: 20.67s
I 19:02:00.087370 filesystem created without errors [6.602s]
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
waiting for block compression to finish
4653 dirs, 1244/3649 soft/hard links, 42710/42710 files, 0 other
original size: 990.2 MiB, hashed: 144.4 MiB (29093 files, 60.95 MiB/s)
scanned: 974.6 MiB (36557 files, 158.5 MiB/s), categorizing: 0 B/s
saved by deduplication: 15.55 MiB (2504 files), saved by segmenting: 91.95 MiB
filesystem: 882.7 MiB in 221 blocks (46290 chunks, 36556/36556 fragments, 36557 inodes)
compressed filesystem: 221 blocks/316.3 MiB written
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏100% 🌒
[root@localhost ~]# 

[paisleyrob]

I was able to build it inside a ubuntu:23:10 container it works fine, so it's probably something about FIPS not being linked in and so you get no hashes.

[mhx]

My only guess is machine has FIPS enabled on it.

That's definitely the root cause. I also installed Fedora 29 in a VM. mkdwarfs worked fine right after the system booted. Then I followed the instructions to enable FIPS mode and got:

[root@localhost ~]# uname -a
Linux localhost.localdomain 5.3.11-100.fc29.x86_64 #1 SMP Tue Nov 12 20:41:25 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# sysctl crypto.fips_enabled
crypto.fips_enabled = 1

[root@localhost ~]# ./dwarfs-universal-0.9.7-Linux-x86_64-clang --tool=mkdwarfs -H | grep -A 3 file-hash
  --file-hash arg (=xxh3-128)           choice of file hashing function (none, 
                                        xxh3-128, xxh3-64)
  --progress arg (=unicode)             progress mode (ascii, none, simple, 
                                        unicode)

[paisleyrob]

Well done @mhx. That certainly proves it.

[mhx]

The funny thing is: I can build a statically linked test program on my Gentoo machine, copy it to the FIPS-enabled Fedora 29 VM, and get:

[root@localhost ~]# ./digest 
-> md5-sha1
-> sha512-224
-> sm3
-> sha512
-> sha384
-> sha224
-> md4
-> blake2b512
-> ripemd160
-> sha256
-> sha512-256
-> shake128
-> whirlpool
-> blake2s256
-> sha3-256
-> shake256
-> mdc2
-> sha3-224
-> sha3-512
-> sha3-384
-> md5
-> sha1
Digest is: 0686f0a605973dc1bf035d1e2b9bad1985a0bff712ddd88abd8d2593e5f99030

And I think that works because my Gentoo OpenSSL library is built without FIPS support:

[ebuild   R    ] dev-libs/openssl-3.0.13:0/3::gentoo  USE="asm static-libs -fips -ktls -rfc3779 -sctp -test -tls-compression -vanilla -verify-sig -weak-ssl-ciphers" ABI_X86="32 (64) (-x32)" CPU_FLAGS_X86="(sse2)" 0 KiB

[mhx]

Reopening as I think this is actually fixable.

[mhx]

If you want, you can try dwarfs-universal-0.9.7-2-gdadc2e0254-Linux-x86_64-clang or dwarfs-0.9.7-2-gdadc2e0254-Linux-x86_64-clang.tar.zst. You'll need to be signed in to GitHub to be able to download the build artifacts.

The binaries work fine on my FIPS-enabled VM. The next release will contain the fix.

[paisleyrob]

I used the -clang.tar.zst version and at least the mkdwarfs worked perfectly. I first checked to see the hash sections and it showed the following list:

$ ./bin/mkdwarfs -H | grep -A6 file-hash
  --file-hash arg (=xxh3-128)           choice of file hashing function (none,
                                        blake2b512, blake2s256, md5, md5-sha1,
                                        ripemd160, sha1, sha224, sha256,
                                        sha3-224, sha3-256, sha3-384, sha3-512,
                                        sha384, sha512, sha512-224, sha512-256,
                                        shake128, shake256, sm3, xxh3-128,
                                        xxh3-64)

Well done! 🏁 🚀

[mhx]

Fixed in v0.9.8. Thanks for your help!