journalctl --file=/var/log/journal/*/* --follow -> Segmentation fault

[arwie]

systemctl --version
systemd 245 (245.4)
-PAM -AUDIT -SELINUX -IMA -APPARMOR -SMACK -SYSVINIT -UTMP -LIBCRYPTSETUP -GCRYPT -GNUTLS -ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=unified

Used distribution
PTXdist embedded system

Expected behaviour you didn't see
The following command worked in previous systemd versions as expecteed:
journalctl --file=/var/log/journal/*/* --lines=300 --output=json --all --follow --merge

Unexpected behaviour you saw

journalctl --file=/var/log/journal/*/* --follow
Segmentation fault

Directory structure

ll /var/log/journal/*/*
-rw-r-----    1 root     systemd-  16777216 Apr 22 10:04 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system.journal
-rw-r-----    1 root     systemd-  25165824 Feb  2 10:58 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@00059d95b012febf-f829ebef88294ad4.journal~
-rw-r-----    1 root     systemd-   8388608 Apr  9 10:51 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@0005a2d96639964f-8cb8e942e6fb3f10.journal~
-rw-r-----    1 root     systemd-   8388608 Apr  9 10:51 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@01a6125ef2244ac3ad4bad49ff4bb7c0-0000000000000001-0005a2d96638b3f3.journal
-rw-r-----    1 root     systemd-  25165824 Mar  2 09:15 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@4d2ddd3770ac493ab01ea3813d41ffb8-0000000000000001-00059d95b0121846.journal
-rw-r-----    1 root     systemd-  25165824 Apr  7 13:49 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@4d2ddd3770ac493ab01ea3813d41ffb8-0000000000005d87-00059fdba2a5e18c.journal
-rw-r-----    1 root     systemd-   8388608 Apr  8 13:28 /var/log/journal/92ca8e76cde84a0fb2e1d839a6bbd3c5/system@4d2ddd3770ac493ab01ea3813d41ffb8-0000000000009e3b-0005a2b3a5b023ee.journal
-rw-r-----    1 systemd- systemd-   8388608 Apr 22 09:41 /var/log/journal/remote/remote-mc.journal
-rw-r-----    1 systemd- systemd-   8388608 Feb  2 11:18 /var/log/journal/remote/remote-mc@00059d95f90b72b9-e297620e5342c84e.journal~
-rw-r-----    1 systemd- systemd-  16777216 Apr  9 10:51 /var/log/journal/remote/remote-mc@0005a2d9673b41b7-ca8c02e317ce02b9.journal~

DMESG

[ 1673.697349] journalctl[299]: segfault at 0 ip 00007f1cc8f74c73 sp 00007ffc63d374d8 error 4 in libc-2.28.so[7f1cc8eff000+143000]
[ 1673.697371] Code: 1f 84 00 00 00 00 00 66 0f 6e ce 89 f8 25 ff 0f 00 00 66 0f 60 c9 3d c0 0f 00 00 66 0f 61 c9 66 0f 70 c9 00 0f 8f 5d 01 00 00 <f3> 0f 6f 07 66 0f ef db 66 0f 6f e0 66 0f 74 c1 66 0f 74 e3 66 0f

[mrc0mmand]

Managed to reproduce this on Fedora Rawhide as well (systemd-245.5-1.fc33.x86_64).

# journalctl --file=/var/log/journal/*/* --follow
Segmentation fault (core dumped)

# coredumpctl info 656
           PID: 656 (journalctl)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 11 (SEGV)
     Timestamp: Wed 2020-04-22 07:16:43 EDT (2min 30s ago)
  Command Line: journalctl --file=/var/log/journal/*/* --follow
    Executable: /usr/bin/journalctl
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 4462e082cac54b5ca9390ff456c3122f
    Machine ID: 86378d18a63e428d96eda1d723e62b56
      Hostname: <hostname>
       Storage: /var/lib/systemd/coredump/core.journalctl.0.4462e082cac54b5ca9390ff456c3122f.656.1587554203000000000000.lz4
       Message: Process 656 (journalctl) of user 0 dumped core.
                
                Stack trace of thread 656:
                #0  0x00007f1ddc2345ec __strchr_avx2 (libc.so.6 + 0x1625ec)
                #1  0x00007f1ddc01154c dirname_has_namespace (libsystemd-shared-245.so + 0x1db54c)
                #2  0x00007f1ddc011acc add_directory (libsystemd-shared-245.so + 0x1dbacc)
                #3  0x00007f1ddc013d03 add_current_paths (libsystemd-shared-245.so + 0x1ddd03)
                #4  0x00007f1ddc013e3e sd_journal_get_fd (libsystemd-shared-245.so + 0x1dde3e)
                #5  0x0000565398ee5cd2 main (journalctl + 0x9cd2)
                #6  0x00007f1ddc0f9042 __libc_start_main (libc.so.6 + 0x27042)
                #7  0x0000565398ee618e _start (journalctl + 0xa18e)

Full trace:

#0  0x00007f1ddc2345ec in __strchr_avx2 () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f1ddc01154c in dirname_has_namespace (fn=fn@entry=0x0, namespace=0x0) at ../src/journal/sd-journal.c:1472
        e = <optimized out>
#2  0x00007f1ddc011acc in add_directory (j=0x56539abefd30, prefix=0x56539abf2200 "/var/log/journal/86378d18a63e428d96eda1d723e62b56", dirname=0x0) at ../src/journal/sd-journal.c:1629
        path = 0x56539abefc60 "/var/log/journal/86378d18a63e428d96eda1d723e62b56"
        d = 0x0
        m = <optimized out>
        r = <optimized out>
        k = <optimized out>
        __PRETTY_FUNCTION__ = "add_directory"
        __func__ = "add_directory"
#3  0x00007f1ddc013d03 in add_current_paths (j=<optimized out>) at ../src/journal/sd-journal.c:1854
        dir = 0x56539abf2200 "/var/log/journal/86378d18a63e428d96eda1d723e62b56"
        r = <optimized out>
        i = {idx = 4294967295, next_key = 0x0}
        f = 0x56539abf1f60
        i = <optimized out>
        f = <optimized out>
        __PRETTY_FUNCTION__ = <optimized out>
        dir = <optimized out>
        r = <optimized out>
#4  reiterate_all_paths (j=0x56539abefd30) at ../src/journal/sd-journal.c:2448
        __PRETTY_FUNCTION__ = "reiterate_all_paths"
#5  0x00007f1ddc013e3e in sd_journal_get_fd (j=0x56539abefd30) at ../src/journal/sd-journal.c:2481
        r = <optimized out>
        __PRETTY_FUNCTION__ = "sd_journal_get_fd"
        __func__ = "sd_journal_get_fd"
#6  0x0000565398ee5cd2 in main (argc=<optimized out>, argv=<optimized out>) at ../src/journal/journalctl.c:2447
        previous_boot_id_valid = false
        first_line = true
        ellipsized = false
        need_seek = false
        use_cursor = false
        after_cursor = false
        j = 0x56539abefd30
        previous_boot_id = <optimized out>
        n_shown = 0
        r = 0
        poll_fd = -1
        __func__ = "main"
        __PRETTY_FUNCTION__ = "main"

[mrc0mmand]

Most likely caused by 456aa87 from #14178.

/cc @poettering

[poettering]

My proposed fix is in #15544

[SomePersonSomeWhereInTheWorld]

I posted the following stacktraces and gdb from using Fail2ban with the journal on Fedora 32 and Python 3.8.2..

kernel: f2b/f.sshd[81998]: segfault at 0 ip 00007ffbfcd3e7f7 sp 00007ffbd97fd8b8 error 4 in libc-2.31.so[7ffbfccc4000+150000]
kernel: Code: 00 00 00 f3 0f 1e fa 66 0f 6e ce 89 f8 25 ff 0f 00 00 66 0f 60 c9 3d c0 0f 00 00 66 0f 61 c9 66 0f 70 c9 00 0f 8f 69 01 00 00 <f3> 0f 6f 07 66 0f ef db 66 0f 6f e0 66 0f 74 c1 66 0f 74 e3 66 0f
systemd[1]: Started Process Core Dump (PID 82000/UID 0).
systemd-coredump[82001]: Process 81993 (f2b/server) of user 0 dumped core.

Stack trace of thread 81998:
#0  0x00007ffbfcd3e7f7 __strchr_sse2 (libc.so.6 + 0x9f7f7)
#1  0x00007ffbf450fafc dirname_has_namespace (libsystemd.so.0 + 0x6cafc)
#2  0x00007ffbf45173f9 add_directory (libsystemd.so.0 + 0x743f9)
#3  0x00007ffbf4517c83 reiterate_all_paths (libsystemd.so.0 + 0x74c83)
#4  0x00007ffbf4517da2 sd_journal_get_fd (libsystemd.so.0 + 0x74da2)
#5  0x00007ffbf4518530 sd_journal_wait (libsystemd.so.0 + 0x75530)
#6  0x00007ffbf4eea778 Reader_wait (_reader.cpython-38-x86_64-linux-gnu.so + 0x5778)
#7  0x00007ffbfca8afe4 cfunction_call_varargs (libpython3.8.so.1.0 + 0x11ffe4)
#8  0x00007ffbfca7b823 _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x110823)
#9  0x00007ffbfca78948 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10d948)
#10 0x00007ffbfca72705 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107705)
#11 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#12 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#13 0x00007ffbfca72705 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107705)
#14 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#15 0x00007ffbfca73744 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108744)
#16 0x00007ffbfca728d5 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x1078d5)
#17 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#18 0x00007ffbfca784d7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10d4d7)
#19 0x00007ffbfca728d5 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x1078d5)
#20 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#21 0x00007ffbfca897ec method_vectorcall (libpython3.8.so.1.0 + 0x11e7ec)
#22 0x00007ffbfca89a90 PyVectorcall_Call (libpython3.8.so.1.0 + 0x11ea90)
#23 0x00007ffbfca75736 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10a736)
#24 0x00007ffbfca72705 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107705)
#25 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#26 0x00007ffbfca784d7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10d4d7)
#27 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#28 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#29 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#30 0x00007ffbfca89603 method_vectorcall (libpython3.8.so.1.0 + 0x11e603)
#31 0x00007ffbfca784d7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10d4d7)
#32 0x00007ffbfca72705 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107705)
#33 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#34 0x00007ffbfca897ec method_vectorcall (libpython3.8.so.1.0 + 0x11e7ec)
#35 0x00007ffbfca89a90 PyVectorcall_Call (libpython3.8.so.1.0 + 0x11ea90)
#36 0x00007ffbfcb23e81 t_bootstrap (libpython3.8.so.1.0 + 0x1b8e81)
#37 0x00007ffbfcb23e24 pythread_wrapper (libpython3.8.so.1.0 + 0x1b8e24)
#38 0x00007ffbfc952432 start_thread (libpthread.so.0 + 0x9432)
#39 0x00007ffbfcda09d3 __clone (libc.so.6 + 0x1019d3)

Stack trace of thread 81993:
#0  0x00007ffbfca7348a _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10848a)
#1  0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#2  0x00007ffbfca87949 property_descr_get (libpython3.8.so.1.0 + 0x11c949)
#3  0x00007ffbfca7fe87 _PyObject_GenericGetAttrWithDict (libpython3.8.so.1.0 + 0x114e87)
#4  0x00007ffbfca73bfc _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108bfc)
#5  0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#6  0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#7  0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#8  0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#9  0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#10 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#11 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#12 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#13 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#14 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#15 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#16 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#17 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#18 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#19 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#20 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#21 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#22 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#23 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#24 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#25 0x00007ffbfca73744 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108744)
#26 0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#27 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#28 0x00007ffbfca73744 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108744)
#29 0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#30 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#31 0x00007ffbfca7460a _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10960a)
#32 0x00007ffbfca728d5 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x1078d5)
#33 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#34 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#35 0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#36 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#37 0x00007ffbfca89603 method_vectorcall (libpython3.8.so.1.0 + 0x11e603)
#38 0x00007ffbfca7460a _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10960a)
#39 0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#40 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#41 0x00007ffbfca784d7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10d4d7)
#42 0x00007ffbfca728d5 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x1078d5)
#43 0x00007ffbfca80f59 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x115f59)
#44 0x00007ffbfca73b3b _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108b3b)
#45 0x00007ffbfca812b8 function_code_fastcall (libpython3.8.so.1.0 + 0x1162b8)
#46 0x00007ffbfca73744 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x108744)
#47 0x00007ffbfca72231 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x107231)
#48 0x00007ffbfcaed1c9 PyEval_EvalCodeEx (libpython3.8.so.1.0 + 0x1821c9)
#49 0x00007ffbfcaed18b PyEval_EvalCode (libpython3.8.so.1.0 + 0x18218b)
#50 0x00007ffbfcb0dc98 run_eval_code_obj (libpython3.8.so.1.0 + 0x1a2c98)
#51 0x00007ffbfcb0d3c3 run_mod (libpython3.8.so.1.0 + 0x1a23c3)
#52 0x00007ffbfc9f9a62 PyRun_FileExFlags (libpython3.8.so.1.0 + 0x8ea62)
#53 0x00007ffbfc9f93fe PyRun_SimpleFileExFlags (libpython3.8.so.1.0 + 0x8e3fe)
#54 0x00007ffbfc9f0661 Py_RunMain.cold (libpython3.8.so.1.0 + 0x85661)
#55 0x00007ffbfcae0aa9 Py_BytesMain (libpython3.8.so.1.0 + 0x175aa9)
#56 0x00007ffbfccc6042 __libc_start_main (libc.so.6 + 0x27042)
#57 0x00005616e504d07e _start (python3.8 + 0x107e)

Stack trace of thread 81994:
#0  0x00007ffbfcda1c0c recv (libc.so.6 + 0x102c0c)
#1  0x00

gdb:

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/python3 -s /usr/bin/fail2ban-server -xf --logtarget=stdout --loglevel='.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strchr_sse2 () at ../sysdeps/x86_64/multiarch/../strchr.S:32
32              movdqu  (%rdi), %xmm0
[Current thread is 1 (Thread 0x7f24097ff700 (LWP 82442))]

What's the ETA on this fix getting into the testing repo?