systemd-journald drops all bytes after '\0'

[evverx]

I'm sorry for not being specific. I will have to update the description so that it won't cause confusion like #6838 (comment).

$ printf '<13>Sep 15 15:07:58 HOST: null\0here' | nc -w1 -u -U /run/systemd/journal/dev-log

$ journalctl -t HOST -o verbose
...
    MESSAGE=null

From #1460

Of course, this will make the patch more complicated, as this means we have to patch much further down to ensure we always keep the ptr+length bit intact.

[poettering]

Fix waiting in #6838.

[poettering]

Since ec20fe5 we will no consider '\n', '\0' as well as line size limits and EOF all as fully equivalent EOL markers. This means the command proposed will now cleanly result in multiple log lines generated. Here's an example:

$ printf 'ab\0cd\nef' | systemd-cat --identifier drop-after-zero
$ journalctl -n 3
-- Logs begin at Thu 2016-08-18 23:09:37 CEST, end at Thu 2018-05-24 13:21:25 CEST. --
Mai 24 13:21:25 sigma drop-after-zero[51513]: ab
Mai 24 13:21:25 sigma drop-after-zero[51513]: cd
Mai 24 13:21:25 sigma drop-after-zero[51513]: ef

I am pretty sure this new behaviour makes a ton of sense, and no data is lost. In fact the structure log message contains enough information to even reconstruct whether the line was broken due to \n, \0, line limit or EOF through the _LINE_BREAK= property:

-- Logs begin at Thu 2016-08-18 23:09:37 CEST, end at Thu 2018-05-24 13:22:41 CEST. --
Thu 2018-05-24 13:22:41.713886 CEST [s=65e4e3725e08482e859073adf7e3ab40;i=559c3;b=47906844deef4cbe971b732ad7db2129;m=364e98d427;t=56cf1dea346de;x=179b7582b4966cdb]
    _UID=1000
    _GID=1000
    _SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
    _MACHINE_ID=c455a9847b7b47cb8a6d206e1a8d04c0
    _HOSTNAME=sigma
    _CAP_EFFECTIVE=0
    _TRANSPORT=stdout
    PRIORITY=6
    _BOOT_ID=47906844deef4cbe971b732ad7db2129
    SYSLOG_IDENTIFIER=drop-after-zero
    _LINE_BREAK=nul
    MESSAGE=ab
    _COMM=cat
    _STREAM_ID=91bb4a4b07c248cd98d8ef74f29f6219
    _PID=51541
Thu 2018-05-24 13:22:41.713886 CEST [s=65e4e3725e08482e859073adf7e3ab40;i=559c4;b=47906844deef4cbe971b732ad7db2129;m=364e98d427;t=56cf1dea346de;x=76f0cb041ca50029]
    _UID=1000
    _GID=1000
    _SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
    _MACHINE_ID=c455a9847b7b47cb8a6d206e1a8d04c0
    _HOSTNAME=sigma
    _CAP_EFFECTIVE=0
    _TRANSPORT=stdout
    PRIORITY=6
    _BOOT_ID=47906844deef4cbe971b732ad7db2129
    SYSLOG_IDENTIFIER=drop-after-zero
    _COMM=cat
    MESSAGE=cd
    _STREAM_ID=91bb4a4b07c248cd98d8ef74f29f6219
    _PID=51541
Thu 2018-05-24 13:22:41.727466 CEST [s=65e4e3725e08482e859073adf7e3ab40;i=559c5;b=47906844deef4cbe971b732ad7db2129;m=364e990933;t=56cf1dea37bea;x=baf41e133020694f]
    _UID=1000
    _GID=1000
    _SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
    _MACHINE_ID=c455a9847b7b47cb8a6d206e1a8d04c0
    _HOSTNAME=sigma
    _CAP_EFFECTIVE=0
    _TRANSPORT=stdout
    PRIORITY=6
    _BOOT_ID=47906844deef4cbe971b732ad7db2129
    _LINE_BREAK=eof
    SYSLOG_IDENTIFIER=drop-after-zero
    _COMM=cat
    MESSAGE=ef
    _STREAM_ID=91bb4a4b07c248cd98d8ef74f29f6219
    _PID=51541

[evverx]

It's not fixed. That's why I wrote in the description that I should update it and we discussed it in #6838 (comment).

[poettering]

ah, indeed. ok, let's clarify on this issue hence, that the offending command is this:

printf '<13>Sep 15 15:07:58 HOST: null\0here' \| nc -w1 -u -U /run/systemd/journal/dev-log

And that indeed remains unfixed.

Sorry for the confusion

[poettering]

Hmm, so while I agree that simply ignoring everything coming after a NUL byte in a BSD syslog message is definitely the wrong approach, I am not sure what the right approach though is. If we get an UDP or AF_UNIX datagram with an embedded NUL byte, does this means the datagram contains two log records and the NUL is the separator? or does it mean that NUL shall be considered part of the log message?

[evverx]

I think the latter. According to https://tools.ietf.org/html/rfc5424

This document does not impose any mandatory restrictions on the MSG
or PARAM-VALUE content. As such, they MAY contain control
characters, including the NUL character.

In some programming languages (most notably C and C++), the NUL
character (ABNF %d00) traditionally has a special significance as
string terminator. Most implementations of these languages assume
that a string will not extend beyond the first NUL character. This
is primarily a restriction of the supporting run-time libraries.
This restriction is often carried over to programs and script
languages written in those languages. As such, NUL characters must
be considered with great care and be properly handled. An attacker
may deliberately include NUL characters to hide information after
them. Incorrect handling of the NUL character may also invalidate
cryptographic checksums that are transmitted inside the message.

rsyslog seems to replace control characters while receiving messages by default, but it's configurable via other knobs like escape-cc, drop-cc and escape-cc.

[evverx]

As shown in #10918, journald still uses string functions to handle binary messages and therefore loses everything after the first null byte. I guess the issue should be reopened.

[evverx]

The issue was also reported in #12484. I don't think it makes much sense to keep this one open.