-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A fuzzer for journal-remote #9014
Commits on May 31, 2018
-
Configuration menu - View commit details
-
Copy full SHA for 5d889c1 - Browse repository at this point
Copy the full SHA 5d889c1View commit details -
journal-remote: split out µhttpd support and main() into a separate file
This is in preparation to reusing the RemoteServer in other concepts. I tried to keep changes to minimum: - arg_* global variables are now passed as state in RemoteServer - exported functions get the "journal_remote_" prefix - some variables are renamed In particular, there is an ugly global RemoveServer* variable. It was originally added because µhttpd did not allow state to be passed to the callbacks. I'm not sure if this has been remediated in µhttpd, but either way, this is not changed here, the global variable is only renamed for clarity.
Configuration menu - View commit details
-
Copy full SHA for c064d8d - Browse repository at this point
Copy the full SHA c064d8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 864876e - Browse repository at this point
Copy the full SHA 864876eView commit details -
log-generator: make message size configurable, add short options
Also remove "b''" from the generated MESSAGE= field.
Configuration menu - View commit details
-
Copy full SHA for 757ed4f - Browse repository at this point
Copy the full SHA 757ed4fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 40c10d3 - Browse repository at this point
Copy the full SHA 40c10d3View commit details -
fuzz-journal-remote: allow fuzzer to be built without µhttpd
journal-remote still requires µhttpd, but things are easier if the fuzzer can be built without.
Configuration menu - View commit details
-
Copy full SHA for 63e2ebc - Browse repository at this point
Copy the full SHA 63e2ebcView commit details -
sd-id128: return -ENOMEDIUM on null id
We currently return -ENOMEDIUM when /etc/machine-id is empty, and -EINVAL when it is all zeros. But -EINVAL is also used for invalid args. The distinction between empty and all-zero is not very important, let's use the same return code. Also document -ENOENT and -ENOMEDIUM since they can be a bit surprising.
Configuration menu - View commit details
-
Copy full SHA for ea03f6b - Browse repository at this point
Copy the full SHA ea03f6bView commit details -
journal: allow writing journal files even if machine-id is missing
The code to open journal files seems like the wrong place to enforce this. We already check during boot and refuse to boot if machine-id is missing, no need to enforce this here. In particular, it seems better to write logs from journald even if they are not completely functional rather than refuse to operate at all, and systemd-journal-remote also writes journal files and may even be run on a system without systemd at all. The docker image that oss-fuzz uses has an empty /etc/machine-id. Obviously this is an error in the docker, but docker is fact of life, and it seems better for systemd-journal-remote to work in such an incomplete environment.
Configuration menu - View commit details
-
Copy full SHA for fd4885d - Browse repository at this point
Copy the full SHA fd4885dView commit details -
journal: rename output_journal to show_journal_entry
We have show_journal, and output_journal, and it's not immediately clear how they related. Rename the first to show that it just prints one entry.
Configuration menu - View commit details
-
Copy full SHA for 9b972c9 - Browse repository at this point
Copy the full SHA 9b972c9View commit details -
shared/logs-show: export show_journal()
This is a nice function to output some journal entries without much ado.
Configuration menu - View commit details
-
Copy full SHA for 889e396 - Browse repository at this point
Copy the full SHA 889e396View commit details -
Configuration menu - View commit details
-
Copy full SHA for a38f7fe - Browse repository at this point
Copy the full SHA a38f7feView commit details -
journal-remote: verify realtime and monotonic timestamps early
We would accept any value, and then journal_file_check_object() would reject the whole entry. Let's just ignore the field.
Configuration menu - View commit details
-
Copy full SHA for 41b0b12 - Browse repository at this point
Copy the full SHA 41b0b12View commit details -
journal-remote: when an entry is rejected with -EBADMSG, do not rotat…
…e the journal file Something is wrong with the entry (probably a missing timestamp), so no point in rotating. But suppress the error in process_source(), so that the processing of the data stream continues. Also, just return 0 from writer_write() on success, the only caller doesn't care.
Configuration menu - View commit details
-
Copy full SHA for e743ce7 - Browse repository at this point
Copy the full SHA e743ce7View commit details -
basic/journal-importer: do not write non-unicode char to log
The type of cescape_char() is changed to int to make it easier to use in "%.*s". We know the value is between 1 and 4, so size_t is overkill.
Configuration menu - View commit details
-
Copy full SHA for b778252 - Browse repository at this point
Copy the full SHA b778252View commit details -
basic/string-util: add a convenience function to cescape mostly-ascii…
… fields It's not supposed to be the most efficient, but instead fast and simple to use. I kept the logic in ellipsize_mem() to use unicode ellipsis even in non-unicode locales. I'm not quite convinced things should be this way, especially that with this patch it'd actually be simpler to always use "…" in unicode locale and "..." otherwise, but Lennart wanted it this way for some reason.
Configuration menu - View commit details
-
Copy full SHA for 8409f68 - Browse repository at this point
Copy the full SHA 8409f68View commit details -
basic/journal-importer: escape & ellipsize bad data in log entries
We shouldn't just log arbitrary stuff, in particular newlines and control chars Now: Unknown dunder line __CURSORFACILITY=6\nSYSLOG_IDENTIFIER=/USR/SBIN/CRON\nMES…, ignoring. Unknown dunder line __REALTIME_TIME[TAMP=1404101101501874\n__MONOTONIC_TIMEST…, ignoring.
Configuration menu - View commit details
-
Copy full SHA for cca24fc - Browse repository at this point
Copy the full SHA cca24fcView commit details -
Configuration menu - View commit details
-
Copy full SHA for bbdad08 - Browse repository at this point
Copy the full SHA bbdad08View commit details -
basic/random-util: do not use getrandom() under msan
`fuzz-journal-remote` seems to be failing under `msan` as soon as it starts: $ sudo infra/helper.py run_fuzzer systemd fuzz-journal-remote Running: docker run --rm -i --privileged -e FUZZING_ENGINE=libfuzzer -v /home/vagrant/oss-fuzz/build/out/systemd:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer fuzz-journal-remote Using seed corpus: fuzz-journal-remote_seed_corpus.zip /out/fuzz-journal-remote -rss_limit_mb=2048 -timeout=25 /tmp/fuzz-journal-remote_corpus -max_len=65536 < /dev/null INFO: Seed: 3380449479 INFO: Loaded 2 modules (36336 inline 8-bit counters): 36139 [0x7ff36ea31d39, 0x7ff36ea3aa64), 197 [0x9998c8, 0x99998d), INFO: Loaded 2 PC tables (36336 PCs): 36139 [0x7ff36ea3aa68,0x7ff36eac7d18), 197 [0x999990,0x99a5e0), INFO: 2 files found in /tmp/fuzz-journal-remote_corpus INFO: seed corpus: files: 2 min: 4657b max: 7790b total: 12447b rss: 97Mb Uninitialized bytes in __interceptor_pwrite64 at offset 24 inside [0x7fffdd4d7230, 240) ==15==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x7ff36e685e8a in journal_file_init_header /work/build/../../src/systemd/src/journal/journal-file.c:436:13 #1 0x7ff36e683a9d in journal_file_open /work/build/../../src/systemd/src/journal/journal-file.c:3333:21 #2 0x7ff36e68b8f6 in journal_file_open_reliably /work/build/../../src/systemd/src/journal/journal-file.c:3520:13 #3 0x4a3f35 in open_output /work/build/../../src/systemd/src/journal-remote/journal-remote.c:70:13 #4 0x4a34d0 in journal_remote_get_writer /work/build/../../src/systemd/src/journal-remote/journal-remote.c:136:21 #5 0x4a550f in get_source_for_fd /work/build/../../src/systemd/src/journal-remote/journal-remote.c:183:13 #6 0x4a46bd in journal_remote_add_source /work/build/../../src/systemd/src/journal-remote/journal-remote.c:235:13 #7 0x4a271c in LLVMFuzzerTestOneInput /work/build/../../src/systemd/src/fuzz/fuzz-journal-remote.c:36:9 #8 0x4f27cc in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:524:13 #9 0x4efa0b in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/libfuzzer/FuzzerLoop.cpp:448:3 #10 0x4f8e96 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /src/libfuzzer/FuzzerLoop.cpp:732:7 #11 0x4f9f73 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /src/libfuzzer/FuzzerLoop.cpp:752:3 #12 0x4bf329 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:756:6 #13 0x4ac391 in main /src/libfuzzer/FuzzerMain.cpp:20:10 #14 0x7ff36d14982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f) #15 0x41f9d8 in _start (/out/fuzz-journal-remote+0x41f9d8) Uninitialized value was stored to memory at #0 0x7ff36e61cd41 in sd_id128_randomize /work/build/../../src/systemd/src/libsystemd/sd-id128/sd-id128.c:288:16 #1 0x7ff36e685cec in journal_file_init_header /work/build/../../src/systemd/src/journal/journal-file.c:426:13 #2 0x7ff36e683a9d in journal_file_open /work/build/../../src/systemd/src/journal/journal-file.c:3333:21 #3 0x7ff36e68b8f6 in journal_file_open_reliably /work/build/../../src/systemd/src/journal/journal-file.c:3520:13 #4 0x4a3f35 in open_output /work/build/../../src/systemd/src/journal-remote/journal-remote.c:70:13 #5 0x4a34d0 in journal_remote_get_writer /work/build/../../src/systemd/src/journal-remote/journal-remote.c:136:21 #6 0x4a550f in get_source_for_fd /work/build/../../src/systemd/src/journal-remote/journal-remote.c:183:13 #7 0x4a46bd in journal_remote_add_source /work/build/../../src/systemd/src/journal-remote/journal-remote.c:235:13 #8 0x4a271c in LLVMFuzzerTestOneInput /work/build/../../src/systemd/src/fuzz/fuzz-journal-remote.c:36:9 #9 0x4f27cc in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:524:13 #10 0x4efa0b in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool*) /src/libfuzzer/FuzzerLoop.cpp:448:3 #11 0x4f8e96 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /src/libfuzzer/FuzzerLoop.cpp:732:7 #12 0x4f9f73 in fuzzer::Fuzzer::Loop(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, fuzzer::fuzzer_allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /src/libfuzzer/FuzzerLoop.cpp:752:3 #13 0x4bf329 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:756:6 #14 0x4ac391 in main /src/libfuzzer/FuzzerMain.cpp:20:10 #15 0x7ff36d14982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f) Uninitialized value was created by an allocation of 't' in the stack frame of function 'sd_id128_randomize' #0 0x7ff36e61cb00 in sd_id128_randomize /work/build/../../src/systemd/src/libsystemd/sd-id128/sd-id128.c:274 SUMMARY: MemorySanitizer: use-of-uninitialized-value /work/build/../../src/systemd/src/journal/journal-file.c:436:13 in journal_file_init_header Exiting MS: 0 ; base unit: 0000000000000000000000000000000000000000 artifact_prefix='./'; Test unit written to ./crash-847911777b3096783f4ee70a69ab6d28380c810b [vagrant@localhost oss-fuzz]$ sudo infra/helper.py check_build --sanitizer=memory systemd Running: docker run --rm -i --privileged -e FUZZING_ENGINE=libfuzzer -e SANITIZER=memory -v /home/vagrant/oss-fuzz/build/out/systemd:/out -t gcr.io/oss-fuzz-base/base-runner test_all INFO: performing bad build checks for /out/fuzz-dhcp-server. INFO: performing bad build checks for /out/fuzz-journal-remote. INFO: performing bad build checks for /out/fuzz-unit-file. INFO: performing bad build checks for /out/fuzz-dns-packet. 4 fuzzers total, 0 seem to be broken (0%). Check build passed. It's a false positive which is most likely caused by google/sanitizers#852. I think it could be got around by avoiding `getrandom` when the code is compiled with `msan`
Configuration menu - View commit details
-
Copy full SHA for 2e69f41 - Browse repository at this point
Copy the full SHA 2e69f41View commit details -
shared/logs-show: fix mixup between length-based memory duplication a…
…nd string operations We'd look for a '=' separator using memchr, i.e. ignoring any nul bytes in the string, but then do a strndup, which would terminate on any nul byte, and then again do a memcmp, which would access memory past the chunk allocated by strndup. Of course, we probably shouldn't allow keys with nul bytes in them. But we currently do, so there might be journal files like that out there. So let's fix the journal-reading code first.
Configuration menu - View commit details
-
Copy full SHA for 324d6aa - Browse repository at this point
Copy the full SHA 324d6aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9be391d - Browse repository at this point
Copy the full SHA 9be391dView commit details -
basic/journal-importer: reject any field names that journald would re…
…ject $ build-asan/fuzz-journal-remote test/fuzz-regressions/fuzz-journal-remote/crash-96dee870ea66d03e89ac321eee28ea63a9b9aa45 ... Ignoring invalid field: "S\020" Ignoring invalid field: "S\020" ... If the field name includes nul bytes, we won't print all of the name. But that seems enough of a corner case to ignore.
Configuration menu - View commit details
-
Copy full SHA for 1e44873 - Browse repository at this point
Copy the full SHA 1e44873View commit details -
basic/journal-importer: "trusted" fields in binary format are not sup…
…ported The parser never accepted "__"-prefixed fields in binary format, but there was a comment questioning this decision. Let's make it official, and remove the comment. Also, for clarity, let's move the dunder field parsing after the field verification check. This doesn't change much, because invalid fields cannot be known special fields, but is seems cleaner to first verify the validity of the name, and then check if it is one of the known ones.
Configuration menu - View commit details
-
Copy full SHA for bcac982 - Browse repository at this point
Copy the full SHA bcac982View commit details -
fuzz-journal-remote: write to /dev/null not stdout
This makes the fuzzing much more efficient. Optionally provide output is $SYSTEMD_FUZZ_OUTPUT is set, which makes debugging of any failures much easier. The case from 056129d is still detected properly.
Configuration menu - View commit details
-
Copy full SHA for 6dbef30 - Browse repository at this point
Copy the full SHA 6dbef30View commit details -
Always allow timestamps to be printed
If the timestamp is above 9999-12-30, (or 2038-something-something on 32 bit), use XXXX-XX-XX XX:XX:XX as the replacement. The problem with refusing to print timestamps is that our code accepts such timestamps, so we can't really just refuse to process them afterwards. Also, it makes journal files non-portable, because suddently we might completely refuse to print entries which are totally OK on a different machine.
Configuration menu - View commit details
-
Copy full SHA for d3d2802 - Browse repository at this point
Copy the full SHA d3d2802View commit details -
Use const char* for timestamp strings which we don't plan to modify
Makes the intent a bit clearer.
Configuration menu - View commit details
-
Copy full SHA for 4d9685b - Browse repository at this point
Copy the full SHA 4d9685bView commit details -
journal: refuse an entry with invalid timestamp fields
The journal verification functions would reject such an entry. It would probably still display fine (because we prefer _SOURCE_REALTIME_TIMESTAMP= if present), but it seems wrong to create an entry that would not pass verification.
Configuration menu - View commit details
-
Copy full SHA for c627395 - Browse repository at this point
Copy the full SHA c627395View commit details -
shared/logs-show: be more careful before using a _SOURCE_REALTIME_TIM…
…ESTAMP entry journalctl -o short would display those entries, but journalctl -o short-full would refuse. If the entry is bad, just fall back to the receive-side realtime timestamp like we would if it was completely missing.
Configuration menu - View commit details
-
Copy full SHA for 03d1319 - Browse repository at this point
Copy the full SHA 03d1319View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a271b0 - Browse repository at this point
Copy the full SHA 5a271b0View commit details -
journal: allow boot_id to be passed to journal_append_entry()
In this commit, this is done only in testing code, i.e. there is no functional change apart from tests.
Configuration menu - View commit details
-
Copy full SHA for d180c34 - Browse repository at this point
Copy the full SHA d180c34View commit details -
journal-remote: parse the _BOOT_ID field and use the value when writi…
…ng entries The boot id is stored twice, and different code paths use either one or the other. So we need to store it both in the header and as a field for full compatibility.
Configuration menu - View commit details
-
Copy full SHA for c0b6ada - Browse repository at this point
Copy the full SHA c0b6adaView commit details -
journal-remote: do not send _BOOT_ID twice
Also remove the comma from the comment everywhere, I think the comma unnecessarilly put emphasis on the clause after the comma. Fixes systemd#9090. Reproducer: systemd-journal-remote --split-mode=none -o /tmp/msg6.journal --trust=all --listen-http=8080 systemd-journal-upload -u http://localhost:8080 journalctl --file /tmp/msg6.journal -o verbose -n1
Configuration menu - View commit details
-
Copy full SHA for 0ab896b - Browse repository at this point
Copy the full SHA 0ab896bView commit details