New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-journald error causes 50+ load (normally 10) and stops logging. #322

Closed
popsikle opened this Issue Apr 8, 2015 · 16 comments

Comments

Projects
None yet
8 participants
@popsikle

popsikle commented Apr 8, 2015

Problems:
1 - Journal stops recording messages
2 - Load on the node skyrockets

Recurrence:
Multiple times a day

wp-cos-003 ~ # journalctl -f
-- Logs begin at Sat 2015-04-04 16:40:41 UTC. --
Apr 08 02:25:42 wp-cos-003 docker[5576]: [2015-04-08 02:25:42,591: ERROR/Worker-6418] network_error: Cannot connect to host
Apr 08 02:25:42 wp-cos-003 docker[5576]: Traceback (most recent call last):
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/tasks/queue.py", line 114, in _do_crawl
Apr 08 02:25:42 wp-cos-003 docker[5576]: ua_type=ua_type)
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/api/controllers/query.py", line 291, in query_all
Apr 08 02:25:42 wp-cos-003 docker[5576]: crawled_page = crawl_url(url, ua_type)
Apr 08 02:25:42 wp-cos-003 docker[5576]: File "/opt/crick/releases/0054c10264503446aac25ff4ee59b1f137d8c175/crick/api/controllers/query.py", line 168, in crawl_url
Apr 08 02:25:42 wp-cos-003 docker[5576]: raise CrawlerSiteAuthorError(message)
Apr 08 02:25:42 wp-cos-003 docker[5576]: CrawlerSiteAuthorError: network_error: Cannot connect to host
Apr 08 02:25:42 wp-cos-003 docker[5576]: [2015-04-08 02:25:42,593: INFO/Worker-6418] Initializing redis connection pool on TCP:172.21.129.109:6391 [db:0]

Console and dmesg get spammed with failed to write entry messages:

wp-cos-003 ~ # dmesg
[43339.209945] systemd-journald[673]: Failed to write entry (18 items, 818 bytes), ignoring: Cannot assign requested address
[43339.221518] systemd-journald[673]: Failed to write entry (18 items, 862 bytes), ignoring: Cannot assign requested address
[43339.233061] systemd-journald[673]: Failed to write entry (18 items, 819 bytes), ignoring: Cannot assign requested address
[43339.526360] systemd-journald[673]: Failed to write entry (18 items, 858 bytes), ignoring: Cannot assign requested address
[43339.541282] systemd-journald[673]: Failed to write entry (18 items, 882 bytes), ignoring: Cannot assign requested address
.......
2500 more lines like above
.......
[43524.022977] systemd-journald[673]: Failed to write entry (18 items, 773 bytes), ignoring: Cannot assign requested address
[43524.034500] systemd-journald[673]: Failed to write entry (18 items, 669 bytes), ignoring: Cannot assign requested address
[43524.045984] systemd-journald[673]: Failed to write entry (18 items, 690 bytes), ignoring: Cannot assign requested address
[43524.057504] systemd-journald[673]: Failed to write entry (18 items, 644 bytes), ignoring: Cannot assign requested address
[43524.069037] systemd-journald[673]: Failed to write entry (18 items, 722 bytes), ignoring: Cannot assign requested address
[43524.080546] systemd-journald[673]: Failed to write entry (18 items, 626 bytes), ignoring: Cannot assign requested address
[43524.092081] systemd-journald[673]: Failed to write entry (18 items, 732 bytes), ignoring: Cannot assign requested address
[43524.103649] systemd-journald[673]: Failed to write entry (18 items, 648 bytes), ignoring: Cannot assign requested address
[43524.115194] systemd-journald[673]: Failed to write entry (18 items, 732 bytes), ignoring: Cannot assign requested address
[43524.132051] systemd-journald[673]: Failed to write entry (18 items, 647 bytes), ignoring: Cannot assign requested address
[43524.143572] systemd-journald[673]: Failed to write entry (18 items, 671 bytes), ignoring: Cannot assign requested address
[43524.155071] systemd-journald[673]: Failed to write entry (18 items, 768 bytes), ignoring: Cannot assign requested address
[43524.166565] systemd-journald[673]: Failed to write entry (18 items, 760 bytes), ignoring: Cannot assign requested address
[43524.178142] systemd-journald[673]: Failed to write entry (18 items, 898 bytes), ignoring: Cannot assign requested address
[43524.592761] systemd-journald[673]: Failed to write entry (18 items, 899 bytes), ignoring: Cannot assign requested address
[43524.936610] systemd-journald[673]: Failed to write entry (18 items, 898 bytes), ignoring: Cannot assign requested address
[43525.055179] systemd-journald[673]: Failed to write entry (18 items, 770 bytes), ignoring: Cannot assign requested address

Disk space / brts not an issue:

wp-cos-003 ~ # journalctl --disk-usage
Archived and active journals take up 3.9G on disk.

wp-cos-003 ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         24G     0   24G   0% /dev
tmpfs            24G     0   24G   0% /dev/shm
tmpfs            24G  3.0M   24G   1% /run
tmpfs            24G     0   24G   0% /sys/fs/cgroup
/dev/sda9        72G   11G   58G  16% /
/dev/sda4       985M  334M  601M  36% /usr
tmpfs            24G     0   24G   0% /media
tmpfs            24G     0   24G   0% /tmp
/dev/sda6       108M   68K   99M   1% /usr/share/oem
/dev/sr0        374K  374K     0 100% /media/configdrive
/dev/sdb1       630G  3.4G  627G   1% /var/lib/docker

wp-cos-003 ~ # df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
devtmpfs        6175686    392  6175294    1% /dev
tmpfs           6187028      1  6187027    1% /dev/shm
tmpfs           6187028   1264  6185764    1% /run
tmpfs           6187028     14  6187014    1% /sys/fs/cgroup
/dev/sda9       4739808 106178  4633630    3% /
/dev/sda4        260096   5283   254813    3% /usr
tmpfs           6187028      2  6187026    1% /media
tmpfs           6187028      8  6187020    1% /tmp
/dev/sda6         32768     15    32753    1% /usr/share/oem
/dev/sr0              0      0        0     - /media/configdrive
/dev/sdb1      41943040 511906 41431134    2% /var/lib/docker

Verify Command:

wp-cos-003 ~ # journalctl --verify
e95fa0: (null) decompression failed: Bad message
e95fa0: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f67023-000513030a406af4.journal:e95fa0 (of 41943040 bytes, 36%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f67023-000513030a406af4.journal (Bad message)
76c71a0: invalid object
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fa13b5-0005131340ec7fe6.journal:76c71a0 (of 125829120 bytes, 98%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fa13b5-0005131340ec7fe6.journal (Cannot assign requested address)
7ffb0d0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000010b2601-00051328456c9353.journal
7f74250: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000cfdb1a-000512e9fb70cd34.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000001152983-0005132be772f0bd.journal
7fffe28: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e47076-000512f89882f962.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000101bfb3-00051325200c41da.journal
7ffff00: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000eea99e-000512fcab7226d9.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-00000000010da12d-000513292476cca5.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e4986f-000512f8a8c0c80e.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fc78fb-00051323041f7b3e.journal
000000: missing entry array
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal:000000 (of 8388608 bytes, 0%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal (Bad message)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001064115-0005132694e3274c.journal
7fffe68: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e996cc-000512faa86de579.journal
4b54d88: invalid object
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal:4b54d88 (of 83886080 bytes, 94%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal (Cannot assign requested address)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000ce0af6-000512e94e5ba149.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d26628-000512ec47d9fb3e.journal
7fffe48: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000ec1c69-000512fbaa1cd509.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fef00b-000513242cfd2f60.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f7787a-000513128d10345c.journal
32a8e10: (null) decompression failed: Bad message
32a8e10: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001016081-0005132507b717ac.journal:32a8e10 (of 134217728 bytes, 39%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001016081-0005132507b717ac.journal (Bad message)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d5be82-000512f1655f644a.journal
7fffe98: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f3c7ea-000512fefa69a348.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000f495cf-000513003bf2ad3f.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000001105df2-0005132a1c5de7ae.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000103d8c7-00051325c3a25c3c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e1da9d-000512f78adccbc3.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000df5f2c-000512f68bde359e.journal
7fffeb0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000001101da2-0005132a05f262e5.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000dcb237-000512f577180322.journal
7fffe38: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d50f10-000512f057c4def2.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000df3cca-000512f67db8b5ab.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000108ce8e-000513276b912538.journal
7ff9b98: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000115102f-0005132bddf50f07.journal
7fffe00: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e1d75f-000512f78a54363c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-000000000112989c-0005132af2fedb9c.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000cd4b83-000512e8fd58688c.journal
7ffff48: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000fc76f1-0005132303e3acf7.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d35e82-000512edcf1ddc8d.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000010da078-000513292467e738.journal
7fffe18: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000d7ad0e-000512f3f3dd6678.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000f6c9d6-0005130378088072.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000e708d1-000512f9a4627649.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000fb8b05-0005131401a55f04.journal
7fffec0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000dca261-000512f57166fb86.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000cc9aac-000512e8b9c13116.journal
7fffea0: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000e7065d-000512f9a346ea0c.journal
7fffe40: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-00000000011296e0-0005132af27d1b3c.journal
7fffd78: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f13907-000512fdab311d0b.journal
65b28b0: (null) decompression failed: Bad message
65b28b0: invalid object contents: Bad message
File corruption detected at /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000da08b6-000512f46a364904.journal:65b28b0 (of 134217728 bytes, 79%).
FAIL: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000da08b6-000512f46a364904.journal (Bad message)
7fffed8: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-000000000108b1a3-0005132760d916bb.journal
7fffe78: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000feec4c-000513242b875c8d.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000ed2d75-000512fc1b9ee023.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f71e50-00051312660d5cdb.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000da0d0a-000512f46b169af6.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-0000000000d8836b-000512f419dd34c7.journal
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000@c9eb85cba0f44e24b1a4b14680ca8002-00000000010b809f-000513286600498d.journal
7fffe68: unused data (entry_offset==0)
PASS: /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system@790dce070238465c94e16fe4608d7ccf-0000000000f77c5d-000513128dcdb093.journal
wp-cos-003 ~ # lsof -p $(pidof /usr/lib/systemd/systemd-journald)
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
systemd-j 673 root  cwd    DIR                8,9     4096        2 /
systemd-j 673 root  rtd    DIR                8,9     4096        2 /
systemd-j 673 root  txt    REG                8,4   264720    34686 /usr/lib64/systemd/systemd-journald
systemd-j 673 root  mem    REG                8,9 83886080     6701 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal
systemd-j 673 root  mem    REG                8,9  8388608     6712 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal
systemd-j 673 root  mem    REG                8,4    18160    34196 /usr/lib64/libgpg-error.so.0.10.0
systemd-j 673 root  mem    REG                8,4  1787096    33884 /usr/lib64/libc-2.19.so
systemd-j 673 root  mem    REG                8,4   109288    33865 /usr/lib64/libpthread-2.19.so
systemd-j 673 root  mem    REG                8,4    31048    33873 /usr/lib64/librt-2.19.so
systemd-j 673 root  mem    REG                8,4   577128    34321 /usr/lib64/libgcrypt.so.11.8.2
systemd-j 673 root  mem    REG                8,4   153400    34193 /usr/lib64/liblzma.so.5.0.8
systemd-j 673 root  mem    REG                8,4   144720    33863 /usr/lib64/ld-2.19.so
systemd-j 673 root  mem    REG               0,18        8    12333 /run/systemd/journal/kernel-seqnum
systemd-j 673 root    0r   CHR                1,3      0t0     1028 /dev/null
systemd-j 673 root    1w   CHR                1,3      0t0     1028 /dev/null
systemd-j 673 root    2w   CHR                1,3      0t0     1028 /dev/null
systemd-j 673 root    3u  unix 0xffff880858da1180      0t0    24600 /run/systemd/journal/dev-log
systemd-j 673 root    4u  unix 0xffff880858da0700      0t0    24592 /run/systemd/journal/stdout
systemd-j 673 root    5u  unix 0xffff880858da0a80      0t0    24595 /run/systemd/journal/socket
systemd-j 673 root    6w   CHR               1,11      0t0     1034 /dev/kmsg
systemd-j 673 root    7u  0000               0,10        0     8829 anon_inode
systemd-j 673 root    8u  0000               0,10        0     8829 anon_inode
systemd-j 673 root    9u   CHR               1,11      0t0     1034 /dev/kmsg
systemd-j 673 root   10r   REG                0,4        0    12334 /proc/sys/kernel/hostname
systemd-j 673 root   11u  0000               0,10        0     8829 anon_inode
systemd-j 673 root   12u  0000               0,10        0     8829 anon_inode
systemd-j 673 root   13u  unix 0xffff88072dd4d400      0t0  5747188 /run/systemd/journal/stdout
systemd-j 673 root   14u  unix 0xffff880858c27700      0t0      665 /run/systemd/journal/stdout
systemd-j 673 root   15u  unix 0xffff880c687f5780      0t0    14527 /run/systemd/journal/stdout
systemd-j 673 root   16u  unix 0xffff880c68a3e200      0t0    18145 /run/systemd/journal/stdout
systemd-j 673 root   17u  unix 0xffff880c687b3100      0t0    14532 /run/systemd/journal/stdout
systemd-j 673 root   18u  unix 0xffff880c687b6200      0t0    13494 /run/systemd/journal/stdout
systemd-j 673 root   19u  unix 0xffff8806b52f3f00      0t0 22562921 /run/systemd/journal/stdout
systemd-j 673 root   20u  unix 0xffff880c687b0380      0t0    13602 /run/systemd/journal/stdout
systemd-j 673 root   21u  unix 0xffff880c687b4600      0t0    13509 /run/systemd/journal/stdout
systemd-j 673 root   22u  unix 0xffff88084f5b3100      0t0 22251114 /run/systemd/journal/stdout
systemd-j 673 root   23u  unix 0xffff880c687b3b80      0t0    13511 /run/systemd/journal/stdout
systemd-j 673 root   24u  unix 0xffff880c686c6580      0t0    24991 /run/systemd/journal/stdout
systemd-j 673 root   25u  unix 0xffff88084f5b2a00      0t0 22625095 /run/systemd/journal/stdout
systemd-j 673 root   26u  unix 0xffff880c686c6c80      0t0    21731 /run/systemd/journal/stdout
systemd-j 673 root   27u  unix 0xffff8805147c9180      0t0 22588496 /run/systemd/journal/stdout
systemd-j 673 root   28u  unix 0xffff88085059a680      0t0 22280613 /run/systemd/journal/stdout
systemd-j 673 root   29u  unix 0xffff8807162e5080      0t0 22599650 /run/systemd/journal/stdout
systemd-j 673 root   30u  unix 0xffff88041e6ae900      0t0 22622230 /run/systemd/journal/stdout
systemd-j 673 root   31u  unix 0xffff88057c5f5400      0t0 22604510 /run/systemd/journal/stdout
systemd-j 673 root   32u  unix 0xffff880856b56200      0t0 22904697 /run/systemd/journal/stdout
systemd-j 673 root   33u  unix 0xffff880001e0d400      0t0 22904698 /run/systemd/journal/stdout
systemd-j 673 root   34u  unix 0xffff8806ca0f8700      0t0 22904699 /run/systemd/journal/stdout
systemd-j 673 root   35u  unix 0xffff880858e35b00      0t0 22686224 /run/systemd/journal/stdout
systemd-j 673 root   36u  unix 0xffff880858136200      0t0 22686225 /run/systemd/journal/stdout
systemd-j 673 root   37u  unix 0xffff88057c708380      0t0 22686226 /run/systemd/journal/stdout
systemd-j 673 root   38u  unix 0xffff8806a69a8000      0t0 22692030 /run/systemd/journal/stdout
systemd-j 673 root   39u  unix 0xffff88085059bf00      0t0 22708104 /run/systemd/journal/stdout
systemd-j 673 root   40u  unix 0xffff88085059b480      0t0 22704104 /run/systemd/journal/stdout
systemd-j 673 root   41u  unix 0xffff880945269500      0t0 22715904 /run/systemd/journal/stdout
systemd-j 673 root   42u  unix 0xffff880557e0d080      0t0 22714438 /run/systemd/journal/stdout
systemd-j 673 root   43u  unix 0xffff880557e09f80      0t0 22714439 /run/systemd/journal/stdout
systemd-j 673 root   44u  unix 0xffff8807162e6c80      0t0 22742614 /run/systemd/journal/stdout
systemd-j 673 root   45u  unix 0xffff880523246580      0t0 22776062 /run/systemd/journal/stdout
systemd-j 673 root   46u  unix 0xffff880851e12d80      0t0 22776063 /run/systemd/journal/stdout
systemd-j 673 root   47u  unix 0xffff880851fbb480      0t0 22776064 /run/systemd/journal/stdout
systemd-j 673 root   48u  unix 0xffff880856b57380      0t0 22776107 /run/systemd/journal/stdout
systemd-j 673 root   49u  unix 0xffff880523245780      0t0 22808741 /run/systemd/journal/stdout
systemd-j 673 root   50u  unix 0xffff880c6804bb80      0t0 22776108 /run/systemd/journal/stdout
systemd-j 673 root   51u  unix 0xffff880c66ea5400      0t0 22808742 /run/systemd/journal/stdout
systemd-j 673 root   52u  unix 0xffff880c66ea0000      0t0 22808743 /run/systemd/journal/stdout
systemd-j 673 root   53u  unix 0xffff8808574a6580      0t0 22812590 /run/systemd/journal/stdout
systemd-j 673 root   54u  unix 0xffff8808574a4600      0t0 22812591 /run/systemd/journal/stdout
systemd-j 673 root   55u  unix 0xffff880850b9ec80      0t0 22844553 /run/systemd/journal/stdout
systemd-j 673 root   56u  unix 0xffff880851fb9880      0t0 22826885 /run/systemd/journal/stdout
systemd-j 673 root   57u  unix 0xffff8808580ead80      0t0 22855102 /run/systemd/journal/stdout
systemd-j 673 root   58u  unix 0xffff8808503de200      0t0 22861991 /run/systemd/journal/stdout
systemd-j 673 root   59u  unix 0xffff880c66ea5780      0t0 22861992 /run/systemd/journal/stdout
systemd-j 673 root   60u  unix 0xffff88084d9e9f80      0t0 22868703 /run/systemd/journal/stdout
systemd-j 673 root   61u  unix 0xffff88057c517380      0t0 22864259 /run/systemd/journal/stdout
systemd-j 673 root   62u  unix 0xffff880523241f80      0t0 22868704 /run/systemd/journal/stdout
systemd-j 673 root   63u  unix 0xffff880523244980      0t0 22868705 /run/systemd/journal/stdout
systemd-j 673 root   64u  unix 0xffff880858ea5080      0t0 25186561 /run/systemd/journal/stdout
systemd-j 673 root   65u  unix 0xffff88084cd71f80      0t0 22931995 /run/systemd/journal/stdout
systemd-j 673 root   66u  unix 0xffff88085059bb80      0t0 22931996 /run/systemd/journal/stdout
systemd-j 673 root   67u  unix 0xffff88084c450000      0t0 22931997 /run/systemd/journal/stdout
systemd-j 673 root   68u  unix 0xffff880001e0c280      0t0 22904700 /run/systemd/journal/stdout
systemd-j 673 root   69u  unix 0xffff88057c70bb80      0t0 22281130 /run/systemd/journal/stdout
systemd-j 673 root   70u   REG                8,9 83886080     6701 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/system.journal
systemd-j 673 root   71u  unix 0xffff88085a6d2680      0t0 22284303 /run/systemd/journal/stdout
systemd-j 673 root   72u  unix 0xffff880c68119500      0t0 22654482 /run/systemd/journal/stdout
systemd-j 673 root   73u  unix 0xffff880c68258a80      0t0 25280427 /run/systemd/journal/stdout
systemd-j 673 root   74u  unix 0xffff88057c5f3800      0t0 22307568 /run/systemd/journal/stdout
systemd-j 673 root   75u  unix 0xffff88057c709880      0t0 22307569 /run/systemd/journal/stdout
systemd-j 673 root   76u   REG                8,9  8388608     6712 /var/log/journal/e5e1a21afbc94aebb6ad24d684e0525b/user-1000.journal
systemd-j 673 root   77u  unix 0xffff880858ea2680      0t0 25204054 /run/systemd/journal/stdout

The only solution I have found to bring load back under control and restore logging (tbh the load issue is the main problem here) is forcing the process to restart by kill -SIGUSR2 $(pidof /usr/lib/systemd/systemd-journald)

@popsikle

This comment has been minimized.

Show comment
Hide comment
@popsikle

popsikle Apr 8, 2015

wp-cos-001 bin # systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib64/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2015-04-08 03:11:19 UTC; 15min ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 681 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─681 /usr/lib/systemd/systemd-journald

Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Runtime journal is using 8.0M (max allowed 2.3G, trying to leave 3.5G free of 23.5G available → current limit 2.3G).
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Permanent journal is using 4.0G (max allowed 4.0G, trying to leave 4.0G free of 58.0G available → current limit 4.0G).
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Time spent on flushing to /var is 29.621ms for 2 entries.
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Journal started
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

popsikle commented Apr 8, 2015

wp-cos-001 bin # systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib64/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2015-04-08 03:11:19 UTC; 15min ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 681 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─681 /usr/lib/systemd/systemd-journald

Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Runtime journal is using 8.0M (max allowed 2.3G, trying to leave 3.5G free of 23.5G available → current limit 2.3G).
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Permanent journal is using 4.0G (max allowed 4.0G, trying to leave 4.0G free of 58.0G available → current limit 4.0G).
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Time spent on flushing to /var is 29.621ms for 2 entries.
Apr 08 03:11:19 wp-cos-001 systemd-journal[681]: Journal started
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
@cameronmaske

This comment has been minimized.

Show comment
Hide comment
@cameronmaske

cameronmaske Apr 15, 2015

Experiencing the similar problem. Happens very shortly after starting a new box with a few small running services.

$ journalctl --verify
14b768: unused data (entry_offset==0)                                                                           
14b7c0: unused data (entry_offset==0)                                                                           
14b858: unused data (entry_offset==0)                                                                           
14b8a8: unused data (entry_offset==0)                                                                           
14b8f8: unused data (entry_offset==0)                                                                           
14b950: invalid object                                                                                          
File corruption detected at /var/log/journal/ec895e36df29413baa0559c9ddf11d6c/system.journal:14b950 (of 8
systemctl status systemd-journald.service -l
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib64/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2015-04-15 18:14:52 UTC; 35min ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 390 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─390 /usr/lib/systemd/systemd-journald

Apr 15 18:14:52 localhost systemd-journal[390]: Runtime journal is using 8.0M (max allowed 188.4M, trying to leave 282.6M free of 1.8G available → current limit 188.4M).
Apr 15 18:14:52 localhost systemd-journal[390]: Permanent journal is using 8.0M (max allowed 203.0M, trying to leave 304.5M free of 1.8G available → current limit 203.0M).
Apr 15 18:14:52 localhost systemd-journal[390]: Time spent on flushing to /var is 564us for 2 entries.
Apr 15 18:14:52 localhost systemd-journal[390]: Journal started
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

cameronmaske commented Apr 15, 2015

Experiencing the similar problem. Happens very shortly after starting a new box with a few small running services.

$ journalctl --verify
14b768: unused data (entry_offset==0)                                                                           
14b7c0: unused data (entry_offset==0)                                                                           
14b858: unused data (entry_offset==0)                                                                           
14b8a8: unused data (entry_offset==0)                                                                           
14b8f8: unused data (entry_offset==0)                                                                           
14b950: invalid object                                                                                          
File corruption detected at /var/log/journal/ec895e36df29413baa0559c9ddf11d6c/system.journal:14b950 (of 8
systemctl status systemd-journald.service -l
● systemd-journald.service - Journal Service
   Loaded: loaded (/usr/lib64/systemd/system/systemd-journald.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2015-04-15 18:14:52 UTC; 35min ago
     Docs: man:systemd-journald.service(8)
           man:journald.conf(5)
 Main PID: 390 (systemd-journal)
   Status: "Processing requests..."
   CGroup: /system.slice/systemd-journald.service
           └─390 /usr/lib/systemd/systemd-journald

Apr 15 18:14:52 localhost systemd-journal[390]: Runtime journal is using 8.0M (max allowed 188.4M, trying to leave 282.6M free of 1.8G available → current limit 188.4M).
Apr 15 18:14:52 localhost systemd-journal[390]: Permanent journal is using 8.0M (max allowed 203.0M, trying to leave 304.5M free of 1.8G available → current limit 203.0M).
Apr 15 18:14:52 localhost systemd-journal[390]: Time spent on flushing to /var is 564us for 2 entries.
Apr 15 18:14:52 localhost systemd-journal[390]: Journal started
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
@narma

This comment has been minimized.

Show comment
Hide comment
@narma

narma May 8, 2015

Any updates? still experiencing this problem with 633.1.0

narma commented May 8, 2015

Any updates? still experiencing this problem with 633.1.0

@wptad

This comment has been minimized.

Show comment
Hide comment
@wptad

wptad Aug 24, 2015

681.2.0 also has such problem :(

wptad commented Aug 24, 2015

681.2.0 also has such problem :(

@crawford

This comment has been minimized.

Show comment
Hide comment
@crawford

crawford Aug 24, 2015

Member

The systemd in 681.2.0 is three versions behind. It'd be worth it to try a newer image.

Member

crawford commented Aug 24, 2015

The systemd in 681.2.0 is three versions behind. It'd be worth it to try a newer image.

@mhenniges

This comment has been minimized.

Show comment
Hide comment
@mhenniges

mhenniges Oct 19, 2015

Is this issue still being worked? corrupt entries are still happening for me as of 766.4.0.

mhenniges commented Oct 19, 2015

Is this issue still being worked? corrupt entries are still happening for me as of 766.4.0.

@mischief

This comment has been minimized.

Show comment
Hide comment
@mischief

mischief Oct 19, 2015

@mhenniges do you have a way to reproduce the issue?

mischief commented Oct 19, 2015

@mhenniges do you have a way to reproduce the issue?

@mhenniges

This comment has been minimized.

Show comment
Hide comment
@mhenniges

mhenniges Oct 23, 2015

I do have a repro case. I've played with it a lot and I don't quite understand the essential nature of the case - its not purely length driven or content driven..anyways, here goes:

core-02 ~ # cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=766.4.0
VERSION_ID=766.4.0
BUILD_ID=
PRETTY_NAME="CoreOS 766.4.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

core-02 ~ # journalctl --verify
PASS: /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal                    

core-02 ~ # journalctl -f&
[1] 3194
-- Logs begin at Fri 2015-10-23 13:52:18 UTC. --
Oct 23 13:52:18 core-02 systemd-journal[3189]: Permanent journal is using 16.0M (max allowed 1.5G, trying to leave 2.3G free of 14.6G available → current limit 1.5G).
Oct 23 13:52:18 core-02 systemd-journal[3189]: Journal started
Oct 23 13:52:18 core-02 systemd[1]: Starting Flush Journal to Persistent Storage...
Oct 23 13:52:18 core-02 systemd[1]: Started Flush Journal to Persistent Storage.

core-02 ~ # echo "Invoked with flag00001=False key_options=None state=present user=first.last key=ssh-rsa aaaaaaNzaC1yc2EAAAABJQAAAQEAvnCH4C8nl8dt17kJDxwt8q6IePHZgh2OWGftns158CcbomFeGx3FNj0/IPchKt8n0ZjftHA0FN31NUeLUk28ebKn1BCtNBgvUeE3X7tK/xHB7RdfmnI4jivWPtrA2prKf95FNNOzaUcTpTx5mR6UQi6Xc/fGw8DTt1xbPs+Ox0YEBYOxa3O88L1OM0XORKjLAoU4PJkcuVzTkxZSpwnMGJIipQoj8aaaPJ8GVIvXw+HZ6pj7zrPsmxK9MYqPAXRO98AwEPjI+uO1GnnuBr2JDpRziprs+FqUzlPcfoZaJWc2TBbjo91+tB4vz9Nxaqvirez5/w== rsa-key-20141110  flag12=False manage_dir=True aaaaaaaaaaaa" | logger
Failed to get journal fields: Bad message
[1]+  Exit 1                  journalctl -f
core-02 ~ # journalctl --verify
391d78: bad object size (<= 64): 64                                                       
391d78: invalid object contents: Bad message                                              
File corruption detected at /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal:391d78 (of 8388608 bytes, 44%).
FAIL: /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal (Bad message)

journalctl functionality is restored after a rotate and restart (via kill-HUP), although the rotated file is still reported as corrupt.

mhenniges commented Oct 23, 2015

I do have a repro case. I've played with it a lot and I don't quite understand the essential nature of the case - its not purely length driven or content driven..anyways, here goes:

core-02 ~ # cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=766.4.0
VERSION_ID=766.4.0
BUILD_ID=
PRETTY_NAME="CoreOS 766.4.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

core-02 ~ # journalctl --verify
PASS: /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal                    

core-02 ~ # journalctl -f&
[1] 3194
-- Logs begin at Fri 2015-10-23 13:52:18 UTC. --
Oct 23 13:52:18 core-02 systemd-journal[3189]: Permanent journal is using 16.0M (max allowed 1.5G, trying to leave 2.3G free of 14.6G available → current limit 1.5G).
Oct 23 13:52:18 core-02 systemd-journal[3189]: Journal started
Oct 23 13:52:18 core-02 systemd[1]: Starting Flush Journal to Persistent Storage...
Oct 23 13:52:18 core-02 systemd[1]: Started Flush Journal to Persistent Storage.

core-02 ~ # echo "Invoked with flag00001=False key_options=None state=present user=first.last key=ssh-rsa aaaaaaNzaC1yc2EAAAABJQAAAQEAvnCH4C8nl8dt17kJDxwt8q6IePHZgh2OWGftns158CcbomFeGx3FNj0/IPchKt8n0ZjftHA0FN31NUeLUk28ebKn1BCtNBgvUeE3X7tK/xHB7RdfmnI4jivWPtrA2prKf95FNNOzaUcTpTx5mR6UQi6Xc/fGw8DTt1xbPs+Ox0YEBYOxa3O88L1OM0XORKjLAoU4PJkcuVzTkxZSpwnMGJIipQoj8aaaPJ8GVIvXw+HZ6pj7zrPsmxK9MYqPAXRO98AwEPjI+uO1GnnuBr2JDpRziprs+FqUzlPcfoZaJWc2TBbjo91+tB4vz9Nxaqvirez5/w== rsa-key-20141110  flag12=False manage_dir=True aaaaaaaaaaaa" | logger
Failed to get journal fields: Bad message
[1]+  Exit 1                  journalctl -f
core-02 ~ # journalctl --verify
391d78: bad object size (<= 64): 64                                                       
391d78: invalid object contents: Bad message                                              
File corruption detected at /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal:391d78 (of 8388608 bytes, 44%).
FAIL: /var/log/journal/fb18037e4744494b87435c8fb003e477/system.journal (Bad message)

journalctl functionality is restored after a rotate and restart (via kill-HUP), although the rotated file is still reported as corrupt.

@mischief

This comment has been minimized.

Show comment
Hide comment
@mischief

mischief Oct 23, 2015

@mhenniges this repro also appears to affect CoreOS 835.1.0 w/ systemd v225. cursory testing on my local gentoo installation shows it is handled correctly in systemd v226.

mischief commented Oct 23, 2015

@mhenniges this repro also appears to affect CoreOS 835.1.0 w/ systemd v225. cursory testing on my local gentoo installation shows it is handled correctly in systemd v226.

@mhenniges

This comment has been minimized.

Show comment
Hide comment
@mhenniges

mhenniges Oct 23, 2015

Hmm, OK, thanks. Does anybody know when systemd 226 can get on the road to stable?

mhenniges commented Oct 23, 2015

Hmm, OK, thanks. Does anybody know when systemd 226 can get on the road to stable?

@vcaputo

This comment has been minimized.

Show comment
Hide comment
@vcaputo

vcaputo Oct 23, 2015

I'm not convinced v226 fixes this, my attempt to reproduce it in v225 following the steps in this ticket have failed, so we don't have a perfect reproducer.

vcaputo commented Oct 23, 2015

I'm not convinced v226 fixes this, my attempt to reproduce it in v225 following the steps in this ticket have failed, so we don't have a perfect reproducer.

@mischief

This comment has been minimized.

Show comment
Hide comment
@mischief

mischief Oct 23, 2015

core@core1 ~ $ uptime
 23:19:08 up 6 days, 17:34,  1 user,  load average: 0.04, 0.04, 0.07
core@core1 ~ $ journalctl --verify
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000793abf-000520c6c8f53862.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000831810-000521a2d9939da0.journal                                               
7ffff00: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000085b259-000521f63a6eea78.journal                                               
7fffdd0: Unused data (entry_offset==0)                                                                                                                                                  
7fffe50: Unused data (entry_offset==0)                                                                                                                                                  
7fffee0: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000076a2fe-0005207f26154df6.journal                                               
7fc3998: Unused data (entry_offset==0)                                                                                                                                                  
7fc3a20: Unused data (entry_offset==0)                                                                                                                                                  
7fc3ac8: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007baee1-000520f596bf9fa6.journal                                               
7fddf98: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000809a16-0005216034fbd541.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000882a77-000522376806e71b.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007e288d-00052129eb8ff417.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007419c5-0005203ccf4a9520.journal                                               
core@core1 ~ $ journalctl -f -t core
-- Logs begin at Mon 2015-09-21 07:27:11 UTC. --
^C
core@core1 ~ $ journalctl -f -t core &
[1] 3301
core@core1 ~ $ -- Logs begin at Mon 2015-09-21 07:27:11 UTC. --
tK/xHB7RdfmnI4jivWPtrA2prKf95FNNOzaUcTpTx5mR6UQi6Xc/fGw8DTt1xbPs+Ox0YEBYOxa3O88L1OM0XORKjLAoU4PJkcuVzTkxZSpwnMGJIipQoj8aaaPJ8GVIvXw+HZ6pj7zrPsmxK9MYqPAXRO98AwEPjI+uO1GnnuBr2JDpRziprs+FqUzlPcfoZaJWc2TBbjo91+tB4vz9Nxaqvirez5/w== rsa-key-20141110  flag12=False manage_dir=True aaaaaaaaaaaa" | logger
core@core1 ~ $ Failed to get journal fields: Bad message

[1]+  Exit 1                  journalctl -f -t core
core@core1 ~ $ journalctl --verify
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000793abf-000520c6c8f53862.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000831810-000521a2d9939da0.journal                                               
7ffff00: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000085b259-000521f63a6eea78.journal                                               
7fffdd0: Unused data (entry_offset==0)                                                                                                                                                  
7fffe50: Unused data (entry_offset==0)                                                                                                                                                  
7fffee0: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000076a2fe-0005207f26154df6.journal                                               
7fc3998: Unused data (entry_offset==0)                                                                                                                                                  
7fc3a20: Unused data (entry_offset==0)                                                                                                                                                  
7fc3ac8: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007baee1-000520f596bf9fa6.journal                                               
7fddf98: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000809a16-0005216034fbd541.journal                                               
5de90b0: Bad object size (<= 64): 64                                                                                                                                                    
5de90b0: Envalid object contents: Bad message                                                                                                                                           
File corruption detected at /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal:5de90b0 (of 100663296 bytes, 97%).                                                         
FAIL: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal (Bad message)
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000882a77-000522376806e71b.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007e288d-00052129eb8ff417.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007419c5-0005203ccf4a9520.journal                                               
core@core1 ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=835.1.0
VERSION_ID=835.1.0
BUILD_ID=
PRETTY_NAME="CoreOS 835.1.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
core@core1 ~ $

mischief commented Oct 23, 2015

core@core1 ~ $ uptime
 23:19:08 up 6 days, 17:34,  1 user,  load average: 0.04, 0.04, 0.07
core@core1 ~ $ journalctl --verify
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000793abf-000520c6c8f53862.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000831810-000521a2d9939da0.journal                                               
7ffff00: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000085b259-000521f63a6eea78.journal                                               
7fffdd0: Unused data (entry_offset==0)                                                                                                                                                  
7fffe50: Unused data (entry_offset==0)                                                                                                                                                  
7fffee0: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000076a2fe-0005207f26154df6.journal                                               
7fc3998: Unused data (entry_offset==0)                                                                                                                                                  
7fc3a20: Unused data (entry_offset==0)                                                                                                                                                  
7fc3ac8: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007baee1-000520f596bf9fa6.journal                                               
7fddf98: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000809a16-0005216034fbd541.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000882a77-000522376806e71b.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007e288d-00052129eb8ff417.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007419c5-0005203ccf4a9520.journal                                               
core@core1 ~ $ journalctl -f -t core
-- Logs begin at Mon 2015-09-21 07:27:11 UTC. --
^C
core@core1 ~ $ journalctl -f -t core &
[1] 3301
core@core1 ~ $ -- Logs begin at Mon 2015-09-21 07:27:11 UTC. --
tK/xHB7RdfmnI4jivWPtrA2prKf95FNNOzaUcTpTx5mR6UQi6Xc/fGw8DTt1xbPs+Ox0YEBYOxa3O88L1OM0XORKjLAoU4PJkcuVzTkxZSpwnMGJIipQoj8aaaPJ8GVIvXw+HZ6pj7zrPsmxK9MYqPAXRO98AwEPjI+uO1GnnuBr2JDpRziprs+FqUzlPcfoZaJWc2TBbjo91+tB4vz9Nxaqvirez5/w== rsa-key-20141110  flag12=False manage_dir=True aaaaaaaaaaaa" | logger
core@core1 ~ $ Failed to get journal fields: Bad message

[1]+  Exit 1                  journalctl -f -t core
core@core1 ~ $ journalctl --verify
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000793abf-000520c6c8f53862.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000831810-000521a2d9939da0.journal                                               
7ffff00: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000085b259-000521f63a6eea78.journal                                               
7fffdd0: Unused data (entry_offset==0)                                                                                                                                                  
7fffe50: Unused data (entry_offset==0)                                                                                                                                                  
7fffee0: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-000000000076a2fe-0005207f26154df6.journal                                               
7fc3998: Unused data (entry_offset==0)                                                                                                                                                  
7fc3a20: Unused data (entry_offset==0)                                                                                                                                                  
7fc3ac8: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007baee1-000520f596bf9fa6.journal                                               
7fddf98: Unused data (entry_offset==0)                                                                                                                                                  
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000809a16-0005216034fbd541.journal                                               
5de90b0: Bad object size (<= 64): 64                                                                                                                                                    
5de90b0: Envalid object contents: Bad message                                                                                                                                           
File corruption detected at /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal:5de90b0 (of 100663296 bytes, 97%).                                                         
FAIL: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system.journal (Bad message)
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-0000000000882a77-000522376806e71b.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007e288d-00052129eb8ff417.journal                                               
PASS: /var/log/journal/d0f2a92c271c984c80e61c974d9908aa/system@974caf06715d4357945b637fd0e719ef-00000000007419c5-0005203ccf4a9520.journal                                               
core@core1 ~ $ cat /etc/os-release 
NAME=CoreOS
ID=coreos
VERSION=835.1.0
VERSION_ID=835.1.0
BUILD_ID=
PRETTY_NAME="CoreOS 835.1.0"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
core@core1 ~ $
@vcaputo

This comment has been minimized.

Show comment
Hide comment
@vcaputo

vcaputo Oct 24, 2015

Have identified the responsible bug in journald, and it still exists in v226 (and master) upstream. PR's are up for both coreos and upstream systemd.

coreos/systemd#23
systemd/systemd#1662

vcaputo commented Oct 24, 2015

Have identified the responsible bug in journald, and it still exists in v226 (and master) upstream. PR's are up for both coreos and upstream systemd.

coreos/systemd#23
systemd/systemd#1662

@crawford

This comment has been minimized.

Show comment
Hide comment
@crawford
Member

crawford commented Oct 24, 2015

@crawford crawford closed this Oct 24, 2015

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 24, 2015

Thank you very much!

ghost commented Oct 24, 2015

Thank you very much!

@vcaputo

This comment has been minimized.

Show comment
Hide comment
@vcaputo

vcaputo Oct 24, 2015

We should have an alpha up soon with this fixed (which will also include another important oom fix) #489

vcaputo commented Oct 24, 2015

We should have an alpha up soon with this fixed (which will also include another important oom fix) #489

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment