Persistent logs seem to disappear and reappear #1367
Assignees
Comments
4 tasks
|
I took a look at this. Its a Heisen-bug so hard to tell exactly what is going on. Did find one subtle thing to fix #1369 It is also possible the forceful reboots by the supervisor prevented the journal to flush to the disk: balena-os/balena-supervisor#861 That issue has been fixed in the supervisor now. But I think people are hitting the 8 megabyte limit. And the journal is rotating over itself as well. 8M is just enough to hold a few reboots. If there is a chatty service/application, 8M will disappear quickly. |
|
I don't think there is much here except what has been said above. |
|
@ZubairLK I suspect that there is something else here as well. Have you tried without a user application? We shouldn't have that many logs in the journal and a fair amount of boot logs should be easily handled. I think I've seen this issue in the past and it was somehow connected to machine id - the logs are stored per machine id and I was ending up with multiple ones. |
|
The scenario where this happens is if the initramfs fails to mount the state partition. And fails to bind mount /etc/machine-id from the state partition. Then systemd generates a new random machine-id that doesn't persist if the next time the initramfs succeeds in mounting the state partition. There is a 30 second timeout here. |
This was referenced Feb 4, 2019
This was referenced Mar 1, 2019
markcorbinuk
added a commit
that referenced
this issue
Sep 11, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file. A persistent r/w location (root-overlay/etc/fake-hwclock/) has been added to the resin-state partition for storage of this file. The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. The '-s' command line parameter has been dropped from chronyd as restoring time from the drift file is no longer necessary (and the restore from RTC functionality is already covered by the timeinit-rtc service). Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Sep 22, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file. A persistent r/w location (root-overlay/etc/fake-hwclock/) has been added to the resin-state partition for storage of this file. The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. The '-s' command line parameter has been dropped from chronyd as restoring time from the drift file is no longer necessary (and the restore from RTC functionality is already covered by the timeinit-rtc service). Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
4 tasks
markcorbinuk
added a commit
that referenced
this issue
Oct 9, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file. A persistent r/w location (root-overlay/etc/fake-hwclock/) has been added to the resin-state partition for storage of this file. The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. The '-s' command line parameter has been dropped from chronyd as restoring time from the drift file is no longer necessary (and the restore from RTC functionality is already covered by the timeinit-rtc service). Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 12, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 15, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 16, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 21, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 22, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
markcorbinuk
added a commit
that referenced
this issue
Oct 29, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
ghost
pushed a commit
that referenced
this issue
Oct 29, 2020
In order to produce sensible timestamps for journald log messages: a) the system time needs to be maintained correctly over a reboot, and b) the system time needs to be set before journald is started. Currently the system time is maintained over reboots on systems without an RTC using the last modified time of the chrony drift file. However there are a couple of issues with this approach: a) /var/lib/chrony/ is not mounted early enough in the boot process to be available for setting the time before journald is started. b) there is an issue with the current systemd dependencies that result in the last modified time of the drift file not being updated when the system is shutdown or rebooted (see #1995). The Debian fake-hwclock service (as used by Raspberry Pi OS) has been added to overcome these issues. The fake-hwclock service will save and restore the system time from the fake-hwclock.data file (in /etc/fake-hwclock/). The system time is loaded from this file at boot and saved to it on shutdown. An additional timer service has been added to update the file on an hourly basis to cater for unexpected shutdown scenarios, e.g. power failure. The timeinit-timestamp service has improved logging and has been renamed to timeinit-buildtime for clarity. A new time-set.target has been added as per upstream systemd and the fake-hwclock and timeinit-buildtime services have been added to it. Change-type: minor Connects-to: #1367 #1919 Signed-off-by: Mark Corbin <mark@balena.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Enable persistent logging on a board.
Check the output of
journalctl --list-bootsand then reboot.After one ore more reboots you will see less boot logs appear that reboots.
Sometimes they even reappear.
The text was updated successfully, but these errors were encountered: