Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd-journald does not resume automatically after failed rotation (no free disk space) #3968

Closed
1 of 2 tasks
hese10 opened this issue Aug 16, 2016 · 2 comments · Fixed by #3973
Closed
1 of 2 tasks
Labels

Comments

@hese10
Copy link
Contributor

hese10 commented Aug 16, 2016

Submission type

  • Bug report
  • Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

231

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Fedora 24 + systemd 231

In case of bug report: Expected behaviour you didn't see

When root partion is full, journal log rotation fails, and no new logs are seen in "journal -f" output. When disk space is freed, journal logging shuold resume without user restartingservices.

In case of bug report: Unexpected behaviour you saw

journal logging did not resume when disk space was freed. Only by giving service restart it resumed:
systemctl restart systemd-journald

In case of bug report: Steps to reproduce the problem

  1. delete journal logs from /var/log/journal//
    rm /var/log/journal/*/*
  2. restart systemd-journald service
    systemctl restart systemd-journald
  3. check the free space with df -h -BM /
# df -h -BM
Filesystem     1M-blocks  Used Available Use% Mounted on
devtmpfs           3986M    0M     3986M   0% /dev
/dev/vda1          9540M 2822M     6248M  32% /
  1. fill the root file system with dd command so that we get error "no space left on device"
    # dd if=/dev/zero of=/var/log/journal/fillfile.bin bs=1M count=6244

  2. start logging continuously to journal:
    while [ 1 == 1 ]; do logger -p 3 "testlogging to fill journal current file" ; sleep 0.01; done;

  3. wait that journal rotation happens, journal -f stops taking new logs, and dmesg output shows error:
    4990.275879] systemd-journald[21722]: Failed to create new system journal: No space left on device

  4. delete the fillfile:
    rm /var/log/journal/fillfile.bin

  5. check the "journal -f" command again, no new logs are seen in journal output

poettering pushed a commit that referenced this issue Aug 17, 2016
If journals get into a closed state like when rotate fails due to
ENOSPC, when space is made available it currently goes unnoticed leaving
the journals in a closed state indefinitely.

By calling system_journal_open() on entry to find_journal() we ensure
the journal has been opened/created if possible.

Also moved system_journal_open() up to after open_journal(), before
find_journal().

Fixes #3968
@hese10
Copy link
Contributor Author

hese10 commented Aug 24, 2016

@poettering , can this issue be re-opened, as the original correction did not correct totally the situation?

@davestechshop
Copy link

@poettering I am seeing this issue on Arch Linux with latest updates.

jsynacek pushed a commit to jsynacek/systemd-rhel that referenced this issue Oct 10, 2017
If journals get into a closed state like when rotate fails due to
ENOSPC, when space is made available it currently goes unnoticed leaving
the journals in a closed state indefinitely.

By calling system_journal_open() on entry to find_journal() we ensure
the journal has been opened/created if possible.

Also moved system_journal_open() up to after open_journal(), before
find_journal().

Fixes systemd/systemd#3968

(cherry picked from commit 105bdb4)

Resolves: #1493846
jsynacek pushed a commit to jsynacek/systemd-rhel that referenced this issue Oct 19, 2017
If journals get into a closed state like when rotate fails due to
ENOSPC, when space is made available it currently goes unnoticed leaving
the journals in a closed state indefinitely.

By calling system_journal_open() on entry to find_journal() we ensure
the journal has been opened/created if possible.

Also moved system_journal_open() up to after open_journal(), before
find_journal().

Fixes systemd/systemd#3968

(cherry picked from commit 105bdb4)

Resolves: #1493846
lnykryn pushed a commit to lnykryn/systemd-rhel that referenced this issue Oct 30, 2017
If journals get into a closed state like when rotate fails due to
ENOSPC, when space is made available it currently goes unnoticed leaving
the journals in a closed state indefinitely.

By calling system_journal_open() on entry to find_journal() we ensure
the journal has been opened/created if possible.

Also moved system_journal_open() up to after open_journal(), before
find_journal().

Fixes systemd/systemd#3968

(cherry picked from commit 105bdb4)

Resolves: #1493846
caobinxin pushed a commit to caobinxin/systemd-lx that referenced this issue Oct 19, 2019
If journals get into a closed state like when rotate fails due to
ENOSPC, when space is made available it currently goes unnoticed leaving
the journals in a closed state indefinitely.

By calling system_journal_open() on entry to find_journal() we ensure
the journal has been opened/created if possible.

Also moved system_journal_open() up to after open_journal(), before
find_journal().

Fixes systemd/systemd#3968

(cherry picked from commit 105bdb46b4ac7eb658a2f27727216591d0bfe267)

Resolves: #1493846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

3 participants