New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Configuration Reload in journald #2236

Closed
smreichling opened this Issue Dec 28, 2015 · 10 comments

Comments

5 participants
@smreichling

smreichling commented Dec 28, 2015

So far as I am able to tell, making a change to any of journald's configuration files (/etc/systemd/journald.conf, for example) requires subsequently running systemctl restart systemd-journald in order for the changes to take effect. However, restarting the journald service appears to break logging for all other running services until those services are also restarted (journalctl -flu <service_name> shows no new log lines post journald restart).

I'd like to request support for reloading journald configuration without restarting the process and breaking logging for other systemd services. I see this feature as rather critical. Without it, it's necessary for me to take servers out of service just to change the logging configuration.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Dec 29, 2015

Member

However, restarting the journald service appears to break logging

@smreichling , which version of systemd are you using?

See 13790ad

journald: allow restarting journald without losing stream connections
Making use of the fd storage capability of the previous commit, allow
restarting journald by serilizing stream state to /run, and pushing open
fds to PID 1.

(Since v219)

Member

evverx commented Dec 29, 2015

However, restarting the journald service appears to break logging

@smreichling , which version of systemd are you using?

See 13790ad

journald: allow restarting journald without losing stream connections
Making use of the fd storage capability of the previous commit, allow
restarting journald by serilizing stream state to /run, and pushing open
fds to PID 1.

(Since v219)

@smreichling

This comment has been minimized.

Show comment
Hide comment
@smreichling

smreichling Dec 29, 2015

@evverx I had been on 208, but I just tried upgrading to 219 and I still have the same problem.

Here's what I did: I wrote a dummy python service called log-tester which just prints to stdout 5 times a second in an endless loop. To test the upgrade, I ran:

  1. journalctl -flu log-tester -- see a constant stream of output, all appears to be working.
  2. systemctl restart systemd-journald
  3. journalctl -flu log-tester -- stream of output has stopped.
  4. systemctl restart log-tester
  5. journalctl -flu log-tester -- stream of output has resumed.

I repeated these same steps several times with the same results each time. I confirmed the upgrade by running systemctl --version and journalctl --version, both show 219.

smreichling commented Dec 29, 2015

@evverx I had been on 208, but I just tried upgrading to 219 and I still have the same problem.

Here's what I did: I wrote a dummy python service called log-tester which just prints to stdout 5 times a second in an endless loop. To test the upgrade, I ran:

  1. journalctl -flu log-tester -- see a constant stream of output, all appears to be working.
  2. systemctl restart systemd-journald
  3. journalctl -flu log-tester -- stream of output has stopped.
  4. systemctl restart log-tester
  5. journalctl -flu log-tester -- stream of output has resumed.

I repeated these same steps several times with the same results each time. I confirmed the upgrade by running systemctl --version and journalctl --version, both show 219.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Dec 29, 2015

Member

Hm,

  • v225: I can reproduce it.
  • v226: I can reproduce it.
  • v227: I can't reproduce it.
  • v228: I can't reproduce it (maybe, this commit is a fix)

So, something went wrong between v219..v226 (my git bisect is too slow:))
Fixed: a5bd3c3

Try v227 or v228.

Member

evverx commented Dec 29, 2015

Hm,

  • v225: I can reproduce it.
  • v226: I can reproduce it.
  • v227: I can't reproduce it.
  • v228: I can't reproduce it (maybe, this commit is a fix)

So, something went wrong between v219..v226 (my git bisect is too slow:))
Fixed: a5bd3c3

Try v227 or v228.

@poettering

This comment has been minimized.

Show comment
Hide comment
@poettering

poettering Jan 12, 2016

Member

@smreichling 219 is too old. We only track issues with the two most current systemd versions upstream, and afaics you are describing an issue in much older systemd versions that got fixed since. Closing hence.

Member

poettering commented Jan 12, 2016

@smreichling 219 is too old. We only track issues with the two most current systemd versions upstream, and afaics you are describing an issue in much older systemd versions that got fixed since. Closing hence.

@poettering poettering closed this Jan 12, 2016

@poettering poettering added the journal label Jan 12, 2016

tblume pushed a commit to tblume/systemd-testsuite-suse that referenced this issue Mar 8, 2016

tblume added a commit to tblume/systemd-suse-devel that referenced this issue Apr 6, 2016

@lewicki-pk

This comment has been minimized.

Show comment
Hide comment
@lewicki-pk

lewicki-pk Mar 9, 2017

Hi,
I have just reproduced this issue on systemd 230 and then again on systemd 232.

I'm using systemd provided by yocto system so it has some extra patches applied. If you're familiar with Yocto project than you can read the recipe here http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd_232.bb?h=master and also check the patches here: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd

My setup info:
I have a system where my systemd-journal uses only volatile storage and I have rsyslog configured to store logs in the persistent directory.

I thought I can toggle ForwardToSyslog= parameter in journald.conf file in order to enable/disable persistent storage for logs.

Unfortunately this does not work and I need to restart all of the services whose logs I want to have in my syslog files.

Steps to reproduce the issue:

  1. Change the option ForwardToSyslog from yes to no.
  2. Restart the systemd-journal service (systemctl restart systemd-journald)
  3. Check if syslog files are no longer written to (in my case they are not appended).
  4. Change the option ForwardToSyslog back to no.
  5. Restart the systemd-journal service again (systemctl restart systemd-journald)
  6. Syslog files are still not appended.
  7. After restarting only one of running services you can observe that the logs from that service show up in syslog files and logs from other services do not.

Can you reopen this issue?

Extra question that would help me workaround my problem:
Is there an option to restart all running services in systemd?

lewicki-pk commented Mar 9, 2017

Hi,
I have just reproduced this issue on systemd 230 and then again on systemd 232.

I'm using systemd provided by yocto system so it has some extra patches applied. If you're familiar with Yocto project than you can read the recipe here http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd_232.bb?h=master and also check the patches here: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/systemd/systemd

My setup info:
I have a system where my systemd-journal uses only volatile storage and I have rsyslog configured to store logs in the persistent directory.

I thought I can toggle ForwardToSyslog= parameter in journald.conf file in order to enable/disable persistent storage for logs.

Unfortunately this does not work and I need to restart all of the services whose logs I want to have in my syslog files.

Steps to reproduce the issue:

  1. Change the option ForwardToSyslog from yes to no.
  2. Restart the systemd-journal service (systemctl restart systemd-journald)
  3. Check if syslog files are no longer written to (in my case they are not appended).
  4. Change the option ForwardToSyslog back to no.
  5. Restart the systemd-journal service again (systemctl restart systemd-journald)
  6. Syslog files are still not appended.
  7. After restarting only one of running services you can observe that the logs from that service show up in syslog files and logs from other services do not.

Can you reopen this issue?

Extra question that would help me workaround my problem:
Is there an option to restart all running services in systemd?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Mar 10, 2017

Member
  1. Change the option ForwardToSyslog back to no.

Do you mean "back to yes"?

  1. Syslog files are still not appended.

Does journalctl -fu some-service print anything?

Member

evverx commented Mar 10, 2017

  1. Change the option ForwardToSyslog back to no.

Do you mean "back to yes"?

  1. Syslog files are still not appended.

Does journalctl -fu some-service print anything?

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Mar 10, 2017

Member

Can you reopen this issue?

This issue is not about ForwardToSyslog. It's about journalctl -fu some-service. Please create a new issue.

Member

evverx commented Mar 10, 2017

Can you reopen this issue?

This issue is not about ForwardToSyslog. It's about journalctl -fu some-service. Please create a new issue.

@evverx

This comment has been minimized.

Show comment
Hide comment
@evverx

evverx Mar 10, 2017

Member

btw we can lose some logs due to FileDescriptorStoreMax=1024.

systemd-journald.service: Cannot store more fds than FileDescriptorStoreMax=1024, closing remaining.

systemctl restart systemd-journald causes write(1, "Woo\n", 4) = -1 EPIPE (Broken pipe) so I have to restart some services.

Member

evverx commented Mar 10, 2017

btw we can lose some logs due to FileDescriptorStoreMax=1024.

systemd-journald.service: Cannot store more fds than FileDescriptorStoreMax=1024, closing remaining.

systemctl restart systemd-journald causes write(1, "Woo\n", 4) = -1 EPIPE (Broken pipe) so I have to restart some services.

@keszybz

This comment has been minimized.

Show comment
Hide comment
@keszybz

keszybz Mar 11, 2017

Member

@lewicki-pk can you please open a new issue? This is most likely unrelated, because the transport for syslog files is different.

Is there an option to restart all running services in systemd?

No.

Member

keszybz commented Mar 11, 2017

@lewicki-pk can you please open a new issue? This is most likely unrelated, because the transport for syslog files is different.

Is there an option to restart all running services in systemd?

No.

@lewicki-pk

This comment has been minimized.

Show comment
Hide comment
@lewicki-pk

lewicki-pk Mar 14, 2017

Ok. I have opened a new issue #5586

Thanks

lewicki-pk commented Mar 14, 2017

Ok. I have opened a new issue #5586

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment