Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pid1: stopping service freezes when ExecStop= runs "systemctl reload" on dependecy #7324

Open
johnlinp opened this issue Nov 14, 2017 · 6 comments

Comments

@johnlinp
Copy link
Contributor

commented Nov 14, 2017

Submission type

  • Bug report

systemd version the issue has been seen with

v235

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

Successfully stop a service.

In case of bug report: Unexpected behaviour you saw

systemctl freezes when stopping the service.

In case of bug report: Steps to reproduce the problem

  1. Create 2 service units, base.service and app.service:
# /usr/lib/systemd/system/base.service
[Service]
ExecStart=/bin/sleep 5566
ExecReload=/bin/true
# /usr/lib/systemd/system/app.service
[Unit]
Requires=base.service
After=base.service

[Service]
Type=oneshot
ExecStart=/bin/systemctl reload base.service
ExecStop=/bin/systemctl reload base.service
RemainAfterExit=yes
  1. systemctl start app successfully starts base and app.

  2. systemctl stop app freezes.

  3. While the freeze, systemctl status base app shows:

● base.service
   Loaded: loaded (/usr/lib/systemd/system/base.service; static; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-14 16:27:13 CST; 14s ago
  Process: 10082 ExecReload=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 10080 (sleep)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/base.service
           └─10080 /bin/sleep 5566

Nov 14 16:27:13 jacky systemd[1]: Started base.service.
Nov 14 16:27:13 jacky systemd[1]: Reloading base.service.
Nov 14 16:27:13 jacky systemd[1]: Reloaded base.service.

● app.service
   Loaded: loaded (/usr/lib/systemd/system/app.service; static; vendor preset: disabled)
   Active: deactivating (stop) since Tue 2017-11-14 16:27:21 CST; 7s ago
  Process: 10081 ExecStart=/bin/systemctl reload base.service (code=exited, status=0/SUCCESS)
 Main PID: 10081 (code=exited, status=0/SUCCESS); Control PID: 10094 (systemctl)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/app.service
           └─control
             └─10094 /bin/systemctl reload base.service

Nov 14 16:27:13 jacky systemd[1]: Starting app.service...
Nov 14 16:27:13 jacky systemd[1]: Started app.service.
Nov 14 16:27:21 jacky systemd[1]: Stopping app.service...
  1. After timeout, systemctl stop app returns.

  2. Then systemctl status base app shows:

● base.service
   Loaded: loaded (/usr/lib/systemd/system/base.service; static; vendor preset: disabled)
   Active: active (running) since Tue 2017-11-14 16:27:13 CST; 2min 4s ago
  Process: 10112 ExecReload=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 10080 (sleep)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/base.service
           └─10080 /bin/sleep 5566

Nov 14 16:27:13 jacky systemd[1]: Started base.service.
Nov 14 16:27:13 jacky systemd[1]: Reloading base.service.
Nov 14 16:27:13 jacky systemd[1]: Reloaded base.service.
Nov 14 16:28:51 jacky systemd[1]: Reloading base.service.
Nov 14 16:28:51 jacky systemd[1]: Reloaded base.service.

● app.service
   Loaded: loaded (/usr/lib/systemd/system/app.service; static; vendor preset: disabled)
   Active: failed (Result: timeout) since Tue 2017-11-14 16:28:51 CST; 26s ago
  Process: 10094 ExecStop=/bin/systemctl reload base.service (code=killed, signal=TERM)
  Process: 10081 ExecStart=/bin/systemctl reload base.service (code=exited, status=0/SUCCESS)
 Main PID: 10081 (code=exited, status=0/SUCCESS)

Nov 14 16:27:13 jacky systemd[1]: Starting app.service...
Nov 14 16:27:13 jacky systemd[1]: Started app.service.
Nov 14 16:27:21 jacky systemd[1]: Stopping app.service...
Nov 14 16:28:51 jacky systemd[1]: app.service: Stopping timed out. Terminating.
Nov 14 16:28:51 jacky systemd[1]: app.service: Control process exited, code=killed status=15
Nov 14 16:28:51 jacky systemd[1]: app.service: Failed with result 'timeout'.
Nov 14 16:28:51 jacky systemd[1]: Stopped app.service.

Notes

I would like to point out that if I remove After=base.service in app.service, systemctl stop app will successfully stop app.service. However, I don't think After=base.service should cause the freeze because reloading base.service has nothing to do with ordering dependencies, I suppose?

@boucman

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2017

just for debugging purpose, does adding --no-block to systemctl reload help ?

@johnlinp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 14, 2017

@boucman Yes, adding --no-block solves the problem!
However, I am still interested why this happens without --no-block.

@boucman

This comment has been minimized.

Copy link
Contributor

commented Nov 14, 2017

I don't really know. I think there is a sort of interlocking in systemd which makes systemctl never receive the message that the reload is done, and that it can return.

--no-block makes systemctl return immediately instead of waiting for the action to complete, but it will take a real systemd dev to explain exactly what's going on here...

@johnlinp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 15, 2017

@boucman Thank you for your comments. Let's wait for a systemd developer to answer.

@poettering poettering added the pid1 label Dec 6, 2017

@bl33pbl0p

This comment has been minimized.

Copy link
Contributor

commented Jan 15, 2019

It is very clear why. You have a stop job in queue, but trigger a reload job for it. You have After=bar.service, hence the reload job waits on your stop job. Once your stop job is complete (however goes to fail due to timeout that was hit), it then schedules the reload job, which will complete.

Ordering of JOB_RELOAD is treated like JOB_START.

@bl33pbl0p

This comment has been minimized.

Copy link
Contributor

commented Jan 16, 2019

Nothing to fix here. You ask for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
You can’t perform that action at this time.