Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NameOwnerChanged events can be lost when starting a service #12956

Closed
mattiasj-axis opened this issue Jul 4, 2019 · 0 comments · Fixed by #12957
Closed

NameOwnerChanged events can be lost when starting a service #12956

mattiasj-axis opened this issue Jul 4, 2019 · 0 comments · Fixed by #12957
Labels
pid1 regression ⚠️ A bug in something that used to work correctly and broke through some recent commit

Comments

@mattiasj-axis
Copy link
Contributor

systemd version the issue has been seen with

v239, v241

Used distribution

yocto w/ internal extra layers

Expected behaviour you didn't see

Service activated over dbus is properly determined to have started

Unexpected behaviour you saw

Timeout, followed by killing the service, despite the service having started properly

75152a4 changed the NameOwnerChanged event to be installed asynchronously.

Since v239 we have observed occasional failure of one of our services to start, almost exclusively in specific test scenarios which causes additional load on the system during bootup.

An apparently unrelated change caused this to be near 100% reproducible on one of our (least powerful) systems. After debugging we have observed that it takes ~5 seconds from calling add_match_async until the install_callback gets called, and between then the service has been started and the NameOwnerChanged event already fired off.

Reverting back to sd_bus_add_match causes all failures to disappear.
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Jul 4, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
f33b9cb 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed. Revert back
to the synchronous version to avoid this occurring.

Fixes: systemd#12956
@mattiasj-axis mattiasj-axis mentioned this issue Jul 4, 2019
Merged
@yuwata yuwata added pid1 regression ⚠️ A bug in something that used to work correctly and broke through some recent commit has-pr ✨ labels Jul 4, 2019
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Jul 11, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
021e148 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Catch the installed_callback and emulate a NameOwnerChanged to the
name currently on the bus when this happens.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Jul 11, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
1782e89 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Catch the installed_callback and emulate a NameOwnerChanged to the
name currently on the bus when this happens.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Jul 12, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
b729b70 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Jul 24, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
6fad5ec 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Aug 12, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
19582ee 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Aug 12, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
b4c63f5 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Aug 13, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
bba8384 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
mattiasj-axis added a commit to mattiasj-axis/systemd that referenced this issue Aug 13, 2019
@mattiasj-axis
core: Avoid race when starting dbus services
9f1d92c 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: systemd#12956
poettering pushed a commit that referenced this issue Aug 14, 2019
@mattiasj-axis @poettering
core: Avoid race when starting dbus services
a5a8776 
In high load scenarios it is possible for services to be started
before the NameOwnerChanged signal is properly installed.

Emulate a callback by also queuing a GetNameOwner when the match is
installed.

Fixes: #12956
@mdmillerii mdmillerii mentioned this issue Sep 11, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pid1 regression ⚠️ A bug in something that used to work correctly and broke through some recent commit
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

core: Avoid race when starting dbus services mattiasj-axis/systemd
2 participants
@yuwata @mattiasj-axis