Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mutex around writer access in recorder #491

Merged
merged 4 commits into from
Oct 21, 2020

Conversation

Stapelzeiger
Copy link
Contributor

The storage plugins are not designed to be thread safe. The recorder must therefore use a mutex when accessing the writer from different contexts. This happens for topic creation during discovery as well as in message callbacks from subscribed topics.

This fixes #486 and #480.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
@Stapelzeiger Stapelzeiger mentioned this pull request Aug 6, 2020
Copy link
Collaborator

@Karsten1987 Karsten1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for kicking this off. I am requesting changes here though for two reasons:

  • I am not sure if rosbag2_transport is the right location to protect for concurrency. While this would certainly help for the command line interface, I think the right place is within rosbag2_cpp.
  • Call it premature optimization (I have to measure my claim first), but I would expect this to lead to other performance issues as the actual call to write can be blocked if (creating a subscription takes too long or many subscriptions are being created in a row). I am not 100% certain yet what the best way is but I would like to see the call to write basically being wait free.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
@Stapelzeiger
Copy link
Contributor Author

@Karsten1987 I added locking in the rosbag2_cpp writer. I've reverted the changes in recorder, since they are redundant with the rosbag2_cpp ones so this should solve your second point regarding the create_subscription in the critical zone.

Do we need a similar thread-safe interface for the reader class? I have trouble imagining a use-case where you would read the from multiple threads with the current API (in particular the has_next(), read_next() wouldn't work nicely).

Copy link
Collaborator

@Karsten1987 Karsten1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good to me. @emersonknapp any thoughts on this?

@Stapelzeiger please sign your commits to make the DCO bot happy.

rosbag2_cpp/include/rosbag2_cpp/writer.hpp Outdated Show resolved Hide resolved
This reverts commit 714141c.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
@jarledo
Copy link

jarledo commented Oct 21, 2020

Any updates on this? Our system suffers from the bug described in #486, making it almost impossible to record our topics. Would be great to have this merged.

@Karsten1987
Copy link
Collaborator

Thanks for the reminder, kicking off CI on this:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • macOS Build Status
  • Windows Build Status

@Karsten1987 Karsten1987 merged commit 700817c into ros2:master Oct 21, 2020
@Karsten1987
Copy link
Collaborator

@Stapelzeiger thanks for the contribution

@Karsten1987
Copy link
Collaborator

@adamdbrw fyi

adamdbrw pushed a commit that referenced this pull request Nov 2, 2020
* Mutex around writer access in recorder

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* mutex in rosbag2_cpp writer for storage access

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* Revert "Mutex around writer access in recorder"

This reverts commit 714141c.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* alphabetical include order

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
adamdbrw pushed a commit that referenced this pull request Nov 2, 2020
* Mutex around writer access in recorder

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* mutex in rosbag2_cpp writer for storage access

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* Revert "Mutex around writer access in recorder"

This reverts commit 714141c.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* alphabetical include order

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
@almaslov
Copy link

almaslov commented Dec 2, 2020

Hello! It would be great to backport this fix to Foxy. Can I do it myself according to the Guide ?

almaslov pushed a commit to almaslov/rosbag2 that referenced this pull request Dec 2, 2020
* Mutex around writer access in recorder

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* mutex in rosbag2_cpp writer for storage access

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* Revert "Mutex around writer access in recorder"

This reverts commit 714141c.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* alphabetical include order

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>
@emersonknapp
Copy link
Collaborator

Yes, by all means. If the cherry-pick works without issue then it is an easy backport.

emersonknapp pushed a commit that referenced this pull request Dec 8, 2020
* Mutex around writer access in recorder

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* mutex in rosbag2_cpp writer for storage access

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* Revert "Mutex around writer access in recorder"

This reverts commit 714141c.

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

* alphabetical include order

Signed-off-by: Patrick Spieler <stapelzeiger@gmail.com>

Co-authored-by: Patrick Spieler <Stapelzeiger@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Database access race condition when recording
5 participants