You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
This may be more widespread than simply Software Bus, but Software Bus is where I've seen it the most.
The Software Bus function CFE_SB_BroadcastBufferToRoute locks global data via CFE_SB_LockSharedData, then may call OS_QueuePut, then eventually unlocks the global data via CFE_SB_UnlockSharedData. In OSAL, the OS_QueuePut call finds its way into mq_timedsend, and the locking and unlocking of global data is via pthread_mutex_lock and pthread_mutex_unlock.
During shutdown, the various apps are all cancelled. Again in OSAL, this resolves to pthread_cancel calls. This leads to apps terminating when they reach "cancellation points."
mq_timed_send is a cancellation point. pthread_mutex_lock is not a cancellation point.
When a task is in the process of sending a message and is cancelled, it may be terminated while it holds the SB Shared Data mutex. Occasionally this coincides with another task that is pending or is about to pend on that mutex. Any remaining tasks that pend on that mutex are then deadlocked. I have not yet identified why the abort at the end of CFE_PSP_Restart is not called, but the system hangs indefinitely.
My current work-around is to modify OSAL's os-impl-mutex.c. In OS_MutSemCreate_Impl I use pthread_mutexattr_setrobust to make all mutexes robust just before the call to pthread_mutex_init and in OS_MutSemTake_Impl I check the return code for EOWNERDEAD and, if that is returned, call pthread_mutex_consistent to restore the mutex.
To Reproduce
Steps to reproduce the behavior:
This may be difficult to reproduce without a lot of Software Bus traffic. With enough software bus traffic, simply restarting the system should be sufficient to trigger this eventually. Unfortunately this is a race condition that is not easily triggered.
Expected behavior
On shutdown, all tasks terminate.
Code snips
cFE/modules/sb/fsw/src/cfe_sb_api.c line 1548 is the call to CFE_SB_LockSharedData that could trigger the deadlock
cFE/modules/sb/fsw/src/cfe_sb_api.c line 1605 (Call to OS_QueuePut) is a cancellation point where that same function has the global data mutex locked.
osal/src/os/posix/src/os-impl-queues.c line 305 is the actual call to mq_timed_send
osal/src/os/posix/src/os-impl-mutex.c line 179 is the pthread_mutex_lock that ultimately blocks and deadlocks a subsequent task.
System observed on:
Ubuntu 22.04. Analysis indicates any POSIX system would be vulnerable. I have not evaluated vulnerability for other platforms.
Additional context
This may require coordination with the OSAL project. This ticket may be more appropriate for the OSAL team.
Reporter Info
Lorn Miller
Red Canyon Engineering & Software
The text was updated successfully, but these errors were encountered:
Describe the bug
This may be more widespread than simply Software Bus, but Software Bus is where I've seen it the most.
The Software Bus function CFE_SB_BroadcastBufferToRoute locks global data via CFE_SB_LockSharedData, then may call OS_QueuePut, then eventually unlocks the global data via CFE_SB_UnlockSharedData. In OSAL, the OS_QueuePut call finds its way into mq_timedsend, and the locking and unlocking of global data is via pthread_mutex_lock and pthread_mutex_unlock.
During shutdown, the various apps are all cancelled. Again in OSAL, this resolves to pthread_cancel calls. This leads to apps terminating when they reach "cancellation points."
mq_timed_send is a cancellation point. pthread_mutex_lock is not a cancellation point.
When a task is in the process of sending a message and is cancelled, it may be terminated while it holds the SB Shared Data mutex. Occasionally this coincides with another task that is pending or is about to pend on that mutex. Any remaining tasks that pend on that mutex are then deadlocked. I have not yet identified why the abort at the end of CFE_PSP_Restart is not called, but the system hangs indefinitely.
My current work-around is to modify OSAL's os-impl-mutex.c. In OS_MutSemCreate_Impl I use pthread_mutexattr_setrobust to make all mutexes robust just before the call to pthread_mutex_init and in OS_MutSemTake_Impl I check the return code for EOWNERDEAD and, if that is returned, call pthread_mutex_consistent to restore the mutex.
To Reproduce
Steps to reproduce the behavior:
This may be difficult to reproduce without a lot of Software Bus traffic. With enough software bus traffic, simply restarting the system should be sufficient to trigger this eventually. Unfortunately this is a race condition that is not easily triggered.
Expected behavior
On shutdown, all tasks terminate.
Code snips
cFE/modules/sb/fsw/src/cfe_sb_api.c line 1548 is the call to CFE_SB_LockSharedData that could trigger the deadlock
cFE/modules/sb/fsw/src/cfe_sb_api.c line 1605 (Call to OS_QueuePut) is a cancellation point where that same function has the global data mutex locked.
osal/src/os/posix/src/os-impl-queues.c line 305 is the actual call to mq_timed_send
osal/src/os/posix/src/os-impl-mutex.c line 179 is the pthread_mutex_lock that ultimately blocks and deadlocks a subsequent task.
System observed on:
Ubuntu 22.04. Analysis indicates any POSIX system would be vulnerable. I have not evaluated vulnerability for other platforms.
Additional context
This may require coordination with the OSAL project. This ticket may be more appropriate for the OSAL team.
Reporter Info
Lorn Miller
Red Canyon Engineering & Software
The text was updated successfully, but these errors were encountered: