sched/semaphore: fix priority boost restoration for priority inheritance #6318

pkarashchenko · 2022-05-23T22:07:58Z

Summary

Fix priority boost restoration in case of priority inheritance
Details in #6310

Impact

Improve priority inheritance. Reduce size of TCB in case is priority inheritance is enabled

Testing

Priority inheritance ostest pass on SAMe70-QMTECH and STM32F4Discovery boards

acassis · 2022-05-25T12:50:48Z

@pkarashchenko please include more details about the issue you are trying to fix

pkarashchenko · 2022-05-25T12:52:15Z

@pkarashchenko please include more details about the issue you are trying to fix

The issue is #6310
I will update the summary

hartmannathan

I see how the priority is boosted but I don't see how it is restored. It would be helpful to have an explanation of how this works...

sched/semaphore/sem_holder.c

sched/wqueue/kwork_inherit.c

hartmannathan · 2022-09-25T13:37:50Z

Sadly, no joy with tm4c129e-launchpad:ostest (+ CONFIG_PRIORITY_INHERITANCE=y added manually) either.

Although b-g474e-dpow1 is STM32 and tm4c129e-launchpad is Tiva, they are both armv7-m. However, they seem to be crashing in different parts of the test.

I am beginning to wonder if the problem is not in priority inheritance, but in something else that is manifested because of conditions created in the priority inheritance test.

Below is the log.

user_main: priority inheritance test
priority_inheritance: Started
priority_inheritance: Starting lowpri_thread-1 (of 3) at 1
priority_inheritance: Set lowpri_thread-1 priority to 1
priority_inheritance: Starting lowpri_thread-2 (of 3) at 1
lowpri_thread-1: Started
priority_inheritance: Set lowpri_thread-2 priority to 1
lowpri_thread-1: initial priority: 1
priority_inheritance: Starting lowpri_thread-3 (of 3) at 1
lowpri_thread-2: Started
priority_inheritance: Set lowpri_thread-3 priority to 1
lowpri_thread-1: Waiting for the midle pri task to run
priority_inheritance: Waiting...
lowpri_thread-2: initial priority: 1
lowpri_thread-3: Started
    g_middlestate:  0
lowpri_thread-2: Waiting for the midle pri task to run
lowpri_thread-3: initial priority: 1
    g_highstate[0]: 0
    g_middlestate:  0
lowpri_thread-3: Waiting for the midle pri task to run
    I still have a count on the semaphore
    g_highstate[0]: 0
    g_middlestate:  0
    I still have a count on the semaphore
    g_highstate[0]: 0
    I still have a count on the semaphore
lowpri_thread-1: Waiting for the midle pri task to run
    g_middlestate:  0
    g_highstate[0]: 0
    I still have a count on the semaphore
lowpri_thread-2: Waiting for the midle pri task to run
    g_middlestate:  0
    g_highstate[0]: 0
    I still have a count on the semaphore
lowpri_thread-3: Waiting for the midle pri task to run
    g_middlestate:  0
    g_highstate[0]: 0
    I still have a count on the semaphore
priority_inheritance: Starting medpri_thread at 99
priority_inheritance: Set medpri_thread priority to 99
priority_inheritance: Waiting...
medpri_thread: Started ... I won't let go of the CPU!
priority_inheritance: Starting highpri_thread-1 (of 1) at 255
priority_inheritance: Set highpri_thread-1 priority to 255
highpri_thread-1: Started
priority_inheritance: Waiting for highpri_thread-1 to complete
highpri_thread-1: Calling sem_wait()
lowpri_thread-3: Sem count: -1, No. highpri thread: 1
highpri_thread-1: SUCCESS midpri_thread is still running!
highpri_thread-1: Okay... I'm done!
priority_inheritance: Waiting for medpri_thread to complete
medpri_thread: Okay... I'm done!
priority_inheritance: Waiting for lowpri_thread-1 to complete
lowpri_thread-2: Sem count: 1, No. highpri thread: 0
lowpri_thread-2: SUCCESS the middle priority task has already exitted!
               g_middlestate:  3 sem count=1
               g_highstate[0]: 3
lowpri_thread-2: SUCCESS priority before sem_post: 1
lowpri_thread-1: Sem count: 2, No. highpri thread: 0
lowpri_thread-3: SUCCESS priority before sem_post: 255
lowpri_thread-2: SUCCESS final priority: 1
lowpri_thread-1: SUCCESS the middle priority task has already exitted!
lowpri_thread-3: SUCCESS final priority: 1
lowpri_thread-2: Okay... I'm done!
               g_middlestate:  3 sem count=2
lowpri_thread-3: Okay... I'm done!
               g_highstate[0]: 3
lowpri_thread-1: SUCCESS priority before sem_post: 1
lowpri_thread-1: SUCCESS final priority: 1
lowpri_thread-1: Okay... I'm done!
priority_inheritance: Waiting for lowpri_thread-2 to complete
priority_inheritance: Waiting for lowpri_thread-3 to complete
priority_inheritance: Restoraup_assert: Assertion failed at file:assert/lib_sta0
arm_registerdump: R0: 00000000 R1: 2000876c R2: 000466a4  R3: 2000876c
arm_registerdump: R4: 00000000 R5: 00000000 R6: 00000000  FP: 200114b0
arm_registerdump: R8: 00000000 SB: 00000000 SL: 00000000 R11: 00000000
arm_registerdump: IP: 00000000 SP: 20011478 LR: 000169bf  PC: 0001698c
arm_registerdump: xPSR: 61000000 PRIMASK: 00000000 CONTROL: 00000000
arm_registerdump: EXC_RETURN: fffffff9
arm_dump_stack: User Stack:
arm_dump_stack: sp:     200114b0
arm_dump_stack:   base: 20011260
arm_dump_stack:   size: 000003c0
arm_stackdump: 200114a0: 000100c2 00016419 00000001 00000000 200114c0 20012bb0 4
arm_stackdump: 200114c0: 00000000 0001650d 00000039 20012c30 00000039 000466a8 4
arm_stackdump: 200114e0: fffffff5 00011c37 00000039 000466a8 00000001 000466a4 3
arm_stackdump: 20011500: deadbeef 000466a4 00000000 0002a113 00000000 20011594 0
arm_stackdump: 20011520: 20010000 00000003 0004fc28 200115f4 00000002 fffffff5 b
arm_stackdump: 20011540: 00000000 00000000 200115f4 00000003 0000df00 00000000 4
arm_stackdump: 20011560: 00016868 000466a4 20010a9c 0000ab25 200115f4 00000003 0
arm_stackdump: 20011580: 00000000 000466a4 00000034 000016cb 20011e7c 00000003 4
arm_stackdump: 200115a0: 000000c8 000005cb 00000000 00000000 deadbeef 00000000 3
arm_stackdump: 200115c0: deadbeef 2001168c 00000002 000466a4 00000000 00001c5b 0
arm_stackdump: 200115e0: 2001168c 00000003 2001df00 00000000 40000000 20011640 0
arm_stackdump: 20011600: 00000000 00000000 00000000 00000000 00000000 00000000 9
arm_showtasks:    PID    PRI     STACK      USED   FILLED    COMMAND
arm_dump_task:      0      0      1000       880    88.0%!   Idle Task
arm_dump_task:      1    224      1992       492    24.6%    hpwork 0x200000a0
arm_dump_task:      2    100      1992       492    24.6%    lpwork 0x200000b4
arm_dump_task:      3    100      2000      1772    88.6%!   nsh_main
arm_dump_task:      4    100      1984       888    44.7%    Telnet daemon 0x200
arm_dump_task:      5     80      2024      1096    54.1%    netinit 0
arm_dump_task:      6    100      2008      1180    58.7%    ostest
arm_dump_task:      7    100      8112      1444    17.8%    ostest Arg1 Arg2 A4
up_assert: Assertion failed at file:armv7-m/arm_hardfault.c line: 173 task: Tas0
arm_registerdump: R0: dc006160 R1: ffffffff R2: 200113d8  R3: dc006160
arm_registerdump: R4: 00000000 R5: 00000000 R6: 00000000  FP: 200114b0
arm_registerdump: R8: 00000000 SB: 00000000 SL: 00000000 R11: 00000000
arm_registerdump: IP: 00000000 SP: 200112b8 LR: 00012c6b  PC: 000143a4
arm_registerdump: xPSR: a1000000 PRIMASK: 00000001 CONTROL: 00000000
arm_registerdump: EXC_RETURN: fffffff9
arm_dump_stack: User Stack:
arm_dump_stack: sp:     20011168
arm_dump_stack:   base: 20011260
arm_dump_stack:   size: 000003c0
arm_dump_stack: ERROR: User Stack pointer is not within the stack
arm_stackdump: 20011260: 0000000a 20000160 20012bb0 200112b8 00000001 00000000 0
arm_stackdump: 20011280: 200114b0 00000000 00000000 00000000 00000000 fffffff9 f
arm_stackdump: 200112a0: 200113d8 dc006160 00000000 00012c6b 000143a4 a1000000 0
arm_stackdump: 200112c0: dc006160 000466a4 01000000 00012c6b 00000038 20000000 8
arm_stackdump: 200112e0: 00046b43 00000000 00000000 200113ac 01730383 000466a4 d
arm_stackdump: 20011300: 00000000 00000000 dc006160 00000000 00000001 000466a4 0
arm_stackdump: 20011320: 00000010 20000160 20012bb0 000466a4 00000000 000466a4 1
arm_stackdump: 20011340: 200113d0 00046630 00000010 200113d0 00046b44 2001137c 4
arm_stackdump: 20011360: 00000000 000132b1 200113d4 200113a8 00046b44 200113d4 c
arm_stackdump: 20011380: 00000000 000466a4 00000058 0001228b 00001420 00046b40 f
arm_stackdump: 200113a0: 200113d4 200113ac 00000004 00013e35 00013da9 00013f61 f
arm_stackdump: 200113c0: 0000000c 000466a4 000003c0 000161df 00046b40 dc006160 0
arm_stackdump: 200113e0: 000005a4 00000011 00000008 00000020 2000e640 2001141c 4
arm_stackdump: 20011400: 00000000 20011a40 000003e8 20011e2c 00000033 000003c0 0
arm_stackdump: 20011420: 0466a420 46189b20 9a034b05 429a681b f010d001 b005fab2 0
arm_stackdump: 20011440: 02092020 012bb020 2061ac20 200466a4 00000000 00000000 4
arm_stackdump: 20011460: 00046b7c 00000000 00000000 00000000 200114b0 000291af 5
arm_stackdump: 20011480: 00001490 00040000 00000008 000466a4 00046cac 000162a5 4
arm_stackdump: 200114a0: 000100c2 0001641d 00000001 00000000 200114c0 20012bb0 4
arm_stackdump: 200114c0: 00000000 0001650d 00000039 20012c30 00000039 000466a8 4
arm_stackdump: 200114e0: fffffff5 00011c37 00000039 000466a8 00000001 000466a4 3
arm_stackdump: 20011500: deadbeef 000466a4 00000000 0002a113 00000000 20011594 0
arm_stackdump: 20011520: 20010000 00000003 0004fc28 200115f4 00000002 fffffff5 b
arm_stackdump: 20011540: 00000000 00000000 200115f4 00000003 0000df00 00000000 4
arm_stackdump: 20011560: 00016868 000466a4 20010a9c 0000ab25 200115f4 00000003 0
arm_stackdump: 20011580: 00000000 000466a4 00000034 000016cb 20011e7c 00000003 4
arm_stackdump: 200115a0: 000000c8 000005cb 00000000 00000000 deadbeef 00000000 3
arm_stackdump: 200115c0: deadbeef 2001168c 00000002 000466a4 00000000 00001c5b 0
arm_stackdump: 200115e0: 2001168c 00000003 2001df00 00000000 40000000 20011640 0
arm_stackdump: 20011600: 00000000 00000000 00000000 00000000 00000000 00000000 9
arm_showtasks:    PID    PRI     STACK      USED   FILLED    COMMAND
arm_dump_task:      0      0      1000       880    88.0%!   Idle Task
arm_dump_task:      1    224      1992       492    24.6%    hpwork 0x200000a0
arm_dump_task:      2    100      1992       492    24.6%    lpwork 0x200000b4
arm_dump_task:      3    100      2000      1772    88.6%!   nsh_main
arm_dump_task:      4    100      1984       888    44.7%    Telnet daemon 0x200
arm_dump_task:      5     80      2024      1096    54.1%    netinit 0
arm_dump_task:      6    100      2008      1180    58.7%    ostest
arm_dump_task:      7    100      8112      1444    17.8%    ostest Arg1 Arg2 A4

hartmannathan · 2022-09-25T13:39:36Z

When you test with stm32f4, if the test passes, it will be a good idea to compare defconfig and see if there are any differences that give a clue...

sched/semaphore/sem_holder.c

sched/wqueue/kwork_inherit.c

pkarashchenko · 2022-09-29T18:28:04Z

@masayuki2009 @davids5 maybe you have some boards and can run ostest with priority inheritance enabled?

acassis · 2022-09-29T18:45:10Z

Crashing here too: https://pastebin.com/raw/VCGzC6kN

Signed-off-by: Petro Karashchenko <petro.karashchenko@gmail.com>

pkarashchenko · 2022-09-30T14:57:54Z

@hartmannathan I just successfully ran ostest on stm32f4discovery board. Here is a defconfig that I used. Also run log is attached.
defconfig.txt
stm32f4discovery.log

pkarashchenko · 2022-09-30T16:07:39Z

Finally I was managed to catch the crash with next config. I was able to identify that if CONFIG_DEBUG_FULLOPT=y the crash is not reproduced. @acassis @hartmannathan could you please check on your end?
defconfig_stm32disco.txt

pkarashchenko · 2022-09-30T21:57:55Z

@hartmannathan @acassis here is a fix of the crash apache/nuttx-apps#1333
Actually log from Alan helped as it was containing string assert/lib_sta that lead me to libs/libc/assert/lib_stackchk.c and __stack_chk_fail

hartmannathan · 2022-10-02T00:43:57Z

@hartmannathan @acassis here is a fix of the crash apache/incubator-nuttx-apps#1333
Actually log from Alan helped as it was containing string assert/lib_sta that lead me to libs/libc/assert/lib_stackchk.c and __stack_chk_fail

Kudos for catching that! I am currently traveling and expect to be back at the lab on Thursday or Friday. Unfortunately that means I will not be able to test it because I don't have real hardware, and only occasional connectivity. I don't know whether you'd like to wait that long to merge it; if you and @acassis feel comfortable with it now, then feel free to merge it and I'll test it anyway when I get back just to make sure...

pkarashchenko · 2022-10-02T07:51:22Z

The PR is here for quite a long time, so waiting one more week should not be a problem unless other reviewers want to speedup merging of a solution.

masayuki2009 · 2022-10-02T08:02:19Z

@masayuki2009 @davids5 maybe you have some boards and can run ostest with priority inheritance enabled?

The latest PR now works with qemu-intel64:ostest

acassis · 2022-10-03T18:14:34Z

I also confirm it worked fine! Merging!

acassis · 2022-10-03T18:38:34Z

Log test ESP32 here: https://pastebin.com/raw/hskcVcTP

hartmannathan · 2022-10-07T15:41:02Z

@pkarashchenko Finally got back and was able to test. Happy to report the test passes on tm4c129e-launchpad:ostest with priority inheritance enabled!

hartmannathan · 2022-10-07T15:41:42Z

Thank you for all your work on this!

since CONFIG_SEM_NNESTPRIO is removed by: apache/nuttx#6318 Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>

pkarashchenko mentioned this pull request May 23, 2022

Priority Inheritance fails to restore base priority #6310

Closed

pkarashchenko force-pushed the fix_priority_inherit branch 7 times, most recently from fa71756 to 739356c Compare May 24, 2022 08:29

ThomasDebrunner mentioned this pull request May 30, 2022

sched/semaphore: Fix priority restoration #6341

Closed

pkarashchenko requested a review from hartmannathan May 30, 2022 10:49

hartmannathan reviewed May 30, 2022

View reviewed changes

hartmannathan requested review from xiaoxiang781216 and davids5 May 30, 2022 11:04

pkarashchenko commented May 30, 2022

View reviewed changes

sched/semaphore/sem_holder.c Outdated Show resolved Hide resolved

pkarashchenko force-pushed the fix_priority_inherit branch 2 times, most recently from 2a98a68 to 8494203 Compare June 16, 2022 20:41

pkarashchenko mentioned this pull request Jun 16, 2022

[DRAFT] Backport semaphore fixes PX4/NuttX#188

Draft

pkarashchenko force-pushed the fix_priority_inherit branch from 8494203 to d8a2cd7 Compare June 16, 2022 21:11

pkarashchenko force-pushed the fix_priority_inherit branch 6 times, most recently from afe8c96 to c5c0773 Compare June 28, 2022 16:05

pkarashchenko commented Jun 30, 2022

View reviewed changes

sched/semaphore/sem_holder.c Outdated Show resolved Hide resolved

pkarashchenko commented Jun 30, 2022

View reviewed changes

sched/wqueue/kwork_inherit.c Show resolved Hide resolved

pkarashchenko commented Jun 30, 2022

View reviewed changes

sched/wqueue/kwork_inherit.c Outdated Show resolved Hide resolved

pkarashchenko force-pushed the fix_priority_inherit branch from c5c0773 to ba8d32d Compare September 16, 2022 13:59

xiaoxiang781216 reviewed Sep 29, 2022

View reviewed changes

sched/semaphore/sem_holder.c Outdated Show resolved Hide resolved

sched/wqueue/kwork_inherit.c Outdated Show resolved Hide resolved

pkarashchenko force-pushed the fix_priority_inherit branch 3 times, most recently from d5d7208 to 36694c3 Compare September 29, 2022 18:25

sched/semaphore: fix priority boost restoration for priority inheritance

b64bd00

Signed-off-by: Petro Karashchenko <petro.karashchenko@gmail.com>

pkarashchenko force-pushed the fix_priority_inherit branch from 36694c3 to b64bd00 Compare September 30, 2022 14:54

acassis merged commit d247e8d into apache:master Oct 3, 2022

hartmannathan mentioned this pull request Oct 7, 2022

boards/tm4c129e-launchpad: Enable priority inheritance in ostest config #7253

Merged

xiaoxiang781216 mentioned this pull request Oct 21, 2022

testing/ostest: Remove the code which reference CONFIG_SEM_NNESTPRIO apache/nuttx-apps#1370

Merged

pkarashchenko pushed a commit to apache/nuttx-apps that referenced this pull request Oct 21, 2022

testing/ostest: Remove the code which reference CONFIG_SEM_NNESTPRIO

6f01a3b

since CONFIG_SEM_NNESTPRIO is removed by: apache/nuttx#6318 Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>

xiaoxiang781216 mentioned this pull request Oct 22, 2022

Problems with priority inheritance. #7393

Closed

anchao pushed a commit to anchao/nuttx-apps that referenced this pull request Feb 13, 2023

testing/ostest: Remove the code which reference CONFIG_SEM_NNESTPRIO

fc75d39

since CONFIG_SEM_NNESTPRIO is removed by: apache/nuttx#6318 Signed-off-by: Xiang Xiao <xiaoxiang@xiaomi.com>

pkarashchenko deleted the fix_priority_inherit branch August 20, 2023 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sched/semaphore: fix priority boost restoration for priority inheritance #6318

sched/semaphore: fix priority boost restoration for priority inheritance #6318

pkarashchenko commented May 23, 2022 •

edited

Loading

acassis commented May 25, 2022

pkarashchenko commented May 25, 2022

hartmannathan left a comment

hartmannathan commented Sep 25, 2022

hartmannathan commented Sep 25, 2022

pkarashchenko commented Sep 29, 2022

acassis commented Sep 29, 2022

pkarashchenko commented Sep 30, 2022

pkarashchenko commented Sep 30, 2022

pkarashchenko commented Sep 30, 2022

hartmannathan commented Oct 2, 2022

pkarashchenko commented Oct 2, 2022

masayuki2009 commented Oct 2, 2022

acassis commented Oct 3, 2022

acassis commented Oct 3, 2022

hartmannathan commented Oct 7, 2022

hartmannathan commented Oct 7, 2022

sched/semaphore: fix priority boost restoration for priority inheritance #6318

sched/semaphore: fix priority boost restoration for priority inheritance #6318

Conversation

pkarashchenko commented May 23, 2022 • edited Loading

Summary

Impact

Testing

acassis commented May 25, 2022

pkarashchenko commented May 25, 2022

hartmannathan left a comment

Choose a reason for hiding this comment

hartmannathan commented Sep 25, 2022

hartmannathan commented Sep 25, 2022

pkarashchenko commented Sep 29, 2022

acassis commented Sep 29, 2022

pkarashchenko commented Sep 30, 2022

pkarashchenko commented Sep 30, 2022

pkarashchenko commented Sep 30, 2022

hartmannathan commented Oct 2, 2022

pkarashchenko commented Oct 2, 2022

masayuki2009 commented Oct 2, 2022

acassis commented Oct 3, 2022

acassis commented Oct 3, 2022

hartmannathan commented Oct 7, 2022

hartmannathan commented Oct 7, 2022

pkarashchenko commented May 23, 2022 •

edited

Loading