-
Notifications
You must be signed in to change notification settings - Fork 197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[discussion] retrying a spawn
/ schedule
is almost always wrong
#209
Comments
This is a nasty one. The thing is that in event driven systems you should (probably) never retry like this, but schedule an event to try later - not sure how to communicate that though. |
Hi Folks. In the original RTFM-C implementation, sending a message was done using the keyword "async", with optional "after", setting the base-line offset, and "before" setting a new deadline. The return value was a message pointer, allowing to abort outstanding (but not yet scheduled messages). One could think of doing the same in the Rust re-implementation. Possibly with an option, In the original RTFM-C implementation, each task definition came with Worst Case (shortest) Inter-arrival Time. Using this information, the maximum number of outstanding messages can be bounded (and hence sufficient capacity computed). That's why we did not have an "option" return type (since we computed the sufficient capacity). We could go both ways in the Rust re-implementation. Either the "hard" way, towards giving guarantees (with more info/analysis), or the soft/sloppy way. The latter approach is what you typically see in systems like FreeRTOS, which is best effort scheduling.. We could also chose to fork, having a sloppy version, and a hard one. What do you think out there? By the way: Sloppy is not meaning its bad, its just best effort. Best regards |
@korken89 yeah, we should definitively document this and backport the docs into the v0.4.x branch. I don't think we need to decide on this for the v0.5.0 release. We can add a new API and deprecate the existing one in a new patch release (v0.5.x). I just tested that
This sounds like a new feature.
Isn't this equivalent to spawn and ignore the result API? If you rely on user input then the user could enter an arbitrarily large time that produces a capacity of 1 and then proceed to spawn N tasks causing (N-1) messages to be silently lost. Or they could incorrectly enter a value (e.g. forgot a zero or extra zero) that causes the capacity to be miscomputed also leading to missed messages. |
SGTM! |
Ok then I'm going to assign the discussion about a potential API change to the v0.5.x milestone (things to discuss after the v0.5.0 release) |
205: rtfm-syntax refactor + heterogeneous multi-core support r=japaric a=japaric this PR implements RFCs #178, #198, #199, #200, #201, #203 (only the refactor part), #204, #207, #211 and #212. most cfail tests have been removed because the test suite of `rtfm-syntax` already tests what was being tested here. The `rtfm-syntax` crate also has tests for the analysis pass which we didn't have here -- that test suite contains a regression test for #183. the remaining cfail tests have been upgraded into UI test so we can more thoroughly check / test the error message presented to the end user. the cpass tests have been converted into plain examples EDIT: I forgot, there are some examples of the multi-core API for the LPC541xx in [this repository](https://github.com/japaric/lpcxpresso54114) people that would like to try out this API but have no hardware can try out the x86_64 [Linux port] which also has multi-core support. [Linux port]: https://github.com/japaric/linux-rtfm closes #178 #198 #199 #200 #201 #203 #204 #207 #211 #212 closes #163 cc #209 (documents how to deal with errors) Co-authored-by: Jorge Aparicio <jorge@japaric.io>
205: rtfm-syntax refactor + heterogeneous multi-core support r=japaric a=japaric this PR implements RFCs #178, #198, #199, #200, #201, #203 (only the refactor part), #204, #207, #211 and #212. most cfail tests have been removed because the test suite of `rtfm-syntax` already tests what was being tested here. The `rtfm-syntax` crate also has tests for the analysis pass which we didn't have here -- that test suite contains a regression test for #183. the remaining cfail tests have been upgraded into UI test so we can more thoroughly check / test the error message presented to the end user. the cpass tests have been converted into plain examples EDIT: I forgot, there are some examples of the multi-core API for the LPC541xx in [this repository](https://github.com/japaric/lpcxpresso54114) people that would like to try out this API but have no hardware can try out the x86_64 [Linux port] which also has multi-core support. [Linux port]: https://github.com/japaric/linux-rtfm closes #178 #198 #199 #200 #201 #203 #204 #207 #211 #212 closes #163 cc #209 (documents how to deal with errors) Co-authored-by: Jorge Aparicio <jorge@japaric.io>
I think it is time to ping this issue again when we restart our meetings after new-year. |
209: Bump cortex-m-rt-macro version to 0.1.6 for release r=korken89 a=therealprof Closes #206 Signed-off-by: Daniel Egger <daniel@eggers-club.de> Co-authored-by: Daniel Egger <daniel@eggers-club.de>
We close this issue for now, not sure if it is relevant in the case of the single core focus for RTIC 0.6. Feel free to re-open the issue if interested in further elaborating this topic. |
Blocking and non-blocking
Abstractions that provide both non-blocking and blocking versions of an
operation usually name their APIs
try_{action}
(non-blocking) and{action}
(blocking). One example of such convention is the
crossbeam-channel
crate: itsSender
API provides a non-blockingtry_send
method and a blockingsend
method. The blocking method works by (smartly) retrying the non-blocking one.
spawn
andschedule
Due to the fixed capacity nature of our message queues the
spawn
andschedule
APIs are non-blocking and fallible -- the operation fails when thequeue is full. Furthermore, retrying either non-blocking operation to make it
blocking is wrong in most cases. Consider this scenario:
Here a higher priority task sends a message to a lower priority one. Because the
operation may not succeed the user retries the operation until it succeeds.
However, due to how task scheduling works this retry operation may block
forever:
b
cannot run whilea
is executing so ifb
's queue is full whena
starts it will remain like that for the whole execution of taska
. Thissituation also arises if both
a
andb
have the same priority.If in the previous example we reverse the priorities then the retry operation is
not needed at all because
a
can't never fillb
's queue --b
will preempta
as soon as the latter spawns the first.However, if
lock
s are involved then even with these prioritiesa
cansaturate
b
's queue.The only scenario where it is sensible to retry a
spawn
call is when thetarget task runs on another core. The reason for this is that the other core can
drain the message queue regardless of the task priorities involved -- the task
priorities used on one core have no effect on the task scheduling performed on
other cores.
Timer queues are core-local -- the same core drains its own queue -- so
schedule
calls, both core-local and cross-core, can never be successfully retried.Improving the situation
We want to prevent users from running into those retry deadlocks but the current
API certainly lets them run into this pitfall.
Some ideas to counter the problem:
Keep the API as it is but document the pitfall
What the subtitle says.
Change the API to make retrying impossible
AFAIK there are two reasonable operations one can perform on the
Result
returned by
spawn
/schedule
:you
drop
the value / ignore the result; this indicates that either you don'tcare if the operation fails or that you expect the operation to never fail so
you never bother checking the output.
you
unwrap
the result to catch (during development) programmer errors wherea capacity that was too small was selected.
If these are indeed the only two reasonable options then we can change and
extend the API to provide only these two options:
spawn.$task
(schedule.$task
) would have theunwrap
semantics: send themessage or panic if the queue is full
try_spawn.$task
(try_schedule.$task
) would have thedrop
semantics: sendthe message or drop the payload if the queue is full
Both
spawn.$task
(schedule.$task
) andtry_spawn.$task
(try_schedule.$task
)return the unit type:
()
.spawn.$task
is shorter to type and that's the API one should start out with.try_spawn.$task
should be used only where one is sure the operation will notfail or where dropping / missing a message is not important.
With these new APIs it would not be possible to do a blocking cross-core
spawn
call, but that may be a good thing because blocking behaviordoesn't feel like it really belongs within the RTFM API.
The names
spawn
andtry_spawn
appear match the naming convention used forblocking and non-blocking variants but don't so we may want to pick different names
if we go down this route.
Some other option?
Is there some approach we could take here?
The text was updated successfully, but these errors were encountered: