question: How to introduce non-blocking delays on nested methods within main loop

[fred-cardoso]

Hi @Rybec !

I apologize if I missed this on your documentation (and for asking this on a issue) but I've been running around in circles without finding any solution for this specific situation.
If we are nesting code in methods and calling those methods on the main loop of a pyRTOS task, how can we yield into the RTOS within those nested methods?

For eg:

def test():
    print('test 1')

    # Do some delay here within this method but without blocking the whole CPU
    yield [pyRTOS.timeout(5)] # This doesn't work obviously, but just for the example

    printd('test 2 after x seconds')

def task(self):
    # Pass control back to RTOS
    yield

    # Thread loop
    while True:
        test()

        yield [pyRTOS.timeout(0.5)]```

[Rybec]

Ok, so let me make sure I understand what you are trying to do, so I answer the right question:

I sounds like you want test() to run in a separate thread from your task, so that you can call test() within your task, it will print, then it will yield back to your task, but after 5 seconds, it will regain control from your task and print out a second line. If I am wrong, please clarify. If I am right, read on. (I'll write another post for the other thing I think you could be trying to do.)

This isn't how an RTOS works. Each task is a serial thread. It might be possible to achieve what you want using Python's asyncio module. (I don't have much experience with it, so I couldn't explain how to use it, but I know it can provide some thread-like capabilities without creating full threads.)

If you want multiple threads in an RTOS, you make multiple tasks. Note that it is possible to create a new task within an existing task, but make sure you do this carefully, becuse if you don't set your priorities right you can end up with a deadlock.

Now, again assuming I correctly understand what you are trying to do, what you would do to achieve this is to create a new task for test() instead of calling it, register that task (which should start it, if I recall correctly), and then yield. test() must be written as a task for this to work properly, so make sure you have the initial yield after any setup code. If you want test() to immediately print when you register/start it, put your first print statement before the initial yield. (And remember that the initial yield shouldn't return anything, so don't put a delay it in.) After the startup yield in test, then put your delaying yield and the second print after that. In your example, the priority you give test() shouldn't matter much, because it returns after the second print, and your delays will handle the timing, but if you have other tasks in addition to these, you will likely want to set the priority of test() to the same as the task creating it.

Now, note that this isn't really a good way of doing something like this. pyRTOS allows tasks to self terminate by returning (which many RTOS's don't do, for optimization reasons that don't apply to Python), so this is possible, but in general tasks shouldn't return. If you strictly need to be able to create multiple instances of a task that will self-terminate when they are done, go right ahead and use this mechanic, but note that this can create serious memory leaks. (Your example code above will very quickly run out of memory, especially on microcontrollers with limited memory, because you would be creating task instances faster than they terminate.) Generally if you want something to happen in another thread that is triggered by a main thread, you would create a task for that other thing, and then have that task block until another task signals it to activate, at which point it will do its thing and then block again until it is activated.

[fred-cardoso]

First of all let me thank you very much for the time invested on this, very detailed, response.

Actually I think what I want to achieve is more simple than that.
Let's simplify things and abstract a little bit.

If we need to insert a delay in a method to be called inside a pyRTOS task (for better code structure reasons only), how can we yield back into pyRTOS?
What I am really trying to achieve is to create these delays and I could write all the code inside the pyRTOS while loop but I think it would be very hard to read.
To clarify, I will not have any code inside the task running in paralel at all, so no threading. Just a nesting of some code.

def test():
    # this line will do something, but needs to wait 5 seconds before proceeding to the next one

    # pyRTOS timeout here
    time.sleep(5) # for eg.

    # another line, which completes the method

def task():
    yield

    while True:
        print('test')
        
        test()
        
        # now, here, we can call other methods belonging to this task
        
        yield [pyRTOS.timeout(1)]

In this example, I could place all the code inside the main loop, but this will grow bigger and I wanted to, if possible, encapsulate things on classes as well.
Once again, I apologize if this is not the best approach, but it's like something I did back in C on FreeRTOS.

[Rybec]

Ok, so the other thing you could be trying to do:

Perhaps my original guess was wrong, and you want the yield in test() to delay the whole task it is being called from. If this is correct, here's my response:

Nesting yields doesn't work like that in Python. yields are basically like returns, except they store the current context of the function so it can pick up where it left off. Just like you can't return from a function within a function it is calling, you can't yield from a function from within a function it is calling. In RTOS's written in C or assembly, it is possible to do things like this, but Python does not provide any mechanic that can be used this way.

That said, you can work around this to get what you want. When you call test(), store the return value. If that return value is a list of blocking conditions, yield on that list in your main task.

You should really look up how to use generators if you want to do this, but I'll give you a quick tutorial here.

When you call test(), it will return a generator object. (This is true of any function with a yield statement in it. pyRTOS tasks are all generator objects.) It doesn't actually run what is in the function until you use the generator. Normally generators are used in loops as iterators, but this doesn't seem to be how you want to use it. So in your case, you will call send() on the generator object. send() always takes one argument. Normally this argument is returned by the most recent yield, but the first time you call send() on a generator, the argument must be None. If you don't need to pass anything in, you can just use None every time. When a generator returns (rather than yielding), it throws a StopIteration exception. This is how something like a for loop knows it is done, but since you aren't using this way, you must catch this exception yourself, or it will crash your task and possible the whole program.

So here's what you would do to get this particular effect:

In your setup code for task(), do something like test_gen = test(). This will create your generator. Now, where you are calling test() in your while loop, instead you will do something like this:

try:
    yield test_gen.send(None)
except StopIteration:
    test_gen = test()
    yield test_gen.send(None)

If that looks ugly, it's because it is. This isn't how I would do it. You can fix this by making test() into an infinite loop. Just put all of the code in test() in a while True: loop, and now you don't need to catch the exception (because test() never returns), and you can just call yield test_gen.send(None) every task loop. (If you want a delay after the second print statement in test() as well, add a delaying yield after it as well.)

I hope one of these answers helped. If you have more questions, feel free to ask. I don't know your background with RTOS programming, so I apologize if this comes off as patronizing. I know in FreeRTOS and most RTOSs written in languages like C/assembly, you can yield from nested functions. This is possible because C doesn't have strongly enforced flow control, so it can jump out of anywhere in any function to an arbitrary place in code, which allows yielding to the main OS control loop from anywhere. Python does not have this ability, and honestly, using yield the way I am in pyRTOS is sort of a hacky use of Python generators. That said, you should still be able to do the same kind of things, with the right workarounds, and I don't mind helping figure out how to do that! (And maybe I'll eventually add a section to the documentation with how to do things like this.)

[Rybec]

Ok, I think I understand a bit better. You can take advantage of the StopIteration exception to achieve what you want here:

Let's assume test() can have multiple yields within it. There are two ways to do this.

First, you can use a for loop:

for i in test():
    yield i

This works because for loops automatically know how to use generators. The return value of the yield statements is put into i, so when you yield on i, that will pass your blocking condition list on to pyRTOS. And the for loop will catch the ending exception for you, so you don't have to worry about that. This is probably the most elegant way to handle this, but it only works if you only ever yield blocking conditions. (I believe just yield returns None, so yielding without any return value should also work as expected here.)

The other option is to handle this manually:

try:
    test_gen = test()
    while True:
        result = test_gen.send(None)
        yield result
except StopIteration:
    pass

The reason you might want to use this is if it sometimes yields things that you want to handle differently, in which case you could add an if statement that checks what was returned and only yields when blocking condition lists are returned.

I think this covers it. The for loop is probably what you want here, but I think this covers your options!

[Rybec]

On a side note: If I was using the for loop option, I would put a comment in my code explaining that it is just running a function to completion, using it as a generator to pass down blocking conditions, because otherwise it could make your code really confusing despite the elegance.

[Rybec]

Ok, so I realized that accessing return values will be problematic here, since Python generators' return values are generally ignored by Python. So I did some tests, and I have a solution. It's not great, but it works:

Given this generator:

def gen(return_list):
    yield 4
    yield 2
    return_list.append(10)

You can do this:

r = []
for i in gen(r):
    yield i

And now r will be [10]. Strictly speaking you can do this with any pass-by-reference type, so you could make an object specifically for holding return values, and pass that in to get populated with the return value. (This could also be used to output stuff on yields, without having to use the manual method...)