got the error "got Future <Future pending> attached to a different loop"

[billychen2014]

hello, what i did :

1. one class provide connect and run_command method ( let's call it A class)
2. call the A.run method to connect machine (let's call it machine B and C) and keep the connection (b_connection, c_connection)
3. call the A.run_command using 'b_connection' and 'c_connection' which kept in step 2
   tasks = [A.run_command using b_connection, A.run_command using c_connection, ] results = await asyncio.gather(*tasks) # i want to run the command at the same time ,hence i use asyncio

then i got the error as the title .

maybe i don't need to using gather again as the asyncssh.run already used?
thanks.

[ronf]

I'd need to see something closer to the actual code to help diagnose this. For instance, you don't list what code you are using to opening the connection, and whether or not you are calling await on that, which you need to do. If you want even the connect operation to happen in parallel, you could do something similar on those connections of using asyncio.gather(), but you'll need to either combine the connect call with a call to run() on the connection, or you'll need to different calls to asyncio.gather(), where the first one gets you a list of connections and then the second uses those connections to run commands.

Also, you should avoid creating multiple event loops here. There should be exactly one, ideally created as part of a call to asyncio.run() of a "main" async function.

[billychen2014]

ap.zip
thanks for the reply, i attached the example code , pls help to check it .

[ronf]

Sorry, this is a bit hard to follow since you seem to be driving everything through pytest. Do you have a version of it that shows what it would look like under regular execution, and does that have the same problem?

I'm particularly concerned with the way you are calling asyncio.get_event_loop() in your basic_env() setup, as that seems like it has the potential to create multiple event loops, at least in the case where there are multiple test functions being called, and I don't know how that will interact with an event loop set up by pytest when you use @pytest.mark.asyncio. I think it would be better to let pytest worry about getting the event loop set up and not requesting one yourself.

[billychen2014]

example.zip
hello, i modified the code ,take a look at it ,thanks a lot

[ronf]

You are still creating multiple event loops here, since you are calling asyncio.run() multiple times. You can't use any AsyncSSH objects created in the first call to asyncio.run() after that call returns, as the associated event loop stored inside the AsyncSSH objects will be closed at that point. Try changing from:

if __name__ == '__main__':
    driver_ssh = BCSSHDriver()
    asyncio.run(server_connection(driver_ssh))
    asyncio.run(run_command(driver_ssh))

to:

async def main():
    driver_ssh = BCSSHDriver()
    await server_connection(driver_ssh)
    await run_command(driver_ssh)

if __name__ == '__main__':
    asyncio.run(main())

The alternative would be to create an event loop and call run_until_complete() multiple times on it, first for server_connection() and then for run_command(). However, I think the above is cleaner.

[billychen2014]

thanks a lot .
it did works, hmm , it it looks like '@pytest.mark.asyncio' did something in my previous code, i will check it .

[ronf]

Glad to hear that it worked!

If pytest-asyncio is creating a new loop for each test case and you had separate test cases for the creation of the connections vs. the running of commands, you'd see the same problem you did here. To fix that, you'd need to combine these into a single test case.

It's ok to have multiple test cases, but only if they don't share any AsyncSSH objects between them. Each time you start a new event loop, you'd need to create a fresh set of AsyncSSH objects (or any other asyncio objects).

You might be able to control this with the scope option. See more info on this at https://pytest-asyncio.readthedocs.io/en/latest/reference/decorators/index.html. In particular, note the part about needing to increase the scope on the event_loop fixture if you increase it elsewhere.

[billychen2014]

to fix that, you'd need to combine these into a single test case ==> yes, it works . i want to control the steps one by one , that's why i tried to separate the cases. thanks again

[ronf]

Understood.

It's possible to do that using scope. See https://pytest-with-eric.com/pytest-advanced/pytest-fixture-scope/ for more background on that. If you create an event loop feature with a wider scope than a single function (such as class, module, or session), it should avoid the problem you saw. You're already using scope=="session" on your basic_env, but I think you need to use the fixture for the event loop instead of calling asyncio.get_event_loop() yourself. I must admit I haven't done much with pytest fixtures to know exactly what that should look like, but it looks like you can take out the loop from basic_env and set the scope on the event loop with something like the following in conftest.py:

@pytest.fixture(scope='module')
def event_loop():
    loop = asyncio.new_event_loop()
    yield loop
    loop.close()

After that, you should be able to add "event_loop" as an argument in the test case functions if you need to access it, but the primary thing is to change the scope to "session" to make sure the event loop isn't closed between tests.

[billychen2014]

i tried ,but failed, it looks like i don't understand the event_loop clearly, i have to learn it again.

in addition ,as you said, You can't use any AsyncSSH objects created in the first call to asyncio.run() after that call returns, as the associated event loop stored inside the AsyncSSH objects will be closed at that point , the closed event loop can't be used in step 2 (run command). i combined the two steps temp to resolve my problem :)

[ronf]

The intent of setting scope to "module" is that the test framework would not close the event loop and get a new one between tests. It would only do this when moving from testing one module to testing another. So, if all your tests were in the same module, they should all be able to share the same event loop, even if there are multiple tests.

That said, you don't get to choose the order that tests are run in, and the order that the tests are run shouldn't change the result. So, it's probably not a good idea to have one test create state that another test uses (like opening the connections in one test and using those connections in another). So, what you did to combine these into a single test is probably the right thing to do here.