Use new event loop instead of the current loop for pipeline

[AllentDan]

For some jupyter notebook users, the loop can not be used directly.

[grimoire]

Why do you use thread instead of coroutine here?
asyncio.get_event_loop().create_task would give create a new coroutine task parallelized with the current coroutine. and asyncio.Queue is awaitable.

[AllentDan]

Why do you use thread instead of coroutine here? asyncio.get_event_loop().create_task would give create a new coroutine task parallelized with the current coroutine. and asyncio.Queue is awaitable.

Just want to yield items without async. Do you have demo codes?

[grimoire]

Thread -> asyncio.get_event_loop().create_task
Queue -> asyncio.queue

[grimoire]

https://github.com/grimoire/lmdeploy/blob/d75c757f1712ffb1a0958758e1e0d4b18743ca8d/lmdeploy/pytorch/engine/engine.py#L718

[AllentDan]

How can I return a generator for stream infer?

lmdeploy/lmdeploy/serve/async_engine.py

Lines 484 to 488 in 3d355b5

	try:
	out = outputs.get(timeout=0.001)
	if out is None:
	break
	yield out

[grimoire]

 try: 
	     out = await outputs.get() 
	     if out is None: 
	         break 
	     yield out 

[AllentDan]

 try: 
	     out = await outputs.get() 
	     if out is None: 
	         break 
	     yield out 

but this is not an async function.

[grimoire]

lmdeploy/lmdeploy/pytorch/engine/engine.py

Line 1103 in 3d355b5

def __call_async():

[AllentDan]

To avoid the performance drop, I used the original Thread and Queue.

[RunningLeon]

LGTM

[lvhan028]

@AllentDan what's the performance of llama-7b now?

[AllentDan]

@AllentDan what's the performance of llama-7b now?

It does not influence the api_server performance. As for the stream_infer function. I tested 1000 prompts and got 131.17s for the main branch and 165.23s for the PR.

[AllentDan]

@AllentDan what's the performance of llama-7b now?

It does not influence the api_server performance. As for the stream_infer function. I tested 1000 prompts and got 131.17s for the main branch and 165.23s for the PR.

Tested the batch_infer function, no performance drop.

[AllentDan]

@AllentDan what's the performance of llama-7b now?

After recovering the original implementation, performance remained the same.

[lvhan028]

@grimoire According to @AllentDan test result, coroutine slows down the performance.
Any idea about the reason?

[grimoire]

rollback