Python backend BLS unable to handle response from GPU model

[zeruniverse]

Description

I have model A and model B, both in GPU. I wrote a BLS like this:

def execute(data):
 response = request(A, data).exec() # this gives me a probability
 if response['output'].as_numpy()[1] < 0.3:  # this line will fail.
  # model A cannot handle this case well, let's use model B
  return request(B, data).exec()
 return response

The failure is from this line https://github.com/triton-inference-server/python_backend/blob/56727b5b78d5d04720eac718c5f3ef1b9d591a94/src/pb_tensor.cc#L501

I think AsNumpy should be implemented no matter if the tensor is in CPU or GPU, otherwise, many BLS with multiple models (and conditions handling in Python) will fail.

Triton Information
Official 22.02

To Reproduce

See description

Expected behavior

If tensor is in CPU, as_numpy should just return the numpy array. Otherwise, tensor should be copied from GRAM to RAM and a Python object would own this copied numpy array

[zeruniverse]

I understand dl_pack can do such conversion. But with pytorch, every python stub will take ~2G GPU memory even I just convert one array. This greatly limits how many BLS I can wrote.

[the-david-oy]

Thanks for submitting this request and providing additional details!

@Tabrizian Is there a better way to do this? Or would an enhancement be worth filing?

[Tabrizian]

Thanks for your feature request @zeruniverse. We do have a feature request in the backlog for converting GPU tensors to CPU without any third-party library.

But with pytorch, every python stub will take ~2G GPU memory even I just convert one array. This greatly limits how many BLS I can wrote.

You could use any other library that supports DLPack to use the GPU tensors. CuPy also supports DLPack and does not have the memory usage issue:

https://docs.cupy.dev/en/stable/reference/generated/cupy.fromDlpack.html

[zeruniverse]

Thanks for your feature request @zeruniverse. We do have a feature request in the backlog for converting GPU tensors to CPU without any third-party library.

But with pytorch, every python stub will take ~2G GPU memory even I just convert one array. This greatly limits how many BLS I can wrote.

You could use any other library that supports DLPack to use the GPU tensors. CuPy also supports DLPack and does not have the memory usage issue:

https://docs.cupy.dev/en/stable/reference/generated/cupy.fromDlpack.html

Thanks for your reply. I tried CuPy, unfortunately, it only takes less GRAM but won't give back GRAM either. My business logic can only process one image at a time but some of my model can take dynamic batch. Therefore, I have to set instance group count of this BLS to a pretty big value to make sure I fully utilized my model. Each python stub taking 279Mi is still too much.

[Tabrizian]

There are some APIs in CuPy to release the memory that might be useful for your scenario to reclaim the memory:

mempool.free_all_blocks()
pinned_mempool.free_all_blocks()

https://docs.cupy.dev/en/stable/user_guide/memory.html#memory-pool-operations

[zeruniverse]

Thanks!

…

On Tue, Apr 12, 2022 at 12:47 AM Iman Tabrizian ***@***.***> wrote: There are some APIs in CuPy to release the memory that might be useful for your scenario to reclaim the memory: mempool.free_all_blocks() pinned_mempool.free_all_blocks() https://docs.cupy.dev/en/stable/user_guide/memory.html#memory-pool-operations — Reply to this email directly, view it on GitHub <#3992 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDO6NFOL6RWHZWLARRZNODVERJSNANCNFSM5PR2JVDA> . You are receiving this because you were mentioned.Message ID: ***@***.***>