How to run the multiple trt engines in a single GPU?

[tinnapopd]

I created a project for detection and classification on image frames.

I implemented detection and classification engines using the following constructor:

def __init__(
        self,
        verbose: Optional[bool] = False,
        workspace: Optional[int] = 8,
    ) -> None:

        self.trt_logger = trt.Logger(trt.Logger.INFO)
        if verbose:
            self.trt_logger.min_severity = trt.Logger.Severity.VERBOSE

        trt.init_libnvinfer_plugins(self.trt_logger, namespace="")

        self.builder = trt.Builder(self.trt_logger)
        self.config = self.builder.create_builder_config()
        self.config.max_workspace_size = workspace * (2**30)

        self.network = None
        self.parser = None
        self.batch_size = None

I designed my pipeline to run inference on detection first, then pass all detected boxes into classification. I deployed this as an API. However, when I scaled the service by replicating it, the Requests Per Second (RPS) remained the same.

For example, when I replicated the service 5 times, GPU usage increased 5x, but the RPS did not improve.

Do you have any ideas on how to double the RPS?

ps. I run this API using docker compose and use nginx as lb

[lix19937]

Maybe you can ref https://github.com/lix19937/tensorrt-insight/tree/main/device-benchmark-mps

[LeoZDong]

How are you enqueueing the requests to the engine? Are they done asynchronously? It seems a bit strange for a single GPU to own multiple engines if it is the same network. You instead try using one engine and enqueue each async request as an async cuda stream. See this related question here: #3967