image tiling for deployment #5623
Replies: 2 comments
|
You can keep the existing one-image request and perform tiling on the server. Put the batchable inference endpoint in a dependent service, then submit tile calls concurrently from the image endpoint. The batching boundary is the model service, so it can batch patches produced inside the server. I tested this with a real BentoML 1.4.39 HTTP server and its adaptive batch dispatcher. For one request producing eight tiles, sequential awaits produced eight single-item batches; allowing eight concurrent tile calls produced batches of 1 and 7 in each of three trials. All tile results retained their expected values and order. The model was a CPU stand-in, so this verifies batching behavior, not GPU throughput. The 1+7 grouping is an observation, not a guaranteed batch layout. This is the small runnable example I used; the integer chunks stand in for image patches: import asyncio
import time
import bentoml
@bentoml.service(workers=1, traffic={"concurrency": 64})
class TileModel:
@bentoml.api(batchable=True, max_batch_size=16, max_latency_ms=150)
def predict(self, tiles: list[list[int]]) -> list[dict[str, int]]:
time.sleep(0.01) # stand-in for inference
return [{"value": sum(t), "batch_size": len(tiles)} for t in tiles]
@bentoml.service(workers=1, traffic={"concurrency": 16})
class ImageAPI:
model = bentoml.depends(TileModel)
@bentoml.api
async def infer(self, image: list[int]) -> list[dict[str, int]]:
tiles = [image[i:i + 4] for i in range(0, len(image), 4)]
semaphore = asyncio.Semaphore(8)
async def one(tile):
async with semaphore:
result = await self.model.to_async.predict([tile])
return result[0]
return await asyncio.gather(*(one(tile) for tile in tiles))For actual large images, I would generate patches lazily and use a bounded worker queue rather than materialize all patches and tasks as this tiny example does. Keep each patch's coordinates for reconstruction, and give the model a consistent patch shape. Tune the model batch size against measured GPU memory and latency. The semaphore above is per request: with multiple image requests, the total pending work can be much larger. Bound image-request concurrency and the shared inference queue as well, so batching doesn't turn into unbounded RAM usage. Start with one model worker per GPU and measure before increasing it, since extra model workers can duplicate GPU allocations. This follows BentoML's adaptive batching model. Which BentoML version and patch shape are you using? |
|
Hi ekusiadadus, I’m using the latest BentoML release, and my tiles are 768 × 768 pixels. I’ll try the dependent service approach and measure the actual batch sizes, inference time, and memory usage with my model. Your point about limiting concurrency across multiple image requests is especially helpful. Thanks again! |
Uh oh!
There was an error while loading. Please reload this page.
Hello everyone
I am currently using BentoML for medical image inference. However, our images are extremely large, so we usually perform image tiling (patch extraction) on the server side before feeding the patches into the model. The downside is that we can no longer fully benefit from BentoML's Adaptive Batching capabilities.
We have also considered performing the tiling on the client side and then sending the patches directly to the server. However, our client-side system has already been finalized and is generally not recommended for modification.
Given this situation, are there any good approaches that can integrate seamlessly with BentoML while still enabling efficient inference and making the best possible use of batching and system resources?
All reactions