New feature optimize_for_inference is not in pip package

[sergiovillanueva]

I can see in the readme that there are a new feature model.optimize_for_inference(), but when I install from pip I can't see this feature, even if I install from github.

[nok]

Hello @sergiovillanueva , you're right, this feature hasn't been released yet. As a workaround, you can install the develop branch directly using:

pip install git+ssh://git@github.com/roboflow/rf-detr.git@develop

In addition there is a small bug in the example code. The method optimize_for_inference() doesn't return anything.

Current:

model = RFDETRBase()
model = model.optimize_for_inference()
# ...
detections = model.predict(image, threshold=0.5)

Fixed:

model = RFDETRBase()
model.optimize_for_inference()  # just call the method
# ...
detections = model.predict(image, threshold=0.5)

I created a small PR for that: #238

---

Finally I'm able to share my experience with this optimization: across all experiments, I measured a speedup ranging from 1.16x to 1.22x.

[isaacrob-roboflow]

hi @nok ! thanks for the catch, my mistake on the README 😅

what hardware are you using? are you happy with that speedup?

[ttw1203]

I did as @nok said but it actually made my inferences way slower, from ~4 FPS to ~6 sec per frame. What am I missing?

def _detect_rf_detr(self, frame: np.ndarray) -> sv.Detections:
     try:
         frame_rgb = frame[:, :, ::-1].copy()

         detections = self.model.predict(frame_rgb, threshold=self.confidence_threshold)

         return detections

[nok]

hi @nok ! thanks for the catch, my mistake on the README 😅

what hardware are you using? are you happy with that speedup?

I tested the base and large model on several environments:

• GPUs of my M2 (mps)
• 1080 & 3060 (in LAN) (cuda)
• L4 via GCE (in GCP) (cuda)

In all cases I measured a small speedup from 1.16x to 1.22x.

Hello @ttw1203 , did you try a profiling tool like pyinstrument? You can open a new issue related to your inspection and measurement.

[isaacrob-roboflow]

@ttw1203 can you say a bit more about your setup? Very surprised to hear about a slowdown

@nok when you say 1.2x, is that comparing to without the method call or to the version of the code prior to the PR being shipped? I also added some changes that generally speed things up, may have been part of the 2x total I observed on my GCP L4

[nok]

Hello @isaacrob-roboflow ,

@nok when you say 1.2x, is that comparing to without the method call or to the version of the code prior to the PR being shipped? I also added some changes that generally speed things up, may have been part of the 2x total I observed on my GCP L4

I measured the duration of 10,000 sequential inferences of the model, both with and without calling the optimize_for_inference method.

Here is the small script which I wrote for testing:

import io
import time
from typing import Union

import requests
from PIL import Image

from rfdetr import RFDETRBase, RFDETRLarge


def load_image():
    url = "https://media.roboflow.com/notebooks/examples/dog-2.jpeg"
    image = Image.open(io.BytesIO(requests.get(url).content))
    image = image.convert("RGB")
    image = image.resize((560, 560))
    return image


def load_model(optimized: bool = True, large: bool = False):
    if large:
        model = RFDETRLarge(resolution=560)
    else:
        model = RFDETRBase(resolution=560)
    if optimized:
        model.optimize_for_inference()
    return model


def run_measurement(model: Union[RFDETRBase, RFDETRLarge], image, n_times=10_000):
    model.predict(image)  # avoid coldstart
    start_time = time.perf_counter()
    for _ in range(n_times):
        _ = model.predict(image)
    elapsed_time = (time.perf_counter() - start_time) * 1000  # milliseconds
    print(f"Time for {n_times} predictions: {elapsed_time:.4f} ms")
    return elapsed_time


def main():
    image = load_image()

    print("Use base model (optimized=False):")
    run_measurement(load_model(optimized=False), image)
    print("Use base model (optimized=True):")
    run_measurement(load_model(optimized=True), image)

    print("Use large model (optimized=False):")
    run_measurement(load_model(optimized=False, large=True), image)
    print("Use large model (optimized=True):")
    run_measurement(load_model(optimized=True, large=True), image)


if __name__ == "__main__":
    main()

Ourput:

Use base model (optimized=False):                                                                                                                          │
Time for 10000 predictions: 264901.3552 ms
Use base model (optimized=True):
Time for 10000 predictions: 205689.4502 ms
# Result:  264901.3552 / 205689.4502 = 1,2878704034

Use large model (optimized=False):
Time for 10000 predictions: 672596.1314 ms
Use large model (optimized=True):
Time for 10000 predictions: 567167.7070 ms
# Result: 672596.1314 / 567167.7070 = 1,1858858026

The next days I will rerun this test with a L4.