[Question] How to interpret boxes in object detection example ?

[geminigeek]

hi,

can anyone help me how to interpret boxes while using object detection with this model "Xenova/detr-resnet-50".
i want to crop out the detected object from the image using sharp (nodejs) ? how can i pass these boxes to sharp resize function ?

[xenova]

Hi there! 👋 I am updating the documentation to make it clearer how the output should be interpreted 👍 Also, while running some example code, I actually noticed that I am returning the post-processed values, but not in the same format as the python library's pipeline function. I'll fix that!

[xenova]

These updates are now in v2.4.1 🚀
Here are the updated docs: https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.ObjectDetectionPipeline

These new outputs are far easier to understand:

let img = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/cats.jpg';

let detector = await pipeline('object-detection', 'Xenova/detr-resnet-50');
let output = await detector(img, { threshold: 0.9 });
// [{
//   "score": 0.9976370930671692,
//   "label": "remote",
//   "box": { "xmin": 31, "ymin": 68, "xmax": 190, "ymax": 118 }
// },
// ...
// {
//   "score": 0.9984092116355896,
//   "label": "cat",
//   "box": { "xmin": 331, "ymin": 19, "xmax": 649, "ymax": 371 }
// }]

---

Looking at the sharp.js documentation (https://sharp.pixelplumbing.com/api-resize#extract), you can use xmin and ymin for the left and top values in the extract function, and calculate the width and height for each box by taking xmax-xmin and ymax-ymin, respectively.

Feel free to ask any further questions 👍