Unification of Closed-Open Industrial detection scenarios: New Large-Scale Benchmarks, Challenges and Baselines [IEEE TPAMI].

The extened version of abstract

Large-scale Visual-Language models (LVLMs) have achieved remarkable success in visual tasks. However, the significant differences between industrial and natural scenes make the application of LVLMs challenging. On the one hand, the scarcity of large-scale industrial defect data makes the application of LVLMs in open industrial scenarios unexplored. On the other hand, existing LVLMs rely on user manual work and single textual or visual prompts to segment objects. This usually leads to poor performance due to the inclusion of irrelevant semantics. To fill this gap, this paper proposes the first large-scale industrial multi-modal dataset. First, this paper constructs a multi-modal industrial unified open-closed dataset (MMIOC-1M) with 1M+ samples. MMIOC-1M supports open and closed scene tasks and contains rich industry categories, including 17 super categories, 31 scenes, and 351 subcategories. MMIOC-1M is the first large-scale multi-modal pre-training dataset for industrial representation learning, providing valuable training data for large models in future industrial scenarios. Based on MMIOC-1M, this paper provides RTVPNet specifically for industrial open-closed detection and visual question-answering tasks. RTVPNet contains two significant advantages: First, we design an expert-guided domain projection mechanism for large models and an industrial open detection method based on Mobile-SAM, which enhances the generalization of large models in industrial scenarios. Second, RTVPNet automatically generates refined visual prompts directly from images and considers the text-visual prompt interactions that were ignored by previous LVLM, improving visual and textual content understanding. Parts of this paper were originally published in its AAAI-2025 conference version. This paper extends earlier work in terms of dataset, network architecture and analysis, etc.

In MMIOC, some bounding boxes (mainly involving wood and fan blades) may be offset due to our cropping, which can be used as negative samples. These samples are masked during testing. These situations will not affect the accuracy of RTVPNet in testing.

MMIOC-1M data has been upload to the BaiduDisk. To use the data, please contact 2673679261@qq.com and sign the open source agreement to obtain it.

The link of MMIOC-1M and RTVPNet weights are as follows:

MMIOC-Closed dataset link: https://pan.baidu.com/s/1hW7wk6D8yJHxRxPgKcfzpg?pwd=88f5

MMIOC-Open dataset link: https://pan.baidu.com/s/1CfBehOiRNVD6I8GkAje2JQ?pwd=01j5

RTVPNet weights link: https://pan.baidu.com/s/1t5JOE5-T5OPUWqwauZX0TQ?pwd=dn62

To the best of our knowledge, this paper constructs the first multi-modal object detection dataset (MMIOC-1M) for industrial open and closed scenes. MMIOC-1M is extended based on MMIO-80K and contains more than 1M samples and 31 industrial scenes, effectively alleviating the lack of domain expertise in industrial open scenes. The comparison with mainstream defect data is as follows: The dataset is visualized as follows: The statistics of the dataset categories are as follows: This paper proposes a refined learnable text-visual prompt to improve the detection ability of visual language models in industrial open and closed scenes. Compared with the previous version of RTVP, RTVPNet designs a new text-visual bidirectional prompt interaction and an energy-based refined visual prompt method. RTVPNet automatically provides specific text-visual prompts for each image, which reduces the noise and effectively improves the knowledge and understanding ability of LVLMs in industrial domains.

MMIOC-1M dataset construct

Name_of_Dataset
|-- Images
|-----|----- train
|-----|--------|------ defect images
|-----|--------|------ ...
|-----|--------|------ ...
|-----|----- validation
|-----|--------|------ defect images
|-----|--------|------ ...
|-----|--------|------ ...
|-- labels
|-----|----- train
|--------|--------|------txt
|-----|--------|--------|------ defect.txt
|-----|--------|--------|------ ...
|-----|--------|--------|------ ...
|--------|--------|------annotations
|-----|--------|--------|------ train.json
|-----|----- validation
|-----|--------|--------|------ defect.txt
|-----|--------|--------|------ ...
|-----|--------|--------|------ ...
|--------|--------|------annotations
|-----|--------|--------|------ val.json

Code

The code has been open-sourced, and users can choose between the basic version and the more powerful version RTVPNet.

Requirements

GPU: 8xNVIDIA A100-SXM4-40GB CPU: Intel(R) Xeon(R) Platinum 8473C Running Memory: 512GB+ PyTorch 2.1.0 Python 3.9

Command: pip install -r requirements.txt

Run Pre-train on MMIOC-1M Open-set

./tools/dist_train.sh configs/pretrain/RTVPNet_S_MMIOC_Open_8xA100gpus_200e_pretrain.py --device cuda:0,1,2,3,4,5,6,7 --amp

Run Pre-train on MMIOC-1M Closed-set

./tools/dist_train.sh configs/pretrain/RTVPNet_S_MMIOC_Closed_8xA100gpus_200e_pretrain.py --device cuda:0,1,2,3,4,5,6,7 --amp

Run val

./tools/dist_test.sh configs/pretrain/RTVPNet_S_MMIOC_Open_8xA100gpus_200e_pretrain.py --device cuda:0,1,2