Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] oclip readme #1505

Merged
merged 6 commits into from
Nov 3, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@ Please see [Quick Run](https://mmocr.readthedocs.io/en/dev-1.x/get_started/quick

Supported algorithms:

<details open>
<summary>BackBone</summary>

- [x] [oCLIP](configs/backbone/oclip/README.md) (ECCV'2022)

</details>

<details open>
<summary>Text Detection</summary>

Expand Down
7 changes: 7 additions & 0 deletions README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,13 @@ pip3 install -e .

支持的算法:

<details open>
<summary>BackBone</summary>

- [x] [oCLIP](configs/backbone/oclip/README.md) (ECCV'2022)

</details>

<details open>
<summary>文字检测</summary>

Expand Down
31 changes: 31 additions & 0 deletions configs/backbone/oclip/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# oCLIP

> [Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting](https://arxiv.org/abs/2203.03911)

<!-- [ALGORITHM] -->

## Abstract

Recently, Vision-Language Pre-training (VLP) techniques have greatly benefited various vision-language tasks by jointly learning visual and textual representations, which intuitively helps in Optical Character Recognition (OCR) tasks due to the rich visual and textual information in scene text images. However, these methods cannot well cope with OCR tasks because of the difficulty in both instance-level text encoding and image-text pair acquisition (i.e. images and captured texts in them). This paper presents a weakly supervised pre-training method, oCLIP, which can acquire effective scene text representations by jointly learning and aligning visual and textual information. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features, respectively, as well as a visual-textual decoder that models the interaction among textual and visual features for learning effective scene text representations. With the learning of textual features, the pre-trained model can attend texts in images well with character awareness. Besides, these designs enable the learning from weakly annotated texts (i.e. partial texts in images without text bounding boxes) which mitigates the data annotation constraint greatly. Experiments over the weakly annotated images in ICDAR2019-LSVT show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks, respectively. In addition, the proposed method outperforms existing pre-training techniques consistently across multiple public datasets (e.g., +3.2% and +1.3% for Total-Text and CTW1500).

<div align=center>
<img src="https://user-images.githubusercontent.com/24622904/199475057-aa688422-518d-4d7a-86fc-1be0cc1b5dc6.png"/>
</div>

## Support Model
Harold-lkk marked this conversation as resolved.
Show resolved Hide resolved

| | DBNet | DBNet++ | FCENet | TextSnake | PSENet | DRRG | Mask R-CNN |
| :-------: | :---: | :-----: | :----: | :-------: | :----: | :--: | :--------: |
| ICDAR2015 | ✓ | ✓ | ✓ | | ✓ | | ✓ |
| CTW1500 | | | | ✓ | ✓ | ✓ | ✓ |
Harold-lkk marked this conversation as resolved.
Show resolved Hide resolved

## Citation

```bibtex
@article{xue2022language,
title={Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting},
author={Xue, Chuhui and Zhang, Wenqing and Hao, Yu and Lu, Shijian and Torr, Philip and Bai, Song},
journal={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2022}
}
```
10 changes: 10 additions & 0 deletions configs/backbone/oclip/metafile.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Collections:
- Name: oCLIP
Metadata:
Training Data: SynthText
Architecture:
- CLIPResNet
Paper:
URL: https://arxiv.org/abs/2203.03911
Title: 'Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting'
README: configs/backbone/oclip/README.md
Harold-lkk marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions configs/textdet/dbnet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ Recently, segmentation-based methods are quite popular in scene text detection,
| :--------------------------------------: | :-------------------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :-----------------------------------------: |
| [DBNet_r18](/configs/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015.py) | ImageNet | ICDAR2015 Train | ICDAR2015 Test | 1200 | 736 | 0.8853 | 0.7583 | 0.8169 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/20220825_221614.log) |
| [DBNet_r50dcn](/configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py) | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test | 1200 | 1024 | 0.8784 | 0.8315 | 0.8543 | [model](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/20220828_124917.log) |
| [DBNet_oclip-r50](/configs/textdet/dbnet/dbnet_oclip-resnet50_fpnc_1200e_icdar2015.py) | [Synthtext](https://download.openmmlab.com/mmocr/textdet/dbnet/tmp_1.0_pretrain/dbnet_r50dcnv2_fpnc_sbn_2e_synthtext_20210325-ed322016.pth) | ICDAR2015 Train | ICDAR2015 Test | 1200 | | | | | [model](<>) \| [log](<>) |

## Citation

Expand Down
12 changes: 12 additions & 0 deletions configs/textdet/dbnet/metafile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,15 @@ Models:
Metrics:
hmean-iou: 0.8543
Weights: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015_20220828_124917-452c443c.pth

- Name: dbnet_oclip-resnet50_fpnc_1200e_icdar2015
In Collection: DBNet
Config: configs/textdet/dbnet/dbnet_resnet50-dcnv2_fpnc_1200e_icdar2015.py
Metadata:
Training Data: ICDAR2015
Results:
- Task: Text Detection
Dataset: ICDAR2015
Metrics:
hmean-iou:
Weights:
1 change: 1 addition & 0 deletions configs/textdet/dbnetpp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Recently, segmentation-based scene text detection methods have drawn extensive a
| Method | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :--------------------------------------: | :-------------------------------------------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :-----------------------------------------: |
| [DBNetpp_r50dcn](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py) | [Synthtext](/configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_100k_synthtext.py) ([model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth)) | ICDAR2015 Train | ICDAR2015 Test | 1200 | 1024 | 0.9116 | 0.8291 | 0.8684 | [model](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/20220829_230108.log) |
| [DBNetpp_oclip-r50](/configs/textdet/dbnetpp/dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015.py) | - | ICDAR2015 Train | ICDAR2015 Test | 1200 | | | | | [model](<>) \| [log](<>) |

## Citation

Expand Down
12 changes: 12 additions & 0 deletions configs/textdet/dbnetpp/metafile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,15 @@ Models:
Metrics:
hmean-iou: 0.8684
Weights: https://download.openmmlab.com/mmocr/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015_20220829_230108-f289bd20.pth

- Name: dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015
In Collection: DBNetpp
Config: configs/textdet/dbnetpp/dbnetpp_oclip-resnet50_fpnc_1200e_icdar2015.py
Metadata:
Training Data: ICDAR2015
Results:
- Task: Text Detection
Dataset: ICDAR2015
Metrics:
hmean-iou:
Weights:
1 change: 1 addition & 0 deletions configs/textdet/drrg/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Arbitrary shape text detection is a challenging task due to the high variety and
| Method | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :----------------------------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------------: |
| [DRRG](/configs/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500.py) | ImageNet | CTW1500 Train | CTW1500 Test | 1200 | 640 | 0.8775 | 0.8179 | 0.8467 | [model](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth) \\ [log](https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/20220827_105233.log) |
| [DRRG_oclip-r50](/configs/textdet/drrg/drrg_oclip-resnet50_fpn-unet_1200e_ctw1500.py) | - | CTW1500 Train | CTW1500 Test | 1200 | | | | | [model](<>) \\ [log](<>) |

## Citation

Expand Down
12 changes: 12 additions & 0 deletions configs/textdet/drrg/metafile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,15 @@ Models:
Metrics:
hmean-iou: 0.8467
Weights: https://download.openmmlab.com/mmocr/textdet/drrg/drrg_resnet50_fpn-unet_1200e_ctw1500/drrg_resnet50_fpn-unet_1200e_ctw1500_20220827_105233-d5c702dd.pth

- Name: drrg_oclip-resnet50_fpn-unet_1200e_ctw1500
In Collection: DRRG
Config: configs/textdet/drrg/drrg_oclip-resnet50_fpn-unet_1200e_ctw1500.py
Metadata:
Training Data: CTW1500
Results:
- Task: Text Detection
Dataset: CTW1500
Metrics:
hmean-iou:
Weights:
2 changes: 2 additions & 0 deletions configs/textdet/fcenet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,14 @@ One of the main challenges for arbitrary-shaped text detection is to design a go
| Method | Backbone | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :-------------------------------------------------: | :--------------: | :--------------: | :-----------: | :----------: | :-----: | :---------: | :-------: | :----: | :----: | :---------------------------------------------------: |
| [FCENet](/configs/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500.py) | ResNet50 + DCNv2 | ImageNet | CTW1500 Train | CTW1500 Test | 1500 | (736, 1080) | 0.8689 | 0.8296 | 0.8488 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/20220825_221510.log) |
| [FCENet_oclip-r50](/configs/textdet/fcenet/fcenet_oclip-resnet50-dcnv2_fpn_1500e_ctw1500.py) | CLIP-ResNet50 | - | CTW1500 Train | CTW1500 Test | 1500 | | | | | [model](<>) \| [log](<>) |

### ICDAR2015

| Method | Backbone | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :------------------------------------------------------: | :------: | :--------------: | :----------: | :-------: | :-----: | :----------: | :-------: | :----: | :----: | :---------------------------------------------------------: |
| [FCENet](/configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py) | ResNet50 | ImageNet | IC15 Train | IC15 Test | 1500 | (2260, 2260) | 0.8243 | 0.8834 | 0.8528 | [model](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/20220826_140941.log) |
| [FCENet_oclip-r50](/configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_icdar2015.py) | ResNet50 | - | IC15 Train | IC15 Test | 1500 | (2260, 2260) | - | - | - | [model](<>) \| [log](<>) |

## Citation

Expand Down
25 changes: 25 additions & 0 deletions configs/textdet/fcenet/metafile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,19 @@ Models:
Metrics:
hmean-iou: 0.8488
Weights: https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500/fcenet_resnet50-dcnv2_fpn_1500e_ctw1500_20220825_221510-4d705392.pth

- Name: fcenet_oclip-resnet50_fpn_1500e_ctw1500
In Collection: FCENet
Config: configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_ctw1500.py
Metadata:
Training Data: CTW1500
Results:
- Task: Text Detection
Dataset: CTW1500
Metrics:
hmean-iou:
Weights:

- Name: fcenet_resnet50_fpn_1500e_icdar2015
In Collection: FCENet
Config: configs/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015.py
Expand All @@ -37,3 +50,15 @@ Models:
Metrics:
hmean-iou: 0.8528
Weights: https://download.openmmlab.com/mmocr/textdet/fcenet/fcenet_resnet50_fpn_1500e_icdar2015/fcenet_resnet50_fpn_1500e_icdar2015_20220826_140941-167d9042.pth

- Name: fcenet_oclip-resnet50_fpn_1500e_icdar2015
In Collection: FCENet
Config: configs/textdet/fcenet/fcenet_oclip-resnet50_fpn_1500e_icdar2015.py
Metadata:
Training Data: ICDAR2015
Results:
- Task: Text Detection
Dataset: ICDAR2015
Metrics:
hmean-iou:
Weights:
2 changes: 2 additions & 0 deletions configs/textdet/maskrcnn/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,14 @@ We present a conceptually simple, flexible, and general framework for object ins
| Method | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :----------------------------------------------------------: | :--------------: | :-----------: | :----------: | :-----: | :-------: | :-------: | :----: | :----: | :------------------------------------------------------------: |
| [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500.py) | ImageNet | CTW1500 Train | CTW1500 Test | 160 | 1600 | 0.7165 | 0.7776 | 0.7458 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/mask-rcnn_resnet50_fpn_160e_ctw1500_20220826_154755-ce68ee8e.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/20220826_154755.log) |
| [MaskRCNN_oclip-r50](/configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_ctw1500.py) | - | CTW1500 Train | CTW1500 Test | 160 | 1600 | | | | [model](<>) \| [log](<>) |

### ICDAR2015

| Method | Pretrained Model | Training set | Test set | #epochs | Test size | Precision | Recall | Hmean | Download |
| :--------------------------------------------------------: | :--------------: | :-------------: | :------------: | :-----: | :-------: | :-------: | :----: | :----: | :----------------------------------------------------------: |
| [MaskRCNN](/configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py) | ImageNet | ICDAR2015 Train | ICDAR2015 Test | 160 | 1920 | 0.8644 | 0.7766 | 0.8182 | [model](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/mask-rcnn_resnet50_fpn_160e_icdar2015_20220826_154808-ff5c30bf.pth) \| [log](https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/20220826_154808.log) |
| [MaskRCNN_oclip-r50](/configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_icdar2015.py) | - | ICDAR2015 Train | ICDAR2015 Test | 160 | 1920 | | | | [model](<>) \| [log](<>) |

## Citation

Expand Down
24 changes: 24 additions & 0 deletions configs/textdet/maskrcnn/metafile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,18 @@ Models:
hmean: 0.7458
Weights: https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_ctw1500/mask-rcnn_resnet50_fpn_160e_ctw1500_20220826_154755-ce68ee8e.pth

- Name: mask-rcnn_oclip-resnet50_fpn_160e_ctw1500
In Collection: Mask R-CNN
Config: configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_ctw1500.py
Metadata:
Training Data: CTW1500
Results:
- Task: Text Detection
Dataset: CTW1500
Metrics:
hmean:
Weights:

- Name: mask-rcnn_resnet50_fpn_160e_icdar2015
In Collection: Mask R-CNN
Config: configs/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015.py
Expand All @@ -39,3 +51,15 @@ Models:
Metrics:
hmean: 0.8182
Weights: https://download.openmmlab.com/mmocr/textdet/maskrcnn/mask-rcnn_resnet50_fpn_160e_icdar2015/mask-rcnn_resnet50_fpn_160e_icdar2015_20220826_154808-ff5c30bf.pth

- Name: mask-rcnn_oclip-resnet50_fpn_160e_icdar2015
In Collection: Mask R-CNN
Config: configs/textdet/maskrcnn/mask-rcnn_oclip-resnet50_fpn_160e_icdar2015.py
Metadata:
Training Data: ICDAR2015
Results:
- Task: Text Detection
Dataset: ICDAR2015
Metrics:
hmean:
Weights:
Loading