Models for the Google Summer of Code 2023 prerequisite task

[adrianboguszewski]

The models listed below are for the GSoC 2023 prerequisite task only.

We provide several potential candidates. Please select only one which hasn't already been selected (look at the checkboxes and comments below). When you decide, assign a model to you by adding a comment with the model name. Then we will tick it to mark reserved.

If you struggle, you can reassign yourself to another non-taken model. However, we can do it only once.

When you create a PR, please follow the self-checklist below:

• each function is described by docstrings and type hints
• notebook contains explicit descriptions and explanatory diagrams
• the notebook doesn't use any data (image, video, etc.) that is not CC4.0 licensed
• there is a README.md file in consistent style (look at other notebooks)
• the notebook is added to the main README
• there are no grammar, punctuation or typo issues (use any free tool for that e.g. Grammarly)
• there are no committed files besides notebook and readme (please use images or videos from data dir)
• your PR doesn't change any other notebooks
• all CI checks passed

Object detection:

• Yolov6 - https://github.com/meituan/YOLOv6 (@ahmd-nish)
• DAMO YOLO - https://github.com/tinyvision/DAMO-YOLO (@Muskan33)
• YoloX - https://github.com/Megvii-BaseDetection/YOLOX (@sawradip)
• RTMDet - https://github.com/open-mmlab/mmyolo/tree/main/configs/rtmdet (@AnuragMaiti)
• EfficientDet - https://github.com/google/automl/tree/master/efficientdet (@ashish-2005)
• CenterNet - https://github.com/xingyizhou/CenterNet/ (@rajuptvs)
• SSD MobileNet V2 http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md (@AlexFierro9)
• FasterRCNN Inception ResNet v2 http://download.tensorflow.org/models/object_detection/tf2/20200711/faster_rcnn_inception_resnet_v2_640x640_coco17_tpu-8.tar.gz https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md (@Paulooh007)
• YOLOS https://huggingface.co/hustvl/yolos-tiny (@SandeepaDevin)
• DETR https://huggingface.co/facebook/detr-resnet-50 (@Tatwansh)
• YoloR https://github.com/WongKinYiu/yolor (@18yz153)
• YoloF https://github.com/megvii-model/YOLOF (@thegeek13242)
• NanoDet https://github.com/RangiLyu/nanodet (@sahilpmehra)
• UltraFace https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB (@JacketChenlll)
• YoloV7 Face https://github.com/derronqi/yolov7-face (@lucifertrj)
• yolov5-blazeface https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/205-vision-background-removal (@AnuragTimilsina)
• RetinaFace https://github.com/biubug6/Pytorch_Retinaface (@VaillaRohit)

Rotated object detection:

• Rotated FCOS - https://github.com/open-mmlab/mmrotate/blob/main/configs/rotated_fcos/README.md
• ReDet - https://github.com/open-mmlab/mmrotate/blob/main/configs/redet/README.md (@nischay7)
• Roi_trans - https://github.com/open-mmlab/mmrotate/blob/main/configs/roi_trans/README.md

Semantic Segmentation:

• SegFormer - https://huggingface.co/nvidia/segformer-b0-finetuned-ade-512-512 (@Kasliwal17)
• ClipSeg - https://huggingface.co/CIDAS/clipseg-rd64-refined (@RishithaR-388)
• SETR - https://github.com/fudan-zvg/SETR (@AniketARS)
• BeIT - https://huggingface.co/microsoft/beit-base-finetuned-ade-640-640 (@hadyy17)
• Segmenter - https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segmenter (@blaz-r)
• DeepLab V3 - https://github.com/tensorflow/models/tree/master/research/deeplab (@chaitravi-ce)
• FaceParsing |& MakeUp - https://github.com/zllrunning/face-parsing.PyTorch https://github.com/zllrunning/face-makeup.PyTorch (@Lj1ang)
• ESPNet https://github.com/sacmehta/ESPNet (@Nouran-Muhammad)
• YoloP https://github.com/hustvl/YOLOP

Instance Segmentation:

• YOLACT https://github.com/dbolya/yolact.git (@Abdullah-Elkasaby)
• Mask RCNN Inception ResNet V2 http://download.tensorflow.org/models/object_detection/tf2/20200711/mask_rcnn_inception_resnet_v2_1024x1024_coco17_gpu-8.tar.gz https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md (@mr-rajashekhar)

Action/Gesture recognition:

• TSM - https://github.com/mit-han-lab/temporal-shift-module (@ntombi)
• Timesformer - https://huggingface.co/facebook/timesformer-base-finetuned-k400 (@BrennoMello)
• SlowFast - https://github.com/open-mmlab/mmaction2/blob/master/configs/recognition/slowfast/README.md (@rajatkrishna)
• YOWOv2 - https://github.com/yjh0410/YOWOv2 (@Matrixmang0)
• movinet - https://github.com/tensorflow/models/tree/master/official/projects/movinet (@sharvesh642)
• xclip https://huggingface.co/microsoft/xclip-base-patch32

Background matting:

• ModNet - https://github.com/ZHKKKe/MODNet
• Robust Video Background matting - https://github.com/PeterL1n/RobustVideoMatting (@wulongjian)
• MGMatting https://github.com/yucornetto/MGMatting
• PortraitNet https://github.com/dong-x16/PortraitNet

Old Photos Restoration/Image colorization/Image denoising/super resolution:

• Bringing Old Photos Back to Life - https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life (@Om-Doiphode )
• DeOldify - https://github.com/jantic/DeOldify (@Dhruvanshu-Joshi)
• Coltran - https://github.com/google-research/google-research/tree/master/coltran (@weronikazak)
• Colorizer https://github.com/richzhang/colorization (@pyther-hub)
• SwinIR - https://github.com/JingyunLiang/SwinIR (@Z-Fran)
• style-swapping https://github.com/irasin/Pytorch_Style_Swap (@m-gopichand)
• Real-ESRGAN (for real images) - https://github.com/xinntao/Real-ESRGAN/blob/master/docs/model_zoo.md (@aadhamm)
• Real-ESRGAN (for animation video) - https://github.com/xinntao/Real-ESRGAN/blob/master/docs/model_zoo.md
• RCAN https://github.com/yulunzhang/RCAN
• Super-SlowMo https://github.com/rmalav15/Super-SloMo
• Photo2Cartoon https://github.com/minivision-ai/photo2cartoon (@sususama)

Depth estimation:

• lite-mono https://github.com/noahzn/lite-monoc
• MiDaS 3.1 https://github.com/isl-org/MiDaS (@nsk126)
• Vi-Depth https://github.com/isl-org/VI-Depth (@pronoym99)

Text classification:

• Roberta - https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment @ABHIJATSARARI)
• XLM-Roberta - https://huggingface.co/papluca/xlm-roberta-base-language-detection (@hazrulakmal)
• DepRoBerta - https://huggingface.co/rafalposwiata/deproberta-large-depression (@SpyzzVVarun)
• CodeBerta - https://huggingface.co/huggingface/CodeBERTa-language-id (@zilto)
• Albert V2 - https://huggingface.co/textattack/albert-base-v2-MRPC (@dwipddalal)
• DistilRoberta - https://huggingface.co/j-hartmann/emotion-english-distilroberta-base (@MR-ENVYR)
• FinBERT https://huggingface.co/yiyanghkust/finbert-tone (@shrey-2803)
• Deberta https://huggingface.co/microsoft/deberta-base-mnli (@mhy-666)

Token classification:

• Part of speech tagging - https://huggingface.co/flair/pos-english (@harish2773)
• Punctuation restoring (bert-restore-punctuation) - https://huggingface.co/felflare/bert-restore-punctuation (@seanjyu)
• Punctuation restoring (punctuate-all) - https://huggingface.co/kredor/punctuate-all (@theNobody-12)
• Typo detection - https://huggingface.co/m3hrdadfi/typo-detector-distilbert-en (@Ravindu987)
• Named entity recognition - https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english (@Aditya-vardhan13)

Text generation:

• BioGPT https://huggingface.co/microsoft/biogpt (@sidyakinian)
• gpt-neo https://huggingface.co/EleutherAI/gpt-neo-125M (@Warlord-K)
• OPT https://huggingface.co/facebook/opt-350m (@zhumakhan)

Text Summarization

• DistilBART https://huggingface.co/sshleifer/distilbart-cnn-12-6 (@samycolen)

Question Answering

• MiniLM https://huggingface.co/deepset/minilm-uncased-squad2 (@Akshit17)
• ELECTRA https://huggingface.co/deepset/electra-base-squad2 (@sanjayk0508)
• DistilBert https://huggingface.co/distilbert-base-cased-distilled-squad (@fajemila)

Sound classification:

• speech emotions recognition (wav2vec) https://huggingface.co/harshit345/xlsr-wav2vec-speech-emotion-recognition (@paxF3E)
• Hubert key words spotting https://huggingface.co/superb/hubert-base-superb-ks (@100-87)

[sawradip]

Hello, firstly I want to thank you for the awesome list. I had a question:

Is there any existing instance/example of using open-mmlab models, or such config based model libraries(Ex: detectron2), with Openvino?

[sawradip]

I would like to assign YoloX to myself.

[adrianboguszewski]

@sawradip, assigned to you. We don't have such examples in this repository and I'm not aware of any existing examples of open-mmlab or detectron2 with OpenVINO.

[eaidova]

@sawradip please check https://github.com/open-mmlab/mmdeploy this library support export mmlab pretrained models to OpenVINO format

[sawradip]

Thank you for assigning me that task. I will submit a PR on that, in a few days.

And I am interested on the open-mm projects is, such a good number of models, are out of openvino accessibility. Though It is not GSOC requirement, neither on planned projects, I was thinking, If I could contribute some tutorial, on integrating open-mm or similar config based models.

[akashAD98]

@eaidova should we need to implement any of the repo in openvino ir format? this is only for GSoC people or anyone from open source community can contribute

[eaidova]

@akashAD98, this activity is only for GSoC participants for now (we use this models list for candidates selection), but we are open for community contributions too :)

If you have any ideas or great models for openvino notebooks, feel free to contribute :)

[Kasliwal17]

Hi, I would like to assign SegFormer to myself.

[ahmd-nish]

Hi, i would like to assign YOLOv6 to myslef and contribute. Can you please help me with that ?

[adrianboguszewski]

Hi @ahmd-nish, please first clone the repo locally and make it run. Then please look at the 226-yolo7 notebook and create a notebook with the similar flow.

[chathuranga-jayanath-99]

Hi, I would like to assign Roberta to myself.

[ahmd-nish]

Hi @ahmd-nish, please first clone the repo locally and make it run. Then please look at the 226-yolo7 notebook and create a notebook with the similar flow.

Sure I will go through and get backto you

[hazrulakmal]

Hello, I would like to work on XLM-RoBERTa.

[raymondlo84]

This is really cool @adrianboguszewski -- Also, would be good to put tags in Discussion just on GSOC2023 topics to keep it organized.

[raymondlo84]

https://github.com/openvinotoolkit/openvino_notebooks/discussions/categories/gsoc-2023 :) Here we go.

[ashish-2005]

Hi , I would like to assign EfficientDet to myself

[SpyzzVVarun]

Hi, I would like to assign XLM-RoBERTa to myself.

[adrianboguszewski]

@SpyzzVVarun XLM-Roberta has already been assigned four comments above. Please select another model :)

[SpyzzVVarun]

Then I would like to work with DepRoBerta.

[pyther-hub]

I would like to work with DAMO YOLO, My proficiency lies in computer vision.

[pyther-hub]

sir/ ma'am by what time shall we put a pr for the Prerequisite task notebook

[adrianboguszewski]

The PR must be created (doesn't have to be merged) by April 4, as the link must be included in your GSoC application.

[Om-Doiphode]

Hi, I would like to assign Bringing Old Photos Back to Life to myself.

[AlexFierro9]

Hi, Would like to assign Timesformer to myself!

[pronoym99]

Hi @adrianboguszewski It would be great if YoLoP for semantic segmentation could be assigned to me as a pre-requisite for GSoC'23. Additionally, it would be great if some SOTA YoLov8 object detection models from the Ultralytics repo could be included in object detection — just a small suggestion from my side.

[ABHIJATSARARI]

I would like to assign Roberta to myself.

[mhy-666]

Hello, I have completed all the requirements requested in my PR, but I am unable to tick the checkbox items. Could you please review my PR?

[matrixbot123]

Hi, I would like to release please.

[BrennoMello]

Hello @adrianboguszewski , is it a problem not using the activity's repository?
My model timesformer specifies this repository (https://huggingface.co/facebook/timesformer-base-finetuned-k400), but huggingface API does not export the model timesformer-base-finetuned-k400 to ONNX. So, I started to use this repository (https://github.com/wanchichen/TimeSformer), a fork from the official with new corrections.

[adrianboguszewski]

Hello, I have completed all the requirements requested in my PR, but I am unable to tick the checkbox items. Could you please review my PR?

You can copy a list and post it yourself. Then you're able to tick.

[adrianboguszewski]

Hi @adrianboguszewski It would be great if YoLoP for semantic segmentation could be assigned to me as a pre-requisite for GSoC'23. Additionally, it would be great if some SOTA YoLov8 object detection models from the Ultralytics repo could be included in object detection — just a small suggestion from my side.

It was already done a few months ago. Check notebook 230.

[adrianboguszewski]

Hello @adrianboguszewski , is it a problem not using the activity's repository? My model timesformer specifies this repository (https://huggingface.co/facebook/timesformer-base-finetuned-k400), but huggingface API does not export the model timesformer-base-finetuned-k400 to ONNX. So, I started to use this repository (https://github.com/wanchichen/TimeSformer), a fork from the official with new corrections.

It's ok to clone another repository, but it should be avoided if more straightforward methods to get a model are available.

[samycolen]

Hello there ........
Would like to assign DistilBART to myself!

[ntombi]

Hi. I would like to assign TSM to myself. Thanks.

[pronoym99]

Hi, @adrianboguszewski I would like to relieve myself of YoLop and would instead assign myself VI-Depth. Sorry for any inconvenience caused.

[sahilpmehra]

Hi, @adrianboguszewski I am having trouble converting the Rotated FCOS pytorch model to onnx format due to certain support issues. I am sure there is a way around it but due to the deadline coming up soon, I would like to release Rotated FCOS and instead assign NanoDet to myself. Apologies for the inconvenience.

[iaminebriki]

I would like to assign NanoDet to myself!

[Lj1ang]

I'd like to assign FaceParsing |& MakeUp to myself

[MR-ENVYR]

Hello, Can we install any other library using !pip install inside the notebook for our specific case or do we need to only make do with the shared libraries of requirements.txt? I wish to use the transformers and optimum library library here.
Thanks in advance :D

[samycolen]

hello @adrianboguszewski where do i have to submit the PR ? is it in My Proposal?

[18yz153]

Hello, I am working on YoloR. The weight and dataset are downloaded by script. It is sh scripts. It does not work on my Windows. But it works on colab. I tried to use notebook_utils download_file. It can not download google drive files.
How am I supposed to download weights and datasets? I already convert the model into openvino. It works on inference.

[VaillaRohit]

Hello, I have completed the prerequisite task.
PR-GSOC-2023 Retinaface by @VaillaRohit #979

[MR-ENVYR]

Hello, I am working on YoloR. The weight and dataset are downloaded by script. It is sh scripts. It does not work on my Windows. But it works on colab. I tried to use notebook_utils download_file. It can not download google drive files. How am I supposed to download weights and datasets? I already convert the model into openvino. It works on inference.

Try to change your google drive link using this website. Passing this link to the notebook_utils download_file function worked for me. 🤠

[18yz153]

Hello, I am working on YoloR. The weight and dataset are downloaded by script. It is sh scripts. It does not work on my Windows. But it works on colab. I tried to use notebook_utils download_file. It can not download google drive files. How am I supposed to download weights and datasets? I already convert the model into openvino. It works on inference.

Try to change your google drive link using this website. Passing this link to the notebook_utils download_file function worked for me. 🤠

I download nothing. Google Drive can't scan this file for viruses.

yolor_p6.pt (143M) is too large for Google to scan for viruses. Would you still like to download this file?
I guess this is the problem. Google ask me to click download again. Anyone know how to fix this?

[adrianboguszewski]

I would like to assign NanoDet to myself!

@iamaminebriki, the model has been selected a one comment above you, please assign another one.

[adrianboguszewski]

Hello, Can we install any other library using !pip install inside the notebook for our specific case or do we need to only make do with the shared libraries of requirements.txt? I wish to use the transformers and optimum library library here. Thanks in advance :D

@MR-ENVYR, yes it's ok to use pip in a notebook. It has been done like that in e.g. 230-yolov8

[adrianboguszewski]

hello @adrianboguszewski where do i have to submit the PR ? is it in My Proposal?

@samycolen , your PR must be open in this repository. If you don't know what it means, you should check this blog

[adrianboguszewski]

Hello, I am working on YoloR. The weight and dataset are downloaded by script. It is sh scripts. It does not work on my Windows. But it works on colab. I tried to use notebook_utils download_file. It can not download google drive files. How am I supposed to download weights and datasets? I already convert the model into openvino. It works on inference.

Try to change your google drive link using this website. Passing this link to the notebook_utils download_file function worked for me. 🤠

I download nothing. Google Drive can't scan this file for viruses.

yolor_p6.pt (143M) is too large for Google to scan for viruses. Would you still like to download this file? I guess this is the problem. Google ask me to click download again. Anyone know how to fix this?

@eaidova, do you know how to deal with that issue?

[sawradip]

Hello, I am working on YoloR. The weight and dataset are downloaded by script. It is sh scripts. It does not work on my Windows. But it works on colab. I tried to use notebook_utils download_file. It can not download google drive files. How am I supposed to download weights and datasets? I already convert the model into openvino. It works on inference.

Try to change your google drive link using this website. Passing this link to the notebook_utils download_file function worked for me. 🤠

I download nothing. Google Drive can't scan this file for viruses.
yolor_p6.pt (143M) is too large for Google to scan for viruses. Would you still like to download this file? I guess this is the problem. Google ask me to click download again. Anyone know how to fix this?

@eaidova, do you know how to deal with that issue?

I think I can help.

You can use gdown. it works for all OS. Just extract the file id from the google drive link and use gdown <drive-id>. For larger files you can use: gdown "<drive-id>&confirm=t". Got from here

It also has a python API where you can import gdown in pythonic way (look in the linked Repo). Hope this helps!

[adrianboguszewski]

As the 4th of April has passed, this task is suspended until further notice. Thank you all for your contribution :)

[raymondlo84]

Thank you everybody <3