Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MFR21] ICCV21 masked face recognition challenge discussion #1564

Open
nttstar opened this issue Jun 4, 2021 · 82 comments
Open

[MFR21] ICCV21 masked face recognition challenge discussion #1564

nttstar opened this issue Jun 4, 2021 · 82 comments

Comments

@nttstar
Copy link
Collaborator

nttstar commented Jun 4, 2021 •
edited
Loading

How to participate:
https://github.com/deepinsight/insightface/tree/master/challenges/iccv21-mfr

Submission server:
http://iccv21-mfr.com/

Workshop homepage:
https://ibug.doc.ic.ac.uk/resources/masked-face-recognition-challenge-workshop-iccv-21/

Any issue can be left in this thread.
@nttstar nttstar pinned this issue Jun 4, 2021
@yBeOne
Copy link

yBeOne commented Jun 7, 2021

有谁测试过baseline 模型在IJBC上的准确率吗？我测试时一直报错。

@vuvko
Copy link

vuvko commented Jun 7, 2021

Hello and thank you for organizing the challenge this year!
In your rules, you prohibit the use of pre-trained models and any external data, but there is no mention of how you'd enforce this rule. Will you demand training pipelines and cooperation to reproduce the results from top teams?

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 7, 2021

@vuvko Yes, top-rank players should send their solutions to organizers after submission closed.

@vuvko
Copy link

vuvko commented Jun 7, 2021

Thank you for your quick response!
Will you check the reproducibility of the training pipeline? If so, what will you do in case of a different result model (it can be hardware-dependent)?

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 7, 2021

@vuvko We will strictly check the reproducibility. There should not be any hardware-dependent issue.

@anxiangsir
Copy link
Contributor

有谁测试过baseline 模型在IJBC上的准确率吗？我测试时一直报错。

Please join the QQ group or WeChat group, let’s talk in detail.

@manideep2510
Copy link

manideep2510 commented Jun 9, 2021 •
edited
Loading

Hello
Uploading the model on the submission page is very very slow. I have checked my internet connection and it is good with an upload speed greater than 25 MB/s. Upload to the submission portal is happening at <50 KB/s speed.
Can someone please help me with this?

@vuvko
Copy link

vuvko commented Jun 9, 2021

Same problem here, I cannot upload the baseline iresnet50 model due to very slow upload speed.

@manideep2510
Copy link

manideep2510 commented Jun 9, 2021 •
edited
Loading

Looks like the speed is even low now, less than 10 KB/sec is being uploaded from my system. @nttstar Can you please look into this?
Thanks in advance 🙂

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 10, 2021

@manideep2510 @vuvko The submission server is located in China and it seems to have bandwidth problems for other countries. We will try to find a solution soon.

@nutsam
Copy link

nutsam commented Jun 13, 2021

Is there any preprocessing like normalization before load to model ?
iccv21-mfr only mentioned that
the input shape should equal to 3x112x112 (RGB order)

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 13, 2021

@nutsam Normalization setting will be auto-detected in our onnx_helper.py, please check it.

@vuvko
Copy link

vuvko commented Jun 23, 2021

@nttstar is there any update on the uploading problem?

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 23, 2021

@vuvko We're trying!

@nqkhai1706
Copy link

Hello @nttstar ,
I just submitted a model and I got: "

2021-06-24 21:39:44 ms1m ∅ ∅ ∅ FAILED ∅ -
"
The error message just said "load_onnx_failed". But as I checked my model using your onnx_helper.py, there is no error message. The output is just like in bellow, on my PC.
So, I would like to ask what error message appears if my model's inference time larger than 10 ms?

Thank you,

file_: ms1m_groupface_resnet101.onnx
use onnx-model: onnx/ms1m_groupface_resnet101/ms1m_groupface_resnet101.onnx
input-shape: ['batch_size', 3, 112, 112]
0 Shape_0
1 Constant_1
2 Gather_2
3 Conv_3
4 PRelu_4
5 MaxPool_5
6 BatchNormalization_6
7 Conv_7
max time cost exceed, given 77.5860

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 24, 2021

@nqkhai1706 Have you checked with onnxruntime==1.6 ?

@nqkhai1706
Copy link

@nqkhai1706 Have you checked with onnxruntime==1.6 ?

No, my onnxruntime version: 1.8.0.
May it different on 1.6?
I will check it.
Thank you,

@nqkhai1706
Copy link

@nttstar I have checked again with onnxruntime==1.6, the result is same with onnxruntume==1.8.0.
Do you have any other idea why my model got error on your side?
Thank you,

@anxiangsir
Copy link
Contributor

@nttstar I have checked again with onnxruntime==1.6, the result is same with onnxruntume==1.8.0.
Do you have any other idea why my model got error on your side?
Thank you,

This is our error:

>>> import onnxruntime
>>> session = onnxruntime.InferenceSession("ms1m_groupface_resnet101.onnx",  None)                                                                                                                      Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/face/miniconda3/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 206, in __init__
    self._create_inference_session(providers, provider_options)
  File "/home/face/miniconda3/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 226, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /mnt/data/models/1624534624858999668/ms1m_groupface_resnet101.onnx failed:Protobuf parsing failed.

The environment of our system is as follows:

Driver Version: 450.80.02
Cuda compilation tools, release 10.2, V10.2.89
onnx==1.8.0
onnx-simplifier==0.3.5
onnxoptimizer==0.2.5
onnxruntime-gpu==1.6.0

@zhanglaplace
Copy link

测试的时候能否知道当前的图片是否有mask

@manideep2510
Copy link

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 30, 2021

@manideep2510 What is your username?

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 30, 2021

测试的时候能否知道当前的图片是否有mask

@zhanglaplace 目前不能

@manideep2510
Copy link

@manideep2510 What is your username?

My user name is sherlock.

@anxiangsir
Copy link
Contributor

anxiangsir commented Jun 30, 2021 •
edited
Loading

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

It's probably a bug about jet lag, and we'll fix that in no time.

@JesseEisen
Copy link

@nttstar Even though I have not submitted any model today or yesterday. The portal is still showing that

Today, you have submitted 3 times, 0 remaining.

Can you please look into this? Thanks in advance.

Hi, please provide some information related to the browser to help better debug. Like which browser you use and the version of the browser.

@nttstar
Copy link
Collaborator Author

nttstar commented Jul 1, 2021

@manideep2510 Hi, @JesseEisen is helping us build the evaluation system, please let him know your detailed issue.

@manideep2510
Copy link

I am using Mozilla Firefox Version 89.02. The portal is now showing

Today, you have submitted 8 times, -5 remaining.

But I am able to upload and make submissions.

@nttstar
Copy link
Collaborator Author

nttstar commented Jul 1, 2021

@manideep2510 Can you try Chrome?

@jacqueline-weng
Copy link

image
My model has passed the check_onnx.py in local environmnet with cuda10.2, onnx1.6.0, ort1.8.0 but failed the test.
Could you give more clues about what is going on?
Thank you in advance.

@v-trungdt21
Copy link

Can anybody update the link for the non-China user? I can't access it and submit via the iccv-mfr website is desperately slow for me in this region :(

@beingjoey
Copy link

I can not sign up MFR ongo. Is there any other server?

@nttstar
Copy link
Collaborator Author

nttstar commented Oct 17, 2022

@beingjoey We're upgrading the whole system, please wait for 1-2 days.

@JackLin-Authme
Copy link

JackLin-Authme commented Dec 15, 2022 •
edited
Loading

I have some problems on testing my model with onnx_helper.py
It's seems that self.providers = None would be broken when using onnxruntime-gpu>1.9.0

I think this issue would lead to the load onnx error

@nttstar
Copy link
Collaborator Author

nttstar commented Dec 15, 2022

@JesseEisen No such problem in our system.

@jovialio
Copy link

jovialio commented Feb 16, 2023 •
edited
Loading

Hi i would like to benchmark my model against the MFR dataset and have registered via http://iccv21-mfr.com/#/login. Is the sign up still valid? Have not receive replies nor confirmation email.

@nttstar
Copy link
Collaborator Author

nttstar commented Feb 16, 2023

@jovialio what is your username?

@jovialio
Copy link

@jovialio what is your username?

dennis. Thanks it's been approved!

@HaoLiuHust
Copy link

减均值除方差的操作是由模型做还是测试时会做？

@nttstar
Copy link
Collaborator Author

nttstar commented Feb 21, 2023

如果onnx模型前几层有Sub Div类似的操作会被自动监测到，这样测试代码不会做额外mean std处理。否则会采用默认的mean=127.5, std=127.5。 @HaoLiuHust

@hyperf0cus
Copy link

hyperf0cus commented Feb 8, 2024 •
edited
Loading

@nttstar
Trained with arcface_torch MobileFaceNet MFN model, converted via torch2onnx.py, then converted onnx model weights can't pass onnx_helper.py, and obviously can't pass submission on MFR challenge...
evaluation tool MFR failed: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running MatMul node. Name:'MatMul_95' Status Message: matmul_helper.h:57 Compute MatMul dimension mismatch
I didn't change architecture, structure, layers inside backbones mobilefacenet, but error happens...

@abdu-khamd
Copy link

@nttstar I am also having problems with uploading my model. It passed onnx_helper.py. However, when submitting to the server I get this error evaluation tool MFR failed: load onnx failed.

@nttstar
Copy link
Collaborator Author

nttstar commented Feb 9, 2024

@hyperf0cus @abdu-khamd Make sure the onnxruntime-gpu version in your test env matches our guide.

@thNam98
Copy link

thNam98 commented Feb 20, 2024

Hi @nttstar , i would like to benchmark my model against the MFR dataset and have registered user. But always received "user unconfirmed after register". Can you check again for me, thanks!

@Cevikalp
Copy link

Hello, I would like to benchmark my model against the MFR dataset and have registered via http://iccv21-mfr.com/#/login. Is the sign up still valid? I did not receive any confirmation email.

@amoonfana
Copy link

Hi, it seems new registering via http://iccv21-mfr.com/#/login is not available anymore. Is there any way if we want to test our face recognition method on MFR?

@nttstar
Copy link
Collaborator Author

nttstar commented Mar 27, 2024

I have approved all new registrations.

@Cevikalp
Copy link

Cevikalp commented Mar 30, 2024 via email

@nttstar
Copy link
Collaborator Author

nttstar commented Mar 30, 2024

@Cevikalp Sorry we don't support the model evaluation with such high feature dimension.

@Cevikalp
Copy link

Cevikalp commented Mar 30, 2024 via email

@Cevikalp
Copy link

Cevikalp commented Mar 30, 2024 via email

@juechengxi
Copy link

您好，我想根据 MFR 数据集对我的模型进行基准测试，但没有收到任何确认电子邮件，能批准一下新注册吗？

@nttstar
Copy link
Collaborator Author

nttstar commented Apr 26, 2024

@juechengxi 已全部通过

@vivianhew
Copy link

您好，我想参加 MFR-Ongoing，注册后没有收到任何确认电子邮件，能批准一下新注册吗？

@HYL-Dave
Copy link

HYL-Dave commented Jun 4, 2024

@nttstar 你好，你们提供的 vit-l 代码训练出来就超过了 1 GB，无法上传测试，请问用什么方式可以 upload 测试？

@onnx20
Copy link

onnx20 commented Jun 7, 2024

I have approved all new registrations.

@nttstar Please request registration of MFR, username: koala, Thanks.

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 7, 2024

@nttstar 你好，你们提供的 vit-l 代码训练出来就超过了 1 GB，无法上传测试，请问用什么方式可以 upload 测试？

@HYL-Dave 没有办法

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 7, 2024

您好，我想参加 MFR-Ongoing，注册后没有收到任何确认电子邮件，能批准一下新注册吗？

已通过

@nttstar
Copy link
Collaborator Author

nttstar commented Jun 7, 2024

I have approved all new registrations.

@nttstar Please request registration of MFR, username: koala, Thanks.

DONE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests