It is too slow for irtr

[shizhediao]

Hi authors,
Thanks for your great work!
I am trying to reproduce the results but found it is too slow for irtr testing.
It seems that it needs 38 hours for inference on advanced computing architectures (like A100), specifically it is the rank loop that costs so much time.
It is normal?

Thanks!

[zdou0830]

Thanks! It is normal as for N test instances, ranking requires computing features for NxN image-text pairs. One potential way to increase the inference speed is caching the image and text features at the bottom layers, which I left as a TODO here https://github.com/zdou0830/METER/blob/main/meter/modules/objectives.py#L315.

[shizhediao]

OK, I see. Thanks for your reply!

…

----------------- Best Regards, Shizhe Diao Ph.D. Student, Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong Email: ***@***.***

________________________________ 发件人: Zi-Yi Dou ***@***.***> 发送时间: 2022年1月26日 9:23 收件人: zdou0830/METER ***@***.***> 抄送: DIAO Shizhe ***@***.***>; Author ***@***.***> 主题: Re: [zdou0830/METER] It is too slow for irtr (Issue #12) Thanks! It is normal as for N test instances, ranking requires computing features for NxN image-text pairs. One potential way to increase the inference speed is caching the image and text features at the bottom layers, which I left as a TODO here https://github.com/zdou0830/METER/blob/main/meter/modules/objectives.py#L315<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fzdou0830%2FMETER%2Fblob%2Fmain%2Fmeter%2Fmodules%2Fobjectives.py%23L315&data=04%7C01%7Cshizhe.diao%40connect.ust.hk%7C787158867b2648e990fc08d9e06a7f2f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637787570271215648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=47rcynYvBFImoLG4moNh3vTCN9TKRot0rtMKUYgiCCo%3D&reserved=0>. ― Reply to this email directly, view it on GitHub<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fzdou0830%2FMETER%2Fissues%2F12%23issuecomment-1021766705&data=04%7C01%7Cshizhe.diao%40connect.ust.hk%7C787158867b2648e990fc08d9e06a7f2f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637787570271215648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lv3ioajPLw5%2FEvxatpK4w9fPQVSUySFTit%2FbklrDDX0%3D&reserved=0>, or unsubscribe<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEKH3F2OCH266IFHPZDJ3FDUX5EJ7ANCNFSM5MZYDTYA&data=04%7C01%7Cshizhe.diao%40connect.ust.hk%7C787158867b2648e990fc08d9e06a7f2f%7C6c1d415239d044ca88d9b8d6ddca0708%7C1%7C0%7C637787570271215648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ocNGuA%2FTYkjVsRf3iejJcb0%2Fx6ooPQD14CI5oqAh3S0%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>