Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Accuracy of the network on the 50000 test images: 0.1% #134

Closed
DCBXZ66 opened this issue Oct 23, 2022 · 4 comments
Closed

Accuracy of the network on the 50000 test images: 0.1% #134

DCBXZ66 opened this issue Oct 23, 2022 · 4 comments
Labels
TinyViT

Comments

@DCBXZ66
Copy link

DCBXZ66 commented Oct 23, 2022

Hello, the author, this is a very meaningful work. I encountered this accuracy problem when running the following code:
python -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/22k_distill/tiny_vit_5m_22k_distill.yaml --data-path ./ImageNet --batch-size 128 --eval --resume ./checkpoints/tiny_vit_5m_22k_distill.pth --opts DATA.DATASET imagenet
Did I do something wrong?The dataset I use is ILSVRC2012. Is this what the project calls ImageNet?
On the other hand, I would like to ask how to evaluate it on a computer only with CPU?
Look forward to your reply, thank you!
@wkcn
Copy link
Contributor

wkcn commented Oct 23, 2022 •
edited

Hi @DCBXZ66, thanks for your attention to our work.

ILSVRC2012 is ImageNet.

The command is correct. I run the same command and it outputs:

[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 16): INFO ==============> Resuming form ./checkpoints/tiny_vit_5m_22k_distill.pth....................
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 34): INFO Convert checkpoint from 21841 to 1k
[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 50): INFO <All keys matched successfully>
[2022-10-24 00:34:55 TinyViT-5M-22k-distill](main.py 468): INFO Test: [0/49]    Time 5.734 (5.734)      Loss 0.2961 (0.2961)    Acc@1 93.750 (93.750)   Acc@5 98.438 (98.438)   Mem 2001MB
[2022-10-24 00:34:59 TinyViT-5M-22k-distill](main.py 468): INFO Test: [10/49]   Time 0.442 (0.855)      Loss 0.9443 (0.7816)    Acc@1 80.469 (80.682)   Acc@5 92.969 (94.744)   Mem 2001MB
[2022-10-24 00:35:02 TinyViT-5M-22k-distill](main.py 468): INFO Test: [20/49]   Time 0.319 (0.593)      Loss 0.9976 (0.7569)    Acc@1 74.219 (81.138)   Acc@5 95.312 (95.647)   Mem 2001MB
[2022-10-24 00:35:05 TinyViT-5M-22k-distill](main.py 468): INFO Test: [30/49]   Time 0.305 (0.502)      Loss 1.1641 (0.8307)    Acc@1 71.875 (79.284)   Acc@5 88.281 (94.556)   Mem 2001MB
[2022-10-24 00:35:08 TinyViT-5M-22k-distill](main.py 468): INFO Test: [40/49]   Time 0.303 (0.454)      Loss 1.2490 (0.8959)    Acc@1 73.438 (77.668)   Acc@5 88.281 (93.979)   Mem 2001MB
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 478): INFO  The number of validation samples is 50000
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 480): INFO  * Acc@1 77.370 Acc@5 94.070
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 158): INFO Accuracy of the network on the 50000 test images: 77.4%

Do other models even other codebases obtain the correct accuracy?

how to evaluate it on a computer only with CPU?

The PR #136 supports CPU inference for TinyViT. It needs an extra argument --only-cpu.

python -m torch.distributed.launch --nproc_per_node 1 main.py --cfg configs/22k_distill/tiny_vit_5m_22k_distill.yaml --data-path ./ImageNet --batch-size 128 --eval --resume ./checkpoints/tiny_vit_5m_22k_distill.pth --only-cpu --opts DATA.DATASET imagenet

@wkcn wkcn added the TinyViT label Oct 23, 2022
@DCBXZ66
Copy link
Author

DCBXZ66 commented Oct 27, 2022

Hi @DCBXZ66, thanks for your attention to our work.

ILSVRC2012 is ImageNet.

The command is correct. I run the same command and it outputs:

[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 16): INFO ==============> Resuming form ./checkpoints/tiny_vit_5m_22k_distill.pth....................
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
All checkpoints founded in output/TinyViT-5M-22k-distill/default: []
[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 34): INFO Convert checkpoint from 21841 to 1k
[2022-10-24 00:34:49 TinyViT-5M-22k-distill](utils.py 50): INFO <All keys matched successfully>
[2022-10-24 00:34:55 TinyViT-5M-22k-distill](main.py 468): INFO Test: [0/49]    Time 5.734 (5.734)      Loss 0.2961 (0.2961)    Acc@1 93.750 (93.750)   Acc@5 98.438 (98.438)   Mem 2001MB
[2022-10-24 00:34:59 TinyViT-5M-22k-distill](main.py 468): INFO Test: [10/49]   Time 0.442 (0.855)      Loss 0.9443 (0.7816)    Acc@1 80.469 (80.682)   Acc@5 92.969 (94.744)   Mem 2001MB
[2022-10-24 00:35:02 TinyViT-5M-22k-distill](main.py 468): INFO Test: [20/49]   Time 0.319 (0.593)      Loss 0.9976 (0.7569)    Acc@1 74.219 (81.138)   Acc@5 95.312 (95.647)   Mem 2001MB
[2022-10-24 00:35:05 TinyViT-5M-22k-distill](main.py 468): INFO Test: [30/49]   Time 0.305 (0.502)      Loss 1.1641 (0.8307)    Acc@1 71.875 (79.284)   Acc@5 88.281 (94.556)   Mem 2001MB
[2022-10-24 00:35:08 TinyViT-5M-22k-distill](main.py 468): INFO Test: [40/49]   Time 0.303 (0.454)      Loss 1.2490 (0.8959)    Acc@1 73.438 (77.668)   Acc@5 88.281 (93.979)   Mem 2001MB
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 478): INFO  The number of validation samples is 50000
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 480): INFO  * Acc@1 77.370 Acc@5 94.070
[2022-10-24 00:35:11 TinyViT-5M-22k-distill](main.py 158): INFO Accuracy of the network on the 50000 test images: 77.4%

Do other models even other codebases obtain the correct accuracy?

how to evaluate it on a computer only with CPU?

The PR #136 supports CPU inference for TinyViT. It needs an extra argument --only-cpu.

python -m torch.distributed.launch --nproc_per_node 1 main.py --cfg configs/22k_distill/tiny_vit_5m_22k_distill.yaml --data-path ./ImageNet --batch-size 128 --eval --resume ./checkpoints/tiny_vit_5m_22k_distill.pth --only-cpu --opts DATA.DATASET imagenet

Thank you for your timely reply. I tried again and found the problem: val data in the tar package downloaded directly from ImageNet does not have a label folder, so it needs to be handled. After unzipping, I added a label folder to solve the problem of accuracy.
On the other hand, the "my_meter. py" file needs to be modified to run code on a PC with only CPU (and no cuda):

    def sync(self):
        # buf = torch.tensor([self._sum, self._count],
        #                    dtype=torch.float32).cuda()
        buf = torch.tensor([self._sum, self._count],
                           dtype=torch.float32)

@wkcn
Copy link
Contributor

wkcn commented Oct 27, 2022

@DCBXZ66 Great!
I will fix the bug in "my_meter.py". Thank you!

@wkcn
Copy link
Contributor

wkcn commented Oct 31, 2022

Close the issue since the issue has been resolved and the PR #136 has been merged.

@wkcn wkcn closed this as completed Oct 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
TinyViT
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wkcn @DCBXZ66