Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reproducibility #50

Closed
nttung1110 opened this issue Jul 4, 2023 · 11 comments
Closed

Reproducibility #50

nttung1110 opened this issue Jul 4, 2023 · 11 comments

Comments

@nttung1110
Copy link

nttung1110 commented Jul 4, 2023 •
edited
Loading

Hi,

Thanks so much for the cool project. I tried to reproduce the results reported in your paper and failed to reproduce it though I did not change anything. Here is the result of icarl training on cifar100 base 0:

It would be great if you could run the same experiment from your side and compare with what I have here. Thanks and really appreciate your help.

2023-05-22 17:20:04,654 [trainer.py] => prefix: reproduce
2023-05-22 17:20:04,654 [trainer.py] => dataset: cifar100
2023-05-22 17:20:04,654 [trainer.py] => memory_size: 2000
2023-05-22 17:20:04,654 [trainer.py] => memory_per_class: 20
2023-05-22 17:20:04,654 [trainer.py] => fixed_memory: False
2023-05-22 17:20:04,654 [trainer.py] => shuffle: True
2023-05-22 17:20:04,654 [trainer.py] => init_cls: 10
2023-05-22 17:20:04,654 [trainer.py] => increment: 10
2023-05-22 17:20:04,654 [trainer.py] => model_name: icarl
2023-05-22 17:20:04,654 [trainer.py] => convnet_type: resnet32
2023-05-22 17:20:04,654 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 17:20:04,654 [trainer.py] => seed: 1993
2023-05-22 17:20:06,016 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 17:20:06,132 [trainer.py] => All params: 464154
2023-05-22 17:20:06,132 [trainer.py] => Trainable params: 464154
2023-05-22 17:20:06,132 [icarl.py] => Learning on 0-10
2023-05-22 18:17:16,663 [icarl.py] => Task 0, Epoch 200/200 => Loss 0.038, Train_accy 99.44
2023-05-22 18:17:16,663 [base.py] => Reducing exemplars...(200 per classes)
2023-05-22 18:17:16,663 [base.py] => Constructing exemplars...(200 per classes)
2023-05-22 18:17:29,541 [icarl.py] => Exemplar size: 2000
2023-05-22 18:17:29,541 [trainer.py] => CNN: {'total': 91.2, '00-09': 91.2, 'old': 0, 'new': 91.2}
2023-05-22 18:17:29,541 [trainer.py] => NME: {'total': 91.6, '00-09': 91.6, 'old': 0, 'new': 91.6}
2023-05-22 18:17:29,541 [trainer.py] => CNN top1 curve: [91.2]
2023-05-22 18:17:29,541 [trainer.py] => CNN top5 curve: [99.3]
2023-05-22 18:17:29,541 [trainer.py] => NME top1 curve: [91.6]
2023-05-22 18:17:29,541 [trainer.py] => NME top5 curve: [99.8]

2023-05-22 18:17:29,541 [trainer.py] => All params: 464804
2023-05-22 18:17:29,541 [trainer.py] => Trainable params: 464804
2023-05-22 18:17:29,545 [icarl.py] => Learning on 10-20
2023-05-22 19:24:28,935 [icarl.py] => Task 1, Epoch 170/170 => Loss 1.043, Train_accy 99.64
2023-05-22 19:24:28,935 [base.py] => Reducing exemplars...(100 per classes)
2023-05-22 19:24:33,625 [base.py] => Constructing exemplars...(100 per classes)
2023-05-22 19:24:45,992 [icarl.py] => Exemplar size: 2000
2023-05-22 19:24:45,992 [trainer.py] => CNN: {'total': 79.6, '00-09': 79.6, '10-19': 79.6, 'old': 79.6, 'new': 79.6}
2023-05-22 19:24:45,992 [trainer.py] => NME: {'total': 79.1, '00-09': 81.8, '10-19': 76.4, 'old': 81.8, 'new': 76.4}
2023-05-22 19:24:45,992 [trainer.py] => CNN top1 curve: [91.2, 79.6]
2023-05-22 19:24:45,992 [trainer.py] => CNN top5 curve: [99.3, 96.5]
2023-05-22 19:24:45,992 [trainer.py] => NME top1 curve: [91.6, 79.1]
2023-05-22 19:24:45,992 [trainer.py] => NME top5 curve: [99.8, 96.3]

2023-05-22 19:24:45,992 [trainer.py] => All params: 465454
2023-05-22 19:24:45,992 [trainer.py] => Trainable params: 465454
2023-05-22 19:24:45,999 [icarl.py] => Learning on 20-30
2023-05-22 20:31:37,699 [icarl.py] => Task 2, Epoch 170/170 => Loss 1.192, Train_accy 99.81
2023-05-22 20:31:37,700 [base.py] => Reducing exemplars...(66 per classes)
2023-05-22 20:31:47,398 [base.py] => Constructing exemplars...(66 per classes)
2023-05-22 20:31:59,640 [icarl.py] => Exemplar size: 1980
2023-05-22 20:31:59,640 [trainer.py] => CNN: {'total': 73.17, '00-09': 71.6, '10-19': 58.9, '20-29': 89.0, 'old': 65.25, 'new': 89.0}
2023-05-22 20:31:59,640 [trainer.py] => NME: {'total': 75.4, '00-09': 76.7, '10-19': 65.1, '20-29': 84.4, 'old': 70.9, 'new': 84.4}
2023-05-22 20:31:59,640 [trainer.py] => CNN top1 curve: [91.2, 79.6, 73.17]
2023-05-22 20:31:59,641 [trainer.py] => CNN top5 curve: [99.3, 96.5, 94.27]
2023-05-22 20:31:59,641 [trainer.py] => NME top1 curve: [91.6, 79.1, 75.4]
2023-05-22 20:31:59,641 [trainer.py] => NME top5 curve: [99.8, 96.3, 94.27]

2023-05-22 20:31:59,641 [trainer.py] => All params: 466104
2023-05-22 20:31:59,641 [trainer.py] => Trainable params: 466104
2023-05-22 20:31:59,648 [icarl.py] => Learning on 30-40
2023-05-22 21:35:22,460 [icarl.py] => Task 3, Epoch 170/170 => Loss 1.494, Train_accy 99.89
2023-05-22 21:35:22,461 [base.py] => Reducing exemplars...(50 per classes)
2023-05-22 21:35:36,634 [base.py] => Constructing exemplars...(50 per classes)
2023-05-22 21:35:48,986 [icarl.py] => Exemplar size: 2000
2023-05-22 21:35:48,987 [trainer.py] => CNN: {'total': 64.18, '00-09': 62.3, '10-19': 46.0, '20-29': 65.7, '30-39': 82.7, 'old': 58.0, 'new': 82.7}
2023-05-22 21:35:48,987 [trainer.py] => NME: {'total': 69.0, '00-09': 71.2, '10-19': 55.4, '20-29': 71.8, '30-39': 77.6, 'old': 66.13, 'new': 77.6}
2023-05-22 21:35:48,987 [trainer.py] => CNN top1 curve: [91.2, 79.6, 73.17, 64.18]
2023-05-22 21:35:48,987 [trainer.py] => CNN top5 curve: [99.3, 96.5, 94.27, 91.05]
2023-05-22 21:35:48,987 [trainer.py] => NME top1 curve: [91.6, 79.1, 75.4, 69.0]
2023-05-22 21:35:48,987 [trainer.py] => NME top5 curve: [99.8, 96.3, 94.27, 91.32]

2023-05-22 21:35:48,987 [trainer.py] => All params: 466754
2023-05-22 21:35:48,987 [trainer.py] => Trainable params: 466754
2023-05-22 21:35:48,991 [icarl.py] => Learning on 40-50
2023-05-22 23:03:43,196 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:03:43,196 [trainer.py] => prefix: reproduce
2023-05-22 23:03:43,196 [trainer.py] => dataset: cifar100
2023-05-22 23:03:43,196 [trainer.py] => memory_size: 2000
2023-05-22 23:03:43,196 [trainer.py] => memory_per_class: 20
2023-05-22 23:03:43,196 [trainer.py] => fixed_memory: False
2023-05-22 23:03:43,196 [trainer.py] => shuffle: True
2023-05-22 23:03:43,196 [trainer.py] => init_cls: 10
2023-05-22 23:03:43,196 [trainer.py] => increment: 10
2023-05-22 23:03:43,196 [trainer.py] => model_name: icarl
2023-05-22 23:03:43,196 [trainer.py] => convnet_type: resnet32
2023-05-22 23:03:43,197 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:03:43,197 [trainer.py] => seed: 1993
2023-05-22 23:03:44,415 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:03:44,523 [trainer.py] => All params: 464154
2023-05-22 23:03:44,524 [trainer.py] => Trainable params: 464154
2023-05-22 23:03:44,524 [icarl.py] => Learning on 0-10
2023-05-22 23:05:02,666 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:05:02,666 [trainer.py] => prefix: reproduce
2023-05-22 23:05:02,666 [trainer.py] => dataset: cifar100
2023-05-22 23:05:02,666 [trainer.py] => memory_size: 2000
2023-05-22 23:05:02,666 [trainer.py] => memory_per_class: 20
2023-05-22 23:05:02,666 [trainer.py] => fixed_memory: False
2023-05-22 23:05:02,666 [trainer.py] => shuffle: True
2023-05-22 23:05:02,666 [trainer.py] => init_cls: 10
2023-05-22 23:05:02,666 [trainer.py] => increment: 10
2023-05-22 23:05:02,666 [trainer.py] => model_name: icarl
2023-05-22 23:05:02,666 [trainer.py] => convnet_type: resnet32
2023-05-22 23:05:02,666 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:05:02,667 [trainer.py] => seed: 1993
2023-05-22 23:05:03,899 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:05:04,014 [trainer.py] => All params: 464154
2023-05-22 23:05:04,014 [trainer.py] => Trainable params: 464154
2023-05-22 23:05:04,014 [icarl.py] => Learning on 0-10
2023-05-22 23:10:46,408 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:10:46,408 [trainer.py] => prefix: reproduce
2023-05-22 23:10:46,408 [trainer.py] => dataset: cifar100
2023-05-22 23:10:46,408 [trainer.py] => memory_size: 2000
2023-05-22 23:10:46,408 [trainer.py] => memory_per_class: 20
2023-05-22 23:10:46,408 [trainer.py] => fixed_memory: False
2023-05-22 23:10:46,408 [trainer.py] => shuffle: True
2023-05-22 23:10:46,408 [trainer.py] => init_cls: 10
2023-05-22 23:10:46,408 [trainer.py] => increment: 10
2023-05-22 23:10:46,408 [trainer.py] => model_name: icarl
2023-05-22 23:10:46,409 [trainer.py] => convnet_type: resnet32
2023-05-22 23:10:46,409 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:10:46,409 [trainer.py] => seed: 1993
2023-05-22 23:10:47,635 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:10:47,750 [trainer.py] => All params: 464154
2023-05-22 23:10:47,751 [trainer.py] => Trainable params: 464154
2023-05-22 23:10:47,751 [icarl.py] => Learning on 0-10
2023-05-22 23:12:14,130 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:12:14,130 [trainer.py] => prefix: reproduce
2023-05-22 23:12:14,130 [trainer.py] => dataset: cifar100
2023-05-22 23:12:14,130 [trainer.py] => memory_size: 2000
2023-05-22 23:12:14,130 [trainer.py] => memory_per_class: 20
2023-05-22 23:12:14,130 [trainer.py] => fixed_memory: False
2023-05-22 23:12:14,130 [trainer.py] => shuffle: True
2023-05-22 23:12:14,130 [trainer.py] => init_cls: 10
2023-05-22 23:12:14,130 [trainer.py] => increment: 10
2023-05-22 23:12:14,130 [trainer.py] => model_name: icarl
2023-05-22 23:12:14,131 [trainer.py] => convnet_type: resnet32
2023-05-22 23:12:14,131 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:12:14,131 [trainer.py] => seed: 1993
2023-05-22 23:12:15,363 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:12:15,473 [trainer.py] => All params: 464154
2023-05-22 23:12:15,474 [trainer.py] => Trainable params: 464154
2023-05-22 23:12:15,474 [icarl.py] => Learning on 0-10
2023-05-22 23:13:36,958 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:13:36,958 [trainer.py] => prefix: reproduce
2023-05-22 23:13:36,958 [trainer.py] => dataset: cifar100
2023-05-22 23:13:36,958 [trainer.py] => memory_size: 2000
2023-05-22 23:13:36,958 [trainer.py] => memory_per_class: 20
2023-05-22 23:13:36,958 [trainer.py] => fixed_memory: False
2023-05-22 23:13:36,958 [trainer.py] => shuffle: True
2023-05-22 23:13:36,958 [trainer.py] => init_cls: 10
2023-05-22 23:13:36,958 [trainer.py] => increment: 10
2023-05-22 23:13:36,958 [trainer.py] => model_name: icarl
2023-05-22 23:13:36,958 [trainer.py] => convnet_type: resnet32
2023-05-22 23:13:36,958 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:13:36,958 [trainer.py] => seed: 1993
2023-05-22 23:13:38,185 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:13:38,297 [trainer.py] => All params: 464154
2023-05-22 23:13:38,298 [trainer.py] => Trainable params: 464154
2023-05-22 23:13:38,298 [icarl.py] => Learning on 0-10
2023-05-22 23:13:51,633 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:13:51,633 [trainer.py] => prefix: reproduce
2023-05-22 23:13:51,633 [trainer.py] => dataset: cifar100
2023-05-22 23:13:51,633 [trainer.py] => memory_size: 2000
2023-05-22 23:13:51,633 [trainer.py] => memory_per_class: 20
2023-05-22 23:13:51,633 [trainer.py] => fixed_memory: False
2023-05-22 23:13:51,633 [trainer.py] => shuffle: True
2023-05-22 23:13:51,633 [trainer.py] => init_cls: 10
2023-05-22 23:13:51,633 [trainer.py] => increment: 10
2023-05-22 23:13:51,633 [trainer.py] => model_name: icarl
2023-05-22 23:13:51,633 [trainer.py] => convnet_type: resnet32
2023-05-22 23:13:51,633 [trainer.py] => device: [device(type='cuda', index=0), device(type='cuda', index=1), device(type='cuda', index=2), device(type='cuda', index=3)]
2023-05-22 23:13:51,633 [trainer.py] => seed: 1993
2023-05-22 23:13:52,851 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:13:52,960 [trainer.py] => All params: 464154
2023-05-22 23:13:52,961 [trainer.py] => Trainable params: 464154
2023-05-22 23:13:52,961 [icarl.py] => Learning on 0-10
2023-05-22 23:17:00,922 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:17:00,922 [trainer.py] => prefix: reproduce
2023-05-22 23:17:00,922 [trainer.py] => dataset: cifar100
2023-05-22 23:17:00,922 [trainer.py] => memory_size: 2000
2023-05-22 23:17:00,922 [trainer.py] => memory_per_class: 20
2023-05-22 23:17:00,922 [trainer.py] => fixed_memory: False
2023-05-22 23:17:00,922 [trainer.py] => shuffle: True
2023-05-22 23:17:00,922 [trainer.py] => init_cls: 10
2023-05-22 23:17:00,922 [trainer.py] => increment: 10
2023-05-22 23:17:00,922 [trainer.py] => model_name: icarl
2023-05-22 23:17:00,922 [trainer.py] => convnet_type: resnet32
2023-05-22 23:17:00,922 [trainer.py] => device: [device(type='cuda', index=1)]
2023-05-22 23:17:00,922 [trainer.py] => seed: 1993
2023-05-22 23:17:02,143 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:17:02,258 [trainer.py] => All params: 464154
2023-05-22 23:17:02,259 [trainer.py] => Trainable params: 464154
2023-05-22 23:17:02,259 [icarl.py] => Learning on 0-10
2023-05-22 23:17:17,629 [trainer.py] => config: ./exps/icarl.json
2023-05-22 23:17:17,629 [trainer.py] => prefix: reproduce
2023-05-22 23:17:17,629 [trainer.py] => dataset: cifar100
2023-05-22 23:17:17,629 [trainer.py] => memory_size: 2000
2023-05-22 23:17:17,629 [trainer.py] => memory_per_class: 20
2023-05-22 23:17:17,629 [trainer.py] => fixed_memory: False
2023-05-22 23:17:17,629 [trainer.py] => shuffle: True
2023-05-22 23:17:17,629 [trainer.py] => init_cls: 10
2023-05-22 23:17:17,630 [trainer.py] => increment: 10
2023-05-22 23:17:17,630 [trainer.py] => model_name: icarl
2023-05-22 23:17:17,630 [trainer.py] => convnet_type: resnet32
2023-05-22 23:17:17,630 [trainer.py] => device: [device(type='cuda', index=1)]
2023-05-22 23:17:17,630 [trainer.py] => seed: 1993
2023-05-22 23:17:18,849 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-05-22 23:17:18,961 [trainer.py] => All params: 464154
2023-05-22 23:17:18,962 [trainer.py] => Trainable params: 464154
2023-05-22 23:17:18,962 [icarl.py] => Learning on 0-10
2023-05-22 23:22:28,264 [icarl.py] => Task 0, Epoch 200/200 => Loss 0.031, Train_accy 99.38
2023-05-22 23:22:28,264 [base.py] => Reducing exemplars...(200 per classes)
2023-05-22 23:22:28,264 [base.py] => Constructing exemplars...(200 per classes)
2023-05-22 23:22:36,462 [icarl.py] => Exemplar size: 2000
2023-05-22 23:22:36,463 [trainer.py] => CNN: {'total': 89.8, '00-09': 89.8, 'old': 0, 'new': 89.8}
2023-05-22 23:22:36,463 [trainer.py] => NME: {'total': 89.8, '00-09': 89.8, 'old': 0, 'new': 89.8}
2023-05-22 23:22:36,463 [trainer.py] => CNN top1 curve: [89.8]
2023-05-22 23:22:36,463 [trainer.py] => CNN top5 curve: [99.5]
2023-05-22 23:22:36,463 [trainer.py] => NME top1 curve: [89.8]
2023-05-22 23:22:36,463 [trainer.py] => NME top5 curve: [99.4]

2023-05-22 23:22:36,463 [trainer.py] => All params: 464804
2023-05-22 23:22:36,464 [trainer.py] => Trainable params: 464804
2023-05-22 23:22:36,464 [icarl.py] => Learning on 10-20
2023-05-22 23:29:08,574 [icarl.py] => Task 1, Epoch 170/170 => Loss 1.007, Train_accy 99.89
2023-05-22 23:29:08,574 [base.py] => Reducing exemplars...(100 per classes)
2023-05-22 23:29:11,659 [base.py] => Constructing exemplars...(100 per classes)
2023-05-22 23:29:19,328 [icarl.py] => Exemplar size: 2000
2023-05-22 23:29:19,328 [trainer.py] => CNN: {'total': 77.95, '00-09': 78.3, '10-19': 77.6, 'old': 78.3, 'new': 77.6}
2023-05-22 23:29:19,328 [trainer.py] => NME: {'total': 79.1, '00-09': 81.8, '10-19': 76.4, 'old': 81.8, 'new': 76.4}
2023-05-22 23:29:19,328 [trainer.py] => CNN top1 curve: [89.8, 77.95]
2023-05-22 23:29:19,328 [trainer.py] => CNN top5 curve: [99.5, 96.25]
2023-05-22 23:29:19,328 [trainer.py] => NME top1 curve: [89.8, 79.1]
2023-05-22 23:29:19,328 [trainer.py] => NME top5 curve: [99.4, 96.0]

2023-05-22 23:29:19,328 [trainer.py] => All params: 465454
2023-05-22 23:29:19,329 [trainer.py] => Trainable params: 465454
2023-05-22 23:29:19,329 [icarl.py] => Learning on 20-30
2023-05-22 23:35:51,984 [icarl.py] => Task 2, Epoch 170/170 => Loss 1.163, Train_accy 99.91
2023-05-22 23:35:51,984 [base.py] => Reducing exemplars...(66 per classes)
2023-05-22 23:35:58,094 [base.py] => Constructing exemplars...(66 per classes)
2023-05-22 23:36:06,453 [icarl.py] => Exemplar size: 1980
2023-05-22 23:36:06,453 [trainer.py] => CNN: {'total': 72.6, '00-09': 69.9, '10-19': 60.4, '20-29': 87.5, 'old': 65.15, 'new': 87.5}
2023-05-22 23:36:06,453 [trainer.py] => NME: {'total': 74.27, '00-09': 76.2, '10-19': 65.1, '20-29': 81.5, 'old': 70.65, 'new': 81.5}
2023-05-22 23:36:06,453 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6]
2023-05-22 23:36:06,454 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77]
2023-05-22 23:36:06,454 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27]
2023-05-22 23:36:06,454 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37]

2023-05-22 23:36:06,454 [trainer.py] => All params: 466104
2023-05-22 23:36:06,454 [trainer.py] => Trainable params: 466104
2023-05-22 23:36:06,455 [icarl.py] => Learning on 30-40
2023-05-22 23:42:45,877 [icarl.py] => Task 3, Epoch 170/170 => Loss 1.461, Train_accy 99.90
2023-05-22 23:42:45,877 [base.py] => Reducing exemplars...(50 per classes)
2023-05-22 23:42:53,898 [base.py] => Constructing exemplars...(50 per classes)
2023-05-22 23:43:01,767 [icarl.py] => Exemplar size: 2000
2023-05-22 23:43:01,767 [trainer.py] => CNN: {'total': 63.72, '00-09': 59.4, '10-19': 46.3, '20-29': 67.2, '30-39': 82.0, 'old': 57.63, 'new': 82.0}
2023-05-22 23:43:01,767 [trainer.py] => NME: {'total': 68.75, '00-09': 70.2, '10-19': 58.3, '20-29': 72.9, '30-39': 73.6, 'old': 67.13, 'new': 73.6}
2023-05-22 23:43:01,767 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72]
2023-05-22 23:43:01,767 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4]
2023-05-22 23:43:01,767 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75]
2023-05-22 23:43:01,767 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35]

2023-05-22 23:43:01,767 [trainer.py] => All params: 466754
2023-05-22 23:43:01,767 [trainer.py] => Trainable params: 466754
2023-05-22 23:43:01,768 [icarl.py] => Learning on 40-50
2023-05-22 23:49:38,147 [icarl.py] => Task 4, Epoch 170/170 => Loss 1.550, Train_accy 99.97
2023-05-22 23:49:38,148 [base.py] => Reducing exemplars...(40 per classes)
2023-05-22 23:49:48,599 [base.py] => Constructing exemplars...(40 per classes)
2023-05-22 23:49:56,533 [icarl.py] => Exemplar size: 2000
2023-05-22 23:49:56,534 [trainer.py] => CNN: {'total': 58.54, '00-09': 51.8, '10-19': 38.1, '20-29': 54.8, '30-39': 60.0, '40-49': 88.0, 'old': 51.18, 'new': 88.0}
2023-05-22 23:49:56,534 [trainer.py] => NME: {'total': 64.44, '00-09': 66.1, '10-19': 48.4, '20-29': 66.4, '30-39': 63.6, '40-49': 77.7, 'old': 61.12, 'new': 77.7}
2023-05-22 23:49:56,534 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54]
2023-05-22 23:49:56,535 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38]
2023-05-22 23:49:56,535 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44]
2023-05-22 23:49:56,535 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32]

2023-05-22 23:49:56,536 [trainer.py] => All params: 467404
2023-05-22 23:49:56,536 [trainer.py] => Trainable params: 467404
2023-05-22 23:49:56,537 [icarl.py] => Learning on 50-60
2023-05-22 23:56:42,554 [icarl.py] => Task 5, Epoch 170/170 => Loss 1.676, Train_accy 99.96
2023-05-22 23:56:42,555 [base.py] => Reducing exemplars...(33 per classes)
2023-05-22 23:56:55,700 [base.py] => Constructing exemplars...(33 per classes)
2023-05-22 23:57:03,836 [icarl.py] => Exemplar size: 1980
2023-05-22 23:57:03,837 [trainer.py] => CNN: {'total': 54.17, '00-09': 46.4, '10-19': 34.5, '20-29': 50.0, '30-39': 47.5, '40-49': 64.3, '50-59': 82.3, 'old': 48.54, 'new': 82.3}
2023-05-22 23:57:03,837 [trainer.py] => NME: {'total': 60.77, '00-09': 59.6, '10-19': 44.4, '20-29': 61.7, '30-39': 56.3, '40-49': 70.0, '50-59': 72.6, 'old': 58.4, 'new': 72.6}
2023-05-22 23:57:03,837 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17]
2023-05-22 23:57:03,837 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02]
2023-05-22 23:57:03,837 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77]
2023-05-22 23:57:03,837 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45]

2023-05-22 23:57:03,837 [trainer.py] => All params: 468054
2023-05-22 23:57:03,837 [trainer.py] => Trainable params: 468054
2023-05-22 23:57:03,838 [icarl.py] => Learning on 60-70
2023-05-23 00:03:44,924 [icarl.py] => Task 6, Epoch 170/170 => Loss 1.724, Train_accy 99.87
2023-05-23 00:03:44,924 [base.py] => Reducing exemplars...(28 per classes)
2023-05-23 00:04:00,661 [base.py] => Constructing exemplars...(28 per classes)
2023-05-23 00:04:08,875 [icarl.py] => Exemplar size: 1960
2023-05-23 00:04:08,875 [trainer.py] => CNN: {'total': 52.29, '00-09': 45.2, '10-19': 31.0, '20-29': 48.7, '30-39': 42.1, '40-49': 54.7, '50-59': 55.3, '60-69': 89.0, 'old': 46.17, 'new': 89.0}
2023-05-23 00:04:08,875 [trainer.py] => NME: {'total': 58.09, '00-09': 56.7, '10-19': 42.5, '20-29': 59.5, '30-39': 49.7, '40-49': 64.5, '50-59': 58.6, '60-69': 75.1, 'old': 55.25, 'new': 75.1}
2023-05-23 00:04:08,875 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17, 52.29]
2023-05-23 00:04:08,876 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02, 80.8]
2023-05-23 00:04:08,876 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77, 58.09]
2023-05-23 00:04:08,876 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45, 83.87]

2023-05-23 00:04:08,876 [trainer.py] => All params: 468704
2023-05-23 00:04:08,876 [trainer.py] => Trainable params: 468704
2023-05-23 00:04:08,877 [icarl.py] => Learning on 70-80
2023-05-23 00:10:58,431 [icarl.py] => Task 7, Epoch 170/170 => Loss 1.712, Train_accy 99.93
2023-05-23 00:10:58,432 [base.py] => Reducing exemplars...(25 per classes)
2023-05-23 00:11:16,920 [base.py] => Constructing exemplars...(25 per classes)
2023-05-23 00:11:25,218 [icarl.py] => Exemplar size: 2000
2023-05-23 00:11:25,218 [trainer.py] => CNN: {'total': 47.32, '00-09': 40.2, '10-19': 30.1, '20-29': 48.8, '30-39': 36.7, '40-49': 43.6, '50-59': 34.8, '60-69': 60.2, '70-79': 84.2, 'old': 42.06, 'new': 84.2}
2023-05-23 00:11:25,219 [trainer.py] => NME: {'total': 54.55, '00-09': 53.9, '10-19': 40.6, '20-29': 57.7, '30-39': 45.8, '40-49': 57.1, '50-59': 45.1, '60-69': 63.8, '70-79': 72.4, 'old': 52.0, 'new': 72.4}
2023-05-23 00:11:25,219 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17, 52.29, 47.32]
2023-05-23 00:11:25,219 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02, 80.8, 77.85]
2023-05-23 00:11:25,219 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77, 58.09, 54.55]
2023-05-23 00:11:25,219 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45, 83.87, 82.19]

2023-05-23 00:11:25,219 [trainer.py] => All params: 469354
2023-05-23 00:11:25,219 [trainer.py] => Trainable params: 469354
2023-05-23 00:11:25,219 [icarl.py] => Learning on 80-90
2023-05-23 00:18:11,022 [icarl.py] => Task 8, Epoch 170/170 => Loss 1.937, Train_accy 99.87
2023-05-23 00:18:11,023 [base.py] => Reducing exemplars...(22 per classes)
2023-05-23 00:18:32,082 [base.py] => Constructing exemplars...(22 per classes)
2023-05-23 00:18:40,388 [icarl.py] => Exemplar size: 1980
2023-05-23 00:18:40,388 [trainer.py] => CNN: {'total': 44.53, '00-09': 39.7, '10-19': 26.6, '20-29': 40.9, '30-39': 33.7, '40-49': 40.1, '50-59': 33.4, '60-69': 44.7, '70-79': 56.2, '80-89': 85.5, 'old': 39.41, 'new': 85.5}
2023-05-23 00:18:40,388 [trainer.py] => NME: {'total': 51.34, '00-09': 51.1, '10-19': 33.7, '20-29': 53.4, '30-39': 43.4, '40-49': 53.7, '50-59': 43.7, '60-69': 56.1, '70-79': 60.8, '80-89': 66.2, 'old': 49.49, 'new': 66.2}
2023-05-23 00:18:40,388 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17, 52.29, 47.32, 44.53]
2023-05-23 00:18:40,389 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02, 80.8, 77.85, 75.43]
2023-05-23 00:18:40,389 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77, 58.09, 54.55, 51.34]
2023-05-23 00:18:40,389 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45, 83.87, 82.19, 79.02]

2023-05-23 00:18:40,389 [trainer.py] => All params: 470004
2023-05-23 00:18:40,389 [trainer.py] => Trainable params: 470004
2023-05-23 00:18:40,390 [icarl.py] => Learning on 90-100
2023-05-23 00:25:32,625 [icarl.py] => Task 9, Epoch 170/170 => Loss 2.071, Train_accy 99.90
2023-05-23 00:25:32,625 [base.py] => Reducing exemplars...(20 per classes)
2023-05-23 00:25:57,312 [base.py] => Constructing exemplars...(20 per classes)
2023-05-23 00:26:06,251 [icarl.py] => Exemplar size: 2000
2023-05-23 00:26:06,251 [trainer.py] => CNN: {'total': 42.06, '00-09': 35.4, '10-19': 23.3, '20-29': 34.9, '30-39': 29.6, '40-49': 38.7, '50-59': 29.2, '60-69': 43.1, '70-79': 44.5, '80-89': 60.9, '90-99': 81.0, 'old': 37.73, 'new': 81.0}
2023-05-23 00:26:06,251 [trainer.py] => NME: {'total': 49.4, '00-09': 47.4, '10-19': 32.0, '20-29': 48.5, '30-39': 40.2, '40-49': 51.3, '50-59': 42.1, '60-69': 55.2, '70-79': 53.2, '80-89': 59.3, '90-99': 64.8, 'old': 47.69, 'new': 64.8}
2023-05-23 00:26:06,251 [trainer.py] => CNN top1 curve: [89.8, 77.95, 72.6, 63.72, 58.54, 54.17, 52.29, 47.32, 44.53, 42.06]
2023-05-23 00:26:06,251 [trainer.py] => CNN top5 curve: [99.5, 96.25, 93.77, 90.4, 87.38, 83.02, 80.8, 77.85, 75.43, 72.11]
2023-05-23 00:26:06,251 [trainer.py] => NME top1 curve: [89.8, 79.1, 74.27, 68.75, 64.44, 60.77, 58.09, 54.55, 51.34, 49.4]
2023-05-23 00:26:06,251 [trainer.py] => NME top5 curve: [99.4, 96.0, 94.37, 91.35, 89.32, 85.45, 83.87, 82.19, 79.02, 77.53]

2023-07-04 17:19:05,818 [trainer.py] => config: ./exps/icarl.json
2023-07-04 17:19:05,818 [trainer.py] => prefix: reproduce
2023-07-04 17:19:05,818 [trainer.py] => dataset: cifar100
2023-07-04 17:19:05,818 [trainer.py] => memory_size: 2000
2023-07-04 17:19:05,818 [trainer.py] => memory_per_class: 20
2023-07-04 17:19:05,818 [trainer.py] => fixed_memory: False
2023-07-04 17:19:05,818 [trainer.py] => shuffle: True
2023-07-04 17:19:05,818 [trainer.py] => init_cls: 10
2023-07-04 17:19:05,818 [trainer.py] => increment: 10
2023-07-04 17:19:05,818 [trainer.py] => model_name: icarl
2023-07-04 17:19:05,818 [trainer.py] => convnet_type: resnet32
2023-07-04 17:19:05,818 [trainer.py] => device: [device(type='cuda', index=1)]
2023-07-04 17:19:05,819 [trainer.py] => seed: 1993
2023-07-04 17:19:07,166 [data_manager.py] => [68, 56, 78, 8, 23, 84, 90, 65, 74, 76, 40, 89, 3, 92, 55, 9, 26, 80, 43, 38, 58, 70, 77, 1, 85, 19, 17, 50, 28, 53, 13, 81, 45, 82, 6, 59, 83, 16, 15, 44, 91, 41, 72, 60, 79, 52, 20, 10, 31, 54, 37, 95, 14, 71, 96, 98, 97, 2, 64, 66, 42, 22, 35, 86, 24, 34, 87, 21, 99, 0, 88, 27, 18, 94, 11, 12, 47, 25, 30, 46, 62, 69, 36, 61, 7, 63, 75, 5, 32, 4, 51, 48, 73, 93, 39, 67, 29, 49, 57, 33]
2023-07-04 17:19:07,273 [trainer.py] => All params: 464154
2023-07-04 17:19:07,273 [trainer.py] => Trainable params: 464154
2023-07-04 17:19:07,274 [icarl.py] => Learning on 0-10```
@nttung1110
Copy link
Author

nttung1110 commented Jul 4, 2023 •
edited
Loading

And here is the json config that I have used in my experiment:

{
    "prefix": "reproduce",
    "dataset": "cifar100",
    "memory_size": 2000,
    "memory_per_class": 20,
    "fixed_memory": false,
    "shuffle": true,
    "init_cls": 10,
    "increment": 10,
    "model_name": "icarl",
    "convnet_type": "resnet32",
    "device": ["1"],
    "seed": [1993]
}

@zhoudw-zdw
Copy link
Collaborator

zhoudw-zdw commented Jul 5, 2023 •
edited
Loading

I am not quite sure which result you would like to reproduce.

It seems your final NME accuracy is about 49.4, which is the same as we reported in figure b (https://github.com/G-U-N/PyCIL/blob/master/resources/cifar100.png) and figure 1(a) here (https://arxiv.org/pdf/2112.12533v2.pdf)

@nttung1110
Copy link
Author

Thanks for your reply. I was confused about NME and CNN and realized that NME is chosen for icarl instead of CNN. How about the following results of foster? I could not reproduce the result reported in the figure (The final CNN accuracy is about 55 but it should be higher than 60 based on your reported figures right?)

2023-05-23 01:18:52,588 [foster.py] => Exemplar size: 2000
2023-05-23 01:18:52,589 [trainer.py] => CNN: {'total': 54.55, '00-09': 48.8, '10-19': 33.5, '20-29': 51.3, '30-39': 46.8, '40-49': 59.0, '50-59': 52.7, '60-69': 66.9, '70-79': 60.5, '80-89': 64.0, '90-99': 62.0, 'old': 53.72, 'new': 62.0}
2023-05-23 01:18:52,589 [trainer.py] => NME: {'total': 48.13, '00-09': 46.9, '10-19': 32.9, '20-29': 49.5, '30-39': 42.9, '40-49': 52.0, '50-59': 42.4, '60-69': 50.2, '70-79': 43.1, '80-89': 45.6, '90-99': 75.8, 'old': 45.06, 'new': 75.8}
2023-05-23 01:18:52,589 [trainer.py] => CNN top1 curve: [90.8, 81.2, 76.27, 67.72, 63.86, 60.65, 59.81, 56.8, 55.58, 54.55]
2023-05-23 01:18:52,589 [trainer.py] => CNN top5 curve: [99.2, 95.75, 93.83, 90.25, 87.82, 85.35, 84.29, 83.19, 82.04, 81.68]
2023-05-23 01:18:52,589 [trainer.py] => NME top1 curve: [90.2, 82.35, 77.07, 68.8, 64.28, 59.82, 57.11, 51.94, 51.09, 48.13]
2023-05-23 01:18:52,589 [trainer.py] => NME top5 curve: [99.2, 96.85, 95.03, 91.02, 88.38, 85.8, 83.57, 81.66, 79.91, 76.7]

@nttung1110
Copy link
Author

And here is the experiment configuration of foster:

{
    "prefix": "cil",
    "dataset": "cifar100",
    "memory_size": 2000,
    "memory_per_class": 20,
    "fixed_memory": false,
    "shuffle": true,
    "init_cls": 10,
    "increment": 10,
    "model_name": "foster",
    "convnet_type": "resnet32",
    "device": ["3"],
    "seed": [1993],
    "beta1":0.96,
    "beta2":0.97,
    "oofc":"ft",
    "is_teacher_wa":false,
    "is_student_wa":false,
    "lambda_okd":1,
    "wa_value":1,
    "init_epochs": 200,
    "init_lr" : 0.1,
    "init_weight_decay" : 5e-4,
    "boosting_epochs" : 170,
    "compression_epochs" : 130,
    "lr" : 0.1,
    "batch_size" : 128,
    "weight_decay" : 5e-4,
    "num_workers" : 8,
    "T" : 2
}

@zhoudw-zdw
Copy link
Collaborator

https://github.com/G-U-N/PyCIL/blob/0011ca4658d779224f0a5d499063ba5356347d9e/models/foster.py#L13C1-L13C103

Check this line in foster. Data augmentation in the source repo is needed to reproduce foster.

@zhoudw-zdw zhoudw-zdw reopened this Jul 5, 2023
@nttung1110
Copy link
Author

Thanks. I'll check that. How about the following results of bic? Both NME and CNN can not reach the reported results in your paper. And there is something strange with performance on class 60-69 of CNN where 0.0 is returned

2023-05-23 01:51:11,906 [bic.py] => Exemplar size: 2000
2023-05-23 01:51:11,906 [trainer.py] => CNN: {'total': 39.06, '00-09': 53.9, '10-19': 36.5, '20-29': 47.1, '30-39': 39.9, '40-49': 51.6, '50-59': 42.1, '60-69': 0.0, '70-79': 11.5, '80-89': 46.4, '90-99': 61.6, 'old': 36.56, 'new': 61.6}
2023-05-23 01:51:11,906 [trainer.py] => NME: {'total': 46.11, '00-09': 49.6, '10-19': 32.3, '20-29': 50.6, '30-39': 38.5, '40-49': 50.4, '50-59': 38.4, '60-69': 46.0, '70-79': 45.5, '80-89': 52.6, '90-99': 57.2, 'old': 44.88, 'new': 57.2}
2023-05-23 01:51:11,906 [trainer.py] => CNN top1 curve: [86.5, 71.3, 68.4, 62.35, 61.42, 58.17, 48.27, 42.31, 40.74, 39.06]
2023-05-23 01:51:11,906 [trainer.py] => CNN top5 curve: [99.0, 94.25, 91.77, 88.98, 87.98, 86.15, 72.44, 70.03, 68.52, 66.57]
2023-05-23 01:51:11,906 [trainer.py] => NME top1 curve: [86.3, 71.6, 68.73, 62.7, 61.72, 57.62, 55.69, 50.46, 48.24, 46.11]
2023-05-23 01:51:11,906 [trainer.py] => NME top5 curve: [98.7, 94.0, 91.43, 88.88, 87.56, 84.62, 82.54, 79.0, 77.08, 75.35]

@nttung1110
Copy link
Author

Here is bic configuration:

{
    "prefix": "reproduce",
    "dataset": "cifar100",
    "memory_size": 2000,
    "memory_per_class": 20,
    "fixed_memory": false,
    "shuffle": true,
    "init_cls": 10,
    "increment": 10,
    "model_name": "bic",
    "convnet_type": "resnet32",
    "device": ["0","1","2","3"],
    "seed": [1993]
}

@zhoudw-zdw
Copy link
Collaborator

Not quite sure about the error. In another survey (https://github.com/zhoudw-zdw/CIL_Survey) we sucessfully reproduce the results of bic with the same code. We shall check it later.

@nttung1110
Copy link
Author

Thanks. That would be great. Do you think it is due to some unexpected errors while running on a single gpu? The only thing that I have modified when running experiment on your code is just single gpu training

@zhoudw-zdw
Copy link
Collaborator

It seems that the default number of epochs for bias tuning could be too high, which could result in poor performance for new classes. You can modify it to achieve better performance.

https://github.com/G-U-N/PyCIL/blob/0b99ae5c4ea34df1c913c15e1d366a9e718e7569/models/bic.py#L177C10-L177C10

@nttung1110
Copy link
Author

@zhoudw-zdw Then what is the exact configuration that you have used to reproduce results of bic in your paper? I would assume that the parameters (epoch, lr, ...) u have specified in training script of bic should be able to reproduce the approximated results. It would be great if u could clarify the correct configuration of reproduction so that other works could benefit from your project. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nttung1110 @zhoudw-zdw