Training speed of mixmo is very slow

[chenshen03]

I followed your instructions to train mixmo on cifar-100. However, the training is very slow compared to the baseline. Can you tell me how much time it takes to train mixmo on CIFAR-100 in your experiments?

$ python3 scripts/train.py --config_path config/cifar100/exp_cifar100_wrn2810-2_cutmixmo-p5_msdacutmix_bar4.yaml --dataplace $dataplace --saveplace $saveplace

[alexrame]

Thank you for your interest.
Indeed MixMo performs best with batch repetition (first introduced in MIMO https://arxiv.org/pdf/2010.06610.pdf).
This option file has batch repetition set to 4, therefore each image is duplicated 4 times in a batch: overall, each epoch is 4 times longer.
Note that we compare different approaches at equal training time in Section 4.3.1 from our paper (https://arxiv.org/pdf/2103.06132.pdf) and that most concurrent approaches also lengthen training.
I hope this answers your question.

[chenshen03]

Thank you very much.
Becides, I ran the experiment exp_cifar100_wrn2810-2_cutmixmo-p5_msdacutmix_bar4.yaml for several days and the best accuracy in epoch 158 is currently 92.03%, which still well below the 95.77% reported in the paper. Can you share your training log? Thank you very much!

########################################
epoch                       158
general/classifier_lr       0.00125
general/l2_reg              3.795e+03
test/accuracy               82.03%
test/accuracy_0             78.45%
test/accuracy_1             79.68%
test/accuracy_mean          79.06%
test/accuracytop5           96.26%
test/auc                    0.894054
test/brier                  0.00250124
test/ce0                    0.79515
test/ce1                    0.73879
test/diversity_ratioerrors  1.0451
test/ece                    0.0234672
test/main                   2.6724
test/nll                    0.626094
train/ce0                   1.2476
train/ce1                   1.2436
train/main                  3.6286
########################################

[alexrame]

That's totally normal to have around 82% at epoch 158. Accuracies are sharply increasing at each learning rate update,
occurring at epochs100, 200, and 225.
Here I provide the logs for the best performing run that reaches 85.92 at epoch 273.

epoch,test/accuracy,test/accuracytop5,test/accuracy_mean,test/accuracy_0,test/accuracy_1,train/accuracy,checkpoint_saved
1,9.12,29.13,08.745,07.98,09.51,4.31,0.0
2,14.41,38.89,12.88,11.61,14.15,8.94,0.0
3,22.47,53.19,20.795,20.15,21.44,13.14,0.0
4,30.08,62.93,28.05,25.87,30.23,16.60,0.0
5,32.88,65.62,30.535,30.46,30.61,20.15,0.0
6,36.93,72.25,35.065,35.31,34.82,22.66,0.0
7,40.23,73.99,36.46,36.25,36.67,25.23,0.0
8,43.39,76.11,37.76,40.01,35.51,27.41,0.0
9,48.08,80.32,42.775,43.02,42.53,29.27,0.0
10,50.56,81.63,46.36,45.85,46.87,31.07,0.0
11,50.11,82.21,46.095,44.2,47.99,32.32,0.0
12,51.43,83.04,47.915,48.01,47.82,33.76,0.0
13,54.25,84.54,49.935,51.5,48.37,34.48,0.0
14,51.91,83.49,47.755,48.38,47.13,36.10,0.0
15,53.29,85.46,50.115,50.48,49.75,36.75,1.0
16,54.01,84.61,50.2,49.63,50.77,37.57,0.0
17,56.67,86.04,52.26,52.88,51.64,38.27,0.0
18,60.02,88.17,56.23,54.39,58.07,38.55,0.0
19,56.17,85.12,51.53,53.79,49.27,39.47,0.0
20,59.80,86.84,55.51,54.68,56.34,39.65,0.0
21,59.96,87.79,54.535,54.8,54.27,40.29,0.0
22,58.71,86.23,54.92,53.85,55.99,40.80,0.0
23,59.92,87.97,55.815,54.91,56.72,41.30,0.0
24,61.14,88.59,57.5,57.41,57.59,41.55,0.0
25,61.60,88.22,57.095,57.09,57.1,41.62,0.0
26,62.69,89.23,58.6,58.31,58.89,42.15,0.0
27,62.95,89.73,58.555,56.36,60.75,42.34,0.0
28,59.57,88.20,56.06,55.41,56.71,42.54,0.0
29,62.29,89.05,58.57,59.04,58.1,42.93,0.0
30,63.07,89.51,58.72,58.66,58.78,43.24,1.0
31,60.87,87.71,57.31,58.11,56.51,43.56,0.0
32,61.63,88.58,56.3,54.07,58.53,43.69,0.0
33,63.81,89.75,60.41,60.81,60.01,43.84,0.0
34,63.12,89.11,58.34,58.17,58.51,43.94,0.0
35,59.36,87.13,56.31,55.63,56.99,43.73,0.0
36,63.57,89.58,58.84,57.69,59.99,44.07,0.0
37,62.90,89.27,59.145,59.6,58.69,44.41,0.0
38,62.71,89.36,57.17,59.56,54.78,44.62,0.0
39,64.47,90.19,60.32,61.34,59.3,44.68,0.0
40,65.94,90.85,61.4,61.05,61.75,45.03,0.0
41,65.18,90.25,61.01,61.59,60.43,45.01,0.0
42,63.57,89.94,59.46,60.81,58.11,45.00,0.0
43,63.28,89.62,60.365,60.83,59.9,44.95,0.0
44,62.12,87.90,57.55,58.87,56.23,44.95,0.0
45,65.06,90.75,60.695,60.25,61.14,45.30,1.0
46,64.92,90.48,59.815,58.62,61.01,45.47,0.0
47,65.55,90.78,60.375,60.0,60.75,45.32,0.0
48,63.47,89.87,58.75,57.22,60.28,45.93,0.0
49,65.52,89.60,60.77,58.25,63.29,45.74,0.0
50,64.31,90.25,60.585,61.49,59.68,45.85,0.0
51,65.29,90.28,60.59,60.02,61.16,45.69,0.0
52,64.80,90.35,61.17,61.86,60.48,45.92,0.0
53,65.67,90.53,61.335,60.55,62.12,46.08,0.0
54,67.10,91.29,61.76,61.71,61.81,46.44,0.0
55,64.39,89.20,58.44,54.73,62.15,46.29,0.0
56,66.71,91.11,62.78,61.25,64.31,46.27,0.0
57,66.45,91.44,62.445,61.48,63.41,46.40,0.0
58,67.12,91.62,62.8,62.97,62.63,46.48,0.0
59,64.23,90.77,60.3,58.66,61.94,46.81,0.0
60,67.20,91.39,62.865,63.48,62.25,46.37,1.0
61,67.76,91.96,63.825,63.22,64.43,46.84,0.0
62,66.24,91.60,62.14,61.06,63.22,46.60,0.0
63,66.46,90.63,62.955,63.17,62.74,46.47,0.0
64,66.90,90.73,62.9,63.14,62.66,46.67,0.0
65,66.80,91.07,62.975,64.3,61.65,47.34,0.0
66,68.03,91.86,63.615,65.14,62.09,46.61,0.0
67,68.01,91.83,63.955,63.9,64.01,47.01,0.0
68,66.04,90.56,61.555,61.94,61.17,47.20,0.0
69,67.59,91.01,63.335,62.71,63.96,47.09,0.0
70,66.31,90.50,62.125,61.66,62.59,47.23,0.0
71,66.92,90.88,62.605,62.62,62.59,47.14,0.0
72,64.25,90.50,60.645,59.39,61.9,47.41,0.0
73,68.19,91.65,63.84,63.36,64.32,47.01,0.0
74,67.04,91.30,62.985,62.8,63.17,47.32,0.0
75,67.46,90.59,62.07,63.72,60.42,47.46,1.0
76,66.76,91.30,62.815,64.02,61.61,47.22,0.0
77,67.92,92.29,63.555,63.1,64.01,47.06,0.0
78,66.89,91.07,62.075,62.06,62.09,47.62,0.0
79,68.39,91.31,63.635,63.76,63.51,47.65,0.0
80,66.62,91.13,61.745,62.24,61.25,47.64,0.0
81,65.99,90.09,60.955,58.77,63.14,47.72,0.0
82,67.42,91.22,61.89,62.42,61.36,47.77,0.0
83,68.13,91.94,63.915,64.06,63.77,47.84,0.0
84,66.89,91.12,62.45,60.83,64.07,47.55,0.0
85,67.56,91.15,63.235,64.93,61.54,47.84,0.0
86,66.99,91.12,62.405,62.2,62.61,47.69,0.0
87,68.81,92.12,63.775,64.77,62.78,47.84,0.0
88,64.46,89.98,60.025,61.68,58.37,47.96,0.0
89,66.35,90.81,62.27,63.25,61.29,47.78,0.0
90,65.25,90.77,60.445,57.25,63.64,48.03,1.0
91,66.90,90.87,62.23,64.34,60.12,47.82,0.0
92,65.18,90.52,61.,003.4,61.66,47.63,0.0
93,65.92,90.82,62.07,62.39,61.75,47.99,0.0
94,68.22,92.07,63.385,62.38,64.39,47.63,0.0
95,66.10,91.15,61.32,60.5,62.14,47.82,0.0
96,67.87,90.67,64.12,63.29,64.95,47.86,0.0
97,67.17,91.33,62.935,63.97,61.9,47.95,0.0
98,67.26,90.36,62.54,63.06,62.02,48.13,0.0
99,68.97,91.59,64.42,65.62,63.22,47.75,0.0
100,65.10,90.20,61.74,62.75,60.73,47.99,0.0
101,79.10,95.88,77.37,77.18,77.56,59.90,0.0
102,80.07,96.53,78.71,78.65,78.77,62.74,0.0
103,80.65,96.46,79.065,79.24,78.89,63.99,0.0
104,80.36,96.54,78.915,78.92,78.91,64.84,0.0
105,81.46,96.67,79.79,79.76,79.82,65.56,1.0
106,81.56,96.81,79.84,79.93,79.75,66.12,0.0
107,81.91,96.87,80.275,80.57,79.98,66.70,0.0
108,81.85,96.96,80.14,80.22,80.06,67.08,0.0
109,81.72,96.90,79.995,80.24,79.75,67.52,0.0
110,81.77,96.86,80.145,79.89,80.4,68.30,0.0
111,82.15,96.87,80.19,80.19,80.19,68.23,0.0
112,81.83,96.65,80.125,80.38,79.87,68.30,0.0
113,82.52,96.74,80.215,80.28,80.15,68.83,0.0
114,82.15,96.70,80.025,79.93,80.12,69.03,0.0
115,81.81,96.64,80.11,80.07,80.15,68.96,0.0
116,82.30,96.88,80.22,80.35,80.09,68.90,0.0
117,82.43,96.92,80.245,80.25,80.24,69.78,0.0
118,82.28,96.74,80.37,80.82,79.92,69.48,0.0
119,82.53,96.90,80.45,80.68,80.22,70.12,0.0
120,81.88,96.67,79.66,79.7,79.62,69.69,1.0
121,81.87,96.61,79.45,79.27,79.63,70.21,0.0
122,81.77,96.69,79.65,79.59,79.71,70.05,0.0
123,82.21,96.79,80.05,80.22,79.88,69.80,0.0
124,81.77,96.66,78.98,78.74,79.22,70.16,0.0
125,81.86,96.86,79.505,79.55,79.46,70.79,0.0
126,82.47,96.66,80.185,79.97,80.4,70.25,0.0
127,81.86,96.64,79.505,79.64,79.37,70.34,0.0
128,81.86,96.54,79.475,79.4,79.55,70.59,0.0
129,81.95,96.70,79.405,79.16,79.65,70.46,0.0
130,82.17,96.67,79.695,79.6,79.79,70.89,0.0
131,81.63,96.66,79.05,79.31,78.79,70.86,0.0
132,81.92,96.70,78.89,78.99,78.79,71.00,0.0
133,82.31,96.56,79.725,79.69,79.76,70.70,0.0
134,82.05,96.47,79.51,79.38,79.64,70.63,0.0
135,81.57,96.39,79.23,79.13,79.33,71.36,1.0
136,82.13,96.77,79.425,79.7,79.15,71.25,0.0
137,82.48,96.62,79.685,79.51,79.86,71.21,0.0
138,82.16,96.73,79.57,80.11,79.03,71.26,0.0
139,81.66,96.24,79.145,79.18,79.11,71.55,0.0
140,82.66,96.79,80.075,80.39,79.76,71.48,0.0
141,81.95,96.40,79.63,79.27,79.99,71.43,0.0
142,82.46,96.63,79.73,79.62,79.84,71.43,0.0
143,82.03,96.35,79.08,79.2,78.96,71.58,0.0
144,81.90,96.72,79.265,79.59,78.94,71.70,0.0
145,81.24,96.44,78.32,78.97,77.67,71.83,0.0
146,81.65,96.20,78.85,78.74,78.96,71.74,0.0
147,82.18,96.61,79.46,79.52,79.4,71.76,0.0
148,81.85,96.52,78.97,79.22,78.72,71.78,0.0
149,82.49,96.58,79.605,79.77,79.44,71.87,0.0
150,82.17,96.55,79.58,79.67,79.49,71.74,1.0
151,82.69,96.58,80.02,80.05,79.99,71.94,0.0
152,81.93,96.34,79.11,79.1,79.12,72.43,0.0
153,82.00,96.42,78.96,78.61,79.31,72.34,0.0
154,82.15,96.67,79.64,79.29,79.99,72.38,0.0
155,81.31,96.62,78.72,78.84,78.6,72.67,0.0
156,81.94,96.38,79.21,79.31,79.11,71.97,0.0
157,81.84,96.47,79.11,78.79,79.43,72.51,0.0
158,82.10,96.53,79.455,79.74,79.17,72.15,0.0
159,81.86,96.46,79.315,79.46,79.17,72.35,0.0
160,81.93,96.56,79.055,79.02,79.09,72.78,0.0
161,81.20,96.13,78.81,79.29,78.33,72.61,0.0
162,81.29,96.27,78.61,78.16,79.06,72.56,0.0
163,81.58,96.21,79.025,78.64,79.41,72.74,0.0
164,81.54,96.16,78.86,79.32,78.4,72.74,0.0
165,81.84,96.43,79.11,79.27,78.95,72.97,1.0
166,81.52,96.38,78.835,79.02,78.65,72.82,0.0
167,82.19,96.43,79.34,79.43,79.25,72.65,0.0
168,82.26,96.38,79.35,79.43,79.27,72.71,0.0
169,81.19,96.34,78.175,78.6,77.75,73.10,0.0
170,81.81,96.49,78.815,79.07,78.56,72.92,0.0
171,82.41,96.63,79.765,79.74,79.79,73.05,0.0
172,82.01,96.27,78.76,78.05,79.47,73.07,0.0
173,81.75,96.48,78.935,79.17,78.7,72.68,0.0
174,81.72,96.45,79.115,79.59,78.64,73.22,0.0
175,81.91,96.48,79.4,79.42,79.38,73.72,0.0
176,82.01,96.69,79.275,79.32,79.23,73.40,0.0
177,81.98,96.43,79.035,79.18,78.89,73.49,0.0
178,82.47,96.44,79.79,79.39,80.19,73.02,0.0
179,82.16,96.25,79.265,79.67,78.86,73.22,0.0
180,82.11,96.50,79.58,79.83,79.33,73.54,1.0
181,81.97,96.41,78.85,78.67,79.03,73.73,0.0
182,81.57,96.27,79.01,78.77,79.25,73.01,0.0
183,81.80,96.62,78.865,78.56,79.17,73.48,0.0
184,81.53,96.51,78.83,78.73,78.93,73.27,0.0
185,81.94,96.28,79.1,79.12,79.08,73.53,0.0
186,83.01,96.38,80.245,80.25,80.24,73.49,1.0
187,80.86,96.06,77.705,78.39,77.02,73.53,0.0
187,82.41,96.41,79.715,79.84,79.59,73.93,1.0
188,81.76,96.54,78.81,78.85,78.77,73.76,0.0
189,82.39,96.48,79.405,79.34,79.47,73.36,0.0
190,81.75,96.25,78.745,78.77,78.72,73.97,0.0
191,81.96,96.33,78.87,79.16,78.58,73.59,0.0
192,81.43,96.29,78.305,77.56,79.05,73.63,0.0
193,81.79,96.18,78.985,79.09,78.88,73.25,0.0
194,82.15,96.44,79.19,79.74,78.64,74.06,0.0
195,81.60,96.30,78.74,79.11,78.37,73.79,1.0
196,82.20,96.29,79.06,79.,091.2,74.03,0.0
197,81.87,96.30,79.15,79.47,78.83,74.02,0.0
198,82.25,96.34,79.035,79.26,78.81,73.86,0.0
199,82.12,96.60,79.22,78.77,79.67,73.79,0.0
200,81.95,96.43,79.365,79.55,79.18,73.37,0.0
201,84.75,97.04,82.79,82.89,82.69,78.14,1.0
202,84.89,97.06,82.975,83.3,82.65,79.53,1.0
203,85.18,97.18,83.095,83.34,82.85,79.61,1.0
204,85.15,97.23,83.25,83.68,82.82,80.14,0.0
205,85.01,97.26,83.23,83.32,83.14,80.38,0.0
206,85.11,97.30,83.245,83.46,83.03,80.88,0.0
207,85.01,97.24,83.42,83.87,82.97,80.95,0.0
208,85.09,97.13,83.49,83.62,83.36,80.67,0.0
209,85.36,97.29,83.555,83.84,83.27,80.99,1.0
210,85.30,97.33,83.495,83.64,83.35,80.75,1.0
211,85.34,97.32,83.4,83.57,83.23,81.07,0.0
212,85.33,97.30,83.52,83.67,83.37,81.07,0.0
213,85.38,97.26,83.68,83.8,83.56,81.46,1.0
214,85.56,97.35,83.715,83.84,83.59,81.52,1.0
215,85.49,97.27,83.73,83.86,83.6,81.49,0.0
216,85.41,97.33,83.635,83.79,83.48,81.62,0.0
217,85.42,97.41,83.54,83.58,83.5,81.49,0.0
218,85.34,97.28,83.255,83.38,83.13,81.64,0.0
219,85.51,97.19,83.8,84.01,83.59,81.95,0.0
220,85.34,97.28,83.675,83.91,83.44,81.84,0.0
221,85.43,97.32,83.8,84.03,83.57,82.01,0.0
222,85.31,97.42,83.51,83.79,83.23,81.56,0.0
223,85.42,97.35,83.62,83.83,83.41,81.70,0.0
224,85.32,97.28,83.48,83.67,83.29,81.99,0.0
225,85.52,97.25,83.8,83.83,83.77,82.13,1.0
226,85.60,97.25,83.62,83.86,83.38,82.58,1.0
227,85.51,97.28,83.95,84.18,83.72,82.29,0.0
228,85.60,97.40,83.875,84.12,83.63,82.64,1.0
229,85.72,97.40,83.955,84.09,83.82,82.52,1.0
230,85.67,97.38,83.89,84.12,83.66,82.51,0.0
231,85.52,97.30,83.65,83.81,83.49,82.64,0.0
232,85.76,97.48,84.02,84.25,83.79,82.79,1.0
233,85.39,97.40,83.8,83.97,83.63,82.57,0.0
234,85.57,97.38,83.835,84.07,83.6,82.47,0.0
235,85.48,97.26,83.89,84.21,83.57,82.55,0.0
236,85.73,97.29,83.825,83.97,83.68,83.15,0.0
237,85.65,97.38,83.735,83.95,83.52,82.77,0.0
238,85.69,97.35,84.065,84.24,83.89,82.48,0.0
239,85.78,97.36,83.835,84.01,83.66,82.64,1.0
240,85.65,97.44,83.95,84.2,83.7,82.99,1.0
241,85.50,97.34,83.68,83.86,83.5,82.71,0.0
242,85.58,97.32,83.95,84.17,83.73,82.77,0.0
243,85.62,97.37,83.92,84.06,83.78,82.60,0.0
244,85.79,97.38,84.02,84.17,83.87,82.73,1.0
245,85.75,97.30,83.945,84.17,83.72,82.94,0.0
246,85.69,97.40,84.04,84.19,83.89,82.47,0.0
247,85.75,97.33,84.04,84.33,83.75,82.55,0.0
248,85.72,97.34,83.995,84.28,83.71,82.56,0.0
249,85.56,97.31,84.075,84.37,83.78,82.73,0.0
250,85.62,97.22,83.7,83.84,83.56,82.67,0.0
251,85.70,97.34,84.0,84.18,83.82,82.64,0.0
252,85.53,97.32,83.88,84.13,83.63,82.75,0.0
253,85.60,97.42,83.935,84.11,83.76,83.00,0.0
254,85.67,97.32,83.98,84.18,83.78,82.69,0.0
255,85.60,97.30,83.83,84.03,83.63,82.57,1.0
256,85.64,97.35,83.97,84.21,83.73,82.74,0.0
257,85.73,97.31,83.85,84.15,83.55,83.08,0.0
258,85.69,97.33,83.88,84.2,83.56,82.91,0.0
259,85.62,97.37,83.925,84.12,83.73,82.86,0.0
260,85.61,97.32,83.88,84.2,83.56,82.84,0.0
261,85.70,97.30,83.945,84.2,83.69,82.87,0.0
262,85.82,97.44,84.12,84.31,83.93,82.92,1.0
263,85.72,97.26,83.82,84.09,83.55,82.46,0.0
264,85.70,97.40,84.07,84.29,83.85,83.01,0.0
265,85.58,97.33,83.86,84.19,83.53,83.12,0.0
266,85.65,97.31,83.89,84.04,83.74,83.01,0.0
267,85.63,97.30,83.69,83.86,83.52,82.85,0.0
268,85.69,97.39,84.08,84.38,83.78,82.85,0.0
269,85.63,97.39,84.1,84.33,83.87,82.92,0.0
270,85.62,97.39,83.91,84.07,83.75,82.74,1.0
271,85.69,97.37,83.785,84.08,83.49,83.20,0.0
272,85.62,97.42,83.91,84.14,83.68,82.59,0.0
273,85.92,97.36,84.085,84.29,83.88,82.97,1.0
274,85.66,97.37,83.77,84.06,83.48,83.13,0.0
275,85.75,97.38,84.085,84.23,83.94,82.84,0.0
276,85.89,97.35,83.85,84.14,83.56,83.18,0.0
277,85.80,97.39,84.01,84.11,83.91,83.33,0.0
278,85.70,97.33,84.025,84.3,83.75,83.52,0.0
279,85.71,97.33,84.1,84.39,83.81,83.86,0.0
280,85.79,97.27,84.0,84.3,83.7,84.30,0.0
281,85.82,97.37,84.03,84.28,83.78,84.29,0.0
282,85.85,97.31,84.13,84.43,83.83,84.52,0.0
283,85.76,97.38,84.09,84.33,83.85,85.12,0.0
284,85.76,97.34,84.055,84.43,83.68,85.20,0.0
285,85.71,97.30,84.02,84.25,83.79,85.27,1.0
286,85.76,97.30,84.01,84.25,83.77,85.75,0.0
287,85.74,97.27,84.075,84.28,83.87,86.15,0.0
288,85.83,97.24,84.045,84.22,83.87,85.92,0.0
289,85.71,97.40,84.205,84.28,84.13,86.27,0.0
290,85.78,97.31,84.165,84.39,83.94,86.78,0.0
291,85.78,97.31,84.05,84.29,83.81,86.95,0.0
292,85.62,97.28,83.96,84.18,83.74,87.18,0.0
293,85.59,97.34,83.97,84.14,83.8,87.49,0.0
294,85.59,97.33,84.,041.3,83.87,87.65,0.0
295,85.55,97.30,83.84,83.99,83.69,87.79,0.0
296,85.63,97.35,84.05,84.18,83.92,88.07,0.0
297,85.77,97.35,83.89,84.16,83.62,88.40,0.0
298,85.67,97.38,83.96,84.05,83.87,88.54,0.0
299,85.61,97.26,83.9,84.05,83.75,88.85,0.0
300,85.63,97.25,83.825,83.98,83.67,89.03,1.0