Accuracy doesn't increase

[GRGargallo]

Hi,
I run tf_cnn_benchmark and my accuracy doesn't grow up. I don't know if I do it wrong horovod's settings for CPU or a mistake in the way of using vgg16 model. Is there something to change? Is normal than accuracy doesn't change after 1 epoch?

Specs of the nodes:
CPU: Cavium ThunderX (ARM)
Number cores per socket: 48
Number threads per socket: 96
Number socket per MOBO: 2
Number of nodes: 2
RAM: 128GB
OS: 14.04.5 LTS

Command using slurm as resource management:

srun  --wait=999999 -K --cpu_bind=verbose python tf_cnn_benchmarks.py \
--variable_update=horovod --device=cpu --data_dir=/home/gramirez/imagenet --num_inter_threads=1 \
--num_intra_threads=48 --model=vgg16 --data_name=imagenet --nodistortions --allow_growth=True \
--optimizer=momentum --data_format=NHWC  --num_batches=1360 --num_warmup_batches=40 \
--batch_size=252 --print_training_accuracy=true --forward_only=true

Output from first MPI rank:

TensorFlow:  1.9
Model:       vgg16
Dataset:     imagenet
Mode:        forward-only
SingleSess:  False
Batch size:  1008 global
             252.0 per device
Num batches: 1360
Num epochs:  1.07
Devices:     ['horovod/cpu:0', 'horovod/cpu:1', 'horovod/cpu:2', 'horovod/cpu:3']
Data format: NHWC
Layout optimizer: False
Optimizer:   momentum
Variables:   horovod
==========
Generating model
Running warm up
Done warm up
Step    Img/sec total_loss      top_1_accuracy  top_5_accuracy
1       images/sec: 4.3 +/- 0.0 (jitter = 0.0)  0.000   0.000   0.004
10      images/sec: 4.2 +/- 0.0 (jitter = 0.0)  0.000   0.000   0.004
.
.
.
1340    images/sec: 4.3 +/- 0.0 (jitter = 0.0)  0.000   0.004   0.008
1350    images/sec: 4.3 +/- 0.0 (jitter = 0.0)  0.000   0.000   0.000
1360    images/sec: 4.2 +/- 0.0 (jitter = 0.0)  0.000   0.000   0.000
----------------------------------------------------------------
total images/sec: 17.00
----------------------------------------------------------------

[reedwm]

We haven't tested vgg convergence. And since we haven't tested it, it almost certainly does not converge correctly. We only tested resnet50. Can you try that?

[tfboyd]

You are running forward-only. There is no training happening.

vgg16 will train but I do not test that it trains to the desired accuracy. I would test with resnet50 which we know trains to 75.7 to 76% FP16 and FP32. It is also tested often.

--forward_only=true <-- remove this arg

[GRGargallo]

@tfboyd, I will try without this arg and post my result to help other people with the same problem.

@reedwm I only interested in vgg16 because I need to improve accuracy for ISC. In future, I will try resnet50 but now I don't have time.

Thanks to both of you

[GRGargallo]

@tfboyd I had tried to train all night without the forward-only flag. But gives me the same result. Please, can you tell me other think to help model converge?

thanks

Specs of the nodes:
CPU: Cavium ThunderX (ARM)
Number cores per socket: 48
Number threads per socket: 96
Number socket per MOBO: 2
Number of nodes: 3
RAM: 128GB
OS: 14.04.5 LTS

Command using slurm as resource management:

srun  --wait=999999 -K --cpu_bind=verbose python tf_cnn_benchmarks.py \
 --variable_update=horovod --device=cpu --data_dir=/home/gramirez/imagenet \
--num_inter_threads=1 --num_intra_threads=6 --model=vgg16 --data_name=imagenet\
 --nodistortions --allow_growth=True --optimizer=momentum --data_format=NHWC  \
--num_batches=320 --num_warmup_batches=20 --batch_size=16 --print_training_accuracy=true

Output from first MPI rank:

TensorFlow:  1.9
Model:       vgg16
Dataset:     imagenet
Mode:        training
SingleSess:  False
Batch size:  768 global
             16.0 per device
Num batches: 320
Num epochs:  0.19
Devices:     ['horovod/cpu:0', 'horovod/cpu:1', 'horovod/cpu:2', 'horovod/cpu:3', 'horovod/cpu:4', 'horovod/cpu:5', 'horovod/cpu:6', 'horovod/cpu:7', 'horovod/cpu:8', 'horovod/cpu:9', 'horovod/cpu:10', 'horovod/cpu:11', 'horovod/cpu:12', 'horovod/cpu:13', 'horovod/cpu:14', 'horovod/cpu:15', 'horovod/cpu:16', 'horovod/cpu:17', 'horovod/cpu:18', 'horovod/cpu:19', 'horovod/cpu:20', 'horovod/cpu:21', 'horovod/cpu:22', 'horovod/cpu:23', 'horovod/cpu:24', 'horovod/cpu:25', 'horovod/cpu:26', 'horovod/cpu:27', 'horovod/cpu:28', 'horovod/cpu:29', 'horovod/cpu:30', 'horovod/cpu:31', 'horovod/cpu:32', 'horovod/cpu:33', 'horovod/cpu:34', 'horovod/cpu:35', 'horovod/cpu:36', 'horovod/cpu:37', 'horovod/cpu:38', 'horovod/cpu:39', 'horovod/cpu:40', 'horovod/cpu:41', 'horovmages/sec: 0.1 +/- 0.0 (jitter = 0.0)     7.237   0.000   0.000
130     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.258   0.000   0.000
140     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.237   0.000   0.062
150     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.239   0.000   0.000
160     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.254   0.000   0.000
170     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.241   0.000   0.000
180     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.226   0.000   0.000
190     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.258   0.000   0.000
200     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.232   0.000   0.000
210     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.194   0.000   0.125
220     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.246   0.000   0.000
230     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.250   0.000   0.000
240     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.234   0.000   0.062
250     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.232   0.000   0.000
260     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.223   0.000   0.000
270     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.222   0.000   0.000
280     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.231   0.000   0.000
290     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.245   0.000   0.000
300     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.230   0.000   0.000
310     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.190   0.000   0.000
320     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.213   0.000   0.000
----------------------------------------------------------------
total images/sec: 5.52
----------------------------------------------------------------

Output from second MPI rank:

TensorFlow:  1.9
Model:       vgg16
Dataset:     imagenet
Mode:        training
SingleSess:  False
Batch size:  768 global
             16.0 per device
Num batches: 320
Num epochs:  0.19
Devices:     ['horovod/cpu:0', 'horovod/cpu:1', 'horovod/cpu:2', 'horovod/cpu:3', 'horovod/cpu:4', 'horovod/cpu:5', 'horovod/cpu:6', 'horovod/cpu:7', 'horovod/cpu:8', 'horovod/cpu:9', 'horovod/cpu:10', 'horovod/cpu:11', 'horovod/cpu:12', 'horovod/cpu:13', 'horovod/cpu:14', 'horovod/cpu:15', 'horovod/cpu:16', 'horovod/cpu:17', 'horovod/cpu:18', 'horovod/cpu:19', 'horovod/cpu:20', 'horovod/cpu:21', 'horovod/cpu:22', 'horovod/cpu:23', 'horovod/cpu:24', 'horovod/cpu:25', 'horovod/cpu:26', 'horovod/cpu:27', 'horovod/cpu:28', 'horovod/cpu:29', 'horovod/cpu:30', 'horovod/cpu:31', 'horovod/cpu:32', 'horovod/cpu:33', 'horovod/cpu:34', 'horovod/cpu:35', 'horovod/cpu:36', 'horovod/cpu:37', 'horovod/cpu:38', 'horovod/cpu:39', 'horovod/cpu:40', 'horovod/cpu:41', 'horovmages/sec: 0.1 +/- 0.0 (jitter = 0.0)     7.237   0.000   0.000
130     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.240   0.000   0.000
140     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.231   0.000   0.000
150     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.253   0.000   0.000
160     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.254   0.000   0.000
170     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.240   0.000   0.000
180     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.240   0.000   0.000
190     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.236   0.000   0.000
200     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.243   0.000   0.000
210     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.236   0.000   0.000
220     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.261   0.000   0.000
230     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.229   0.000   0.000
240     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.248   0.000   0.000
250     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.227   0.000   0.000
260     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.260   0.000   0.062
270     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.214   0.000   0.000
280     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.253   0.000   0.000
290     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.209   0.000   0.000
300     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.236   0.000   0.000
310     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.252   0.000   0.062
320     images/sec: 0.1 +/- 0.0 (jitter = 0.0)  7.228   0.000   0.000
----------------------------------------------------------------
total images/sec: 5.52
----------------------------------------------------------------

[tfboyd]

.19 epochs is not very much. It takes a total of 90 epochs to get to the end. You would want to run a few more epochs. I also do not test Horovod.

…

On Wed, Jun 20, 2018 at 10:04 AM Guillem Ramírez Gargallo < ***@***.***> wrote: @tfboyd <https://github.com/tfboyd> I had tried to train all night without the forward-only flag. But gives me the same result. Please, can you tell me other think to help model converge? thanks *Specs of the nodes:* CPU: Cavium ThunderX (ARM) Number cores per socket: 48 Number threads per socket: 96 Number socket per MOBO: 2 Number of nodes: 3 RAM: 128GB OS: 14.04.5 LTS *Command using slurm as resource management:* srun --wait=999999 -K --cpu_bind=verbose python tf_cnn_benchmarks.py \ --variable_update=horovod --device=cpu --data_dir=/home/gramirez/imagenet \ --num_inter_threads=1 --num_intra_threads=6 --model=vgg16 --data_name=imagenet\ --nodistortions --allow_growth=True --optimizer=momentum --data_format=NHWC \ --num_batches=320 --num_warmup_batches=20 --batch_size=16 --print_training_accuracy=true *Output from first MPI rank:* TensorFlow: 1.9 Model: vgg16 Dataset: imagenet Mode: training SingleSess: False Batch size: 768 global 16.0 per device Num batches: 320 Num epochs: 0.19 Devices: ['horovod/cpu:0', 'horovod/cpu:1', 'horovod/cpu:2', 'horovod/cpu:3', 'horovod/cpu:4', 'horovod/cpu:5', 'horovod/cpu:6', 'horovod/cpu:7', 'horovod/cpu:8', 'horovod/cpu:9', 'horovod/cpu:10', 'horovod/cpu:11', 'horovod/cpu:12', 'horovod/cpu:13', 'horovod/cpu:14', 'horovod/cpu:15', 'horovod/cpu:16', 'horovod/cpu:17', 'horovod/cpu:18', 'horovod/cpu:19', 'horovod/cpu:20', 'horovod/cpu:21', 'horovod/cpu:22', 'horovod/cpu:23', 'horovod/cpu:24', 'horovod/cpu:25', 'horovod/cpu:26', 'horovod/cpu:27', 'horovod/cpu:28', 'horovod/cpu:29', 'horovod/cpu:30', 'horovod/cpu:31', 'horovod/cpu:32', 'horovod/cpu:33', 'horovod/cpu:34', 'horovod/cpu:35', 'horovod/cpu:36', 'horovod/cpu:37', 'horovod/cpu:38', 'horovod/cpu:39', 'horovod/cpu:40', 'horovod/cpu:41', 'horovmages/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.237 0.000 0.000 130 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.258 0.000 0.000 140 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.237 0.000 0.062 150 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.239 0.000 0.000 160 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.254 0.000 0.000 170 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.241 0.000 0.000 180 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.226 0.000 0.000 190 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.258 0.000 0.000 200 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.232 0.000 0.000 210 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.194 0.000 0.125 220 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.246 0.000 0.000 230 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.250 0.000 0.000 240 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.234 0.000 0.062 250 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.232 0.000 0.000 260 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.223 0.000 0.000 270 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.222 0.000 0.000 280 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.231 0.000 0.000 290 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.245 0.000 0.000 300 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.230 0.000 0.000 310 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.190 0.000 0.000 320 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.213 0.000 0.000 ---------------------------------------------------------------- total images/sec: 5.52 ---------------------------------------------------------------- *Output from second MPI rank:* TensorFlow: 1.9 Model: vgg16 Dataset: imagenet Mode: training SingleSess: False Batch size: 768 global 16.0 per device Num batches: 320 Num epochs: 0.19 Devices: ['horovod/cpu:0', 'horovod/cpu:1', 'horovod/cpu:2', 'horovod/cpu:3', 'horovod/cpu:4', 'horovod/cpu:5', 'horovod/cpu:6', 'horovod/cpu:7', 'horovod/cpu:8', 'horovod/cpu:9', 'horovod/cpu:10', 'horovod/cpu:11', 'horovod/cpu:12', 'horovod/cpu:13', 'horovod/cpu:14', 'horovod/cpu:15', 'horovod/cpu:16', 'horovod/cpu:17', 'horovod/cpu:18', 'horovod/cpu:19', 'horovod/cpu:20', 'horovod/cpu:21', 'horovod/cpu:22', 'horovod/cpu:23', 'horovod/cpu:24', 'horovod/cpu:25', 'horovod/cpu:26', 'horovod/cpu:27', 'horovod/cpu:28', 'horovod/cpu:29', 'horovod/cpu:30', 'horovod/cpu:31', 'horovod/cpu:32', 'horovod/cpu:33', 'horovod/cpu:34', 'horovod/cpu:35', 'horovod/cpu:36', 'horovod/cpu:37', 'horovod/cpu:38', 'horovod/cpu:39', 'horovod/cpu:40', 'horovod/cpu:41', 'horovmages/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.237 0.000 0.000 130 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.240 0.000 0.000 140 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.231 0.000 0.000 150 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.253 0.000 0.000 160 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.254 0.000 0.000 170 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.240 0.000 0.000 180 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.240 0.000 0.000 190 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.236 0.000 0.000 200 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.243 0.000 0.000 210 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.236 0.000 0.000 220 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.261 0.000 0.000 230 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.229 0.000 0.000 240 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.248 0.000 0.000 250 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.227 0.000 0.000 260 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.260 0.000 0.062 270 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.214 0.000 0.000 280 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.253 0.000 0.000 290 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.209 0.000 0.000 300 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.236 0.000 0.000 310 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.252 0.000 0.062 320 images/sec: 0.1 +/- 0.0 (jitter = 0.0) 7.228 0.000 0.000 ---------------------------------------------------------------- total images/sec: 5.52 ---------------------------------------------------------------- — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#209 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWZestIE7XXTZLlJltioJZlghL-MGJsgks5t-oCQgaJpZM4UubXP> .