Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Based on FP16, the network training of resnet50 was carried out, and the existence accuracy randomly converged to 0.76 #2164

Open
south-ocean opened this issue Jul 20, 2023 · 1 comment
Open

Based on FP16, the network training of resnet50 was carried out, and the existence accuracy randomly converged to 0.76 #2164

south-ocean opened this issue Jul 20, 2023 · 1 comment

Comments

@south-ocean
Copy link

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

tf2.7、tf.9和tf2.11

Custom code

Yes

OS platform and distribution

centos7.6

Mobile device

No response

Python version

3.7、3.8和3.9

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

Using the inagenet dataset, on the resnet50 network, based on FP16 training, if the algorithm is not disabled, it will not converge directly, after disabling the winograd igemm algorithm, the network has random convergence, is this phenomenon known, is there any solution at present?

Standalone code to reproduce the issue

export MIOPEN_DEBUG_CONV_FFT=1
export MIOPEN_DEBUG_CONV_DIRECT=1
export MIOPEN_DEBUG_CONV_GEMM=1
export MIOPEN_DEBUG_CONV_WINOGRAD=0
export MIOPEN_DEBUG_CONV_IMPLICIT_GEMM=0

python3  /public/home/TF_test/rocm5.1/tf2.7/models-master_fp16/official/vision/image_classification/resnet/resnet_ctl_imagenet_main.py      --num_gpus=4     --batch_size=128  --train_epochs=90  --use_synthetic_data=false    --data_dir=/public/software/apps/DeepLearning/Data/ImageNet-tensorflow/  --enable_checkpoint_and_export  --dtype=fp16   --model_dir=./Checkpoint/
and 
python3  /public/home/TF_test/rocm5.4/tf2.11/models-master_fp16/official/benchmark/models/resnet_imagenet_main.py      --num_gpus=4     --batch_size=512  --train_epochs=90  --use_synthetic_data=false    --data_dir=/public/software/apps/DeepLearning/Data/ImageNet-tensorflow/  --enable_checkpoint_and_export  --dtype=fp16   --model_dir=./Checkpoint/

Relevant log output

I0717 10:42:45.093785 47025319389696 controller.py:291]  eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
I0717 11:19:43.333361 47025319389696 controller.py:291]  eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
I0717 11:56:49.300491 47025319389696 controller.py:291]  eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
I0717 12:33:43.809209 47025319389696 controller.py:291]  eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
I0717 13:10:37.399015 47025319389696 controller.py:291]  eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
I0717 13:47:31.637742 47025319389696 controller.py:291]  eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
I0717 14:24:25.244748 47025319389696 controller.py:291]  eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
I0717 15:01:26.831945 47025319389696 controller.py:291]  eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
I0717 15:39:10.716763 47025319389696 controller.py:291]  eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
I0717 16:17:14.804115 47025319389696 controller.py:291]  eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
I0717 16:54:38.272225 47025319389696 controller.py:291]  eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
I0717 17:32:14.305328 47025319389696 controller.py:291]  eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
I0717 18:19:19.995545 47025319389696 controller.py:291]  eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
I0717 19:01:38.542571 47025319389696 controller.py:291]  eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
I0717 19:39:52.303068 47025319389696 controller.py:291]  eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
I0717 20:17:52.557711 47025319389696 controller.py:291]  eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
I0717 20:55:54.236690 47025319389696 controller.py:291]  eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
I0717 21:33:46.721763 47025319389696 controller.py:291]  eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
I0717 22:11:56.432586 47025319389696 controller.py:291]  eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
I0717 22:49:48.895009 47025319389696 controller.py:291]  eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
I0717 23:27:56.641305 47025319389696 controller.py:291]  eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
I0718 00:06:08.226011 47025319389696 controller.py:291]  eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
I0718 00:43:44.846733 47025319389696 controller.py:291]  eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
I0718 01:21:01.860804 47025319389696 controller.py:291]  eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
I0718 01:58:26.185002 47025319389696 controller.py:291]  eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
I0718 02:35:51.395891 47025319389696 controller.py:291]  eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
I0718 03:12:58.957236 47025319389696 controller.py:291]  eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
I0718 03:50:05.619722 47025319389696 controller.py:291]  eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
I0718 04:27:19.267962 47025319389696 controller.py:291]  eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
I0718 05:04:16.388976 47025319389696 controller.py:291]  eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
I0718 05:41:28.633920 47025319389696 controller.py:291]  eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
I0718 06:18:23.006313 47025319389696 controller.py:291]  eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
I0718 06:55:19.940962 47025319389696 controller.py:291]  eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
I0718 07:32:11.406366 47025319389696 controller.py:291]  eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
I0718 08:09:13.098378 47025319389696 controller.py:291]  eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
I0718 08:46:11.470759 47025319389696 controller.py:291]  eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
I0718 09:23:19.292029 47025319389696 controller.py:291]  eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
I0718 09:59:58.652599 47025319389696 controller.py:291]  eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
I0718 10:36:30.269412 47025319389696 controller.py:291]  eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
I0718 11:12:59.710509 47025319389696 controller.py:291]  eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
I0718 11:49:34.702973 47025319389696 controller.py:291]  eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
I0718 12:26:10.658050 47025319389696 controller.py:291]  eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
I0718 13:02:54.373109 47025319389696 controller.py:291]  eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
I0718 13:39:56.424064 47025319389696 controller.py:291]  eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
I0718 14:16:56.972168 47025319389696 controller.py:291]  eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
I0718 14:53:49.710101 47025319389696 controller.py:291]  eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
I0718 15:31:00.055065 47025319389696 controller.py:291]  eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
I0718 16:26:43.115331 47025319389696 controller.py:291]  eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
I0718 17:14:57.232034 47025319389696 controller.py:291]  eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
I0718 18:08:04.532505 47025319389696 controller.py:291]  eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
I0718 18:45:18.862020 47025319389696 controller.py:291]  eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
I0718 19:22:18.786072 47025319389696 controller.py:291]  eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
I0718 19:59:17.472421 47025319389696 controller.py:291]  eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
I0718 20:46:59.059379 47025319389696 controller.py:291]  eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
I0718 21:45:11.991321 47025319389696 controller.py:291]  eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
I0718 22:22:17.513960 47025319389696 controller.py:291]  eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
I0718 22:59:05.393987 47025319389696 controller.py:291]  eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
I0718 23:36:01.059101 47025319389696 controller.py:291]  eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
I0719 00:12:57.767053 47025319389696 controller.py:291]  eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
I0719 00:50:08.519172 47025319389696 controller.py:291]  eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
I0719 01:27:14.241234 47025319389696 controller.py:291]  eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
I0719 02:04:12.080565 47025319389696 controller.py:291]  eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
I0719 02:41:31.931695 47025319389696 controller.py:291]  eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
I0719 03:18:53.615640 47025319389696 controller.py:291]  eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
I0719 03:56:03.865268 47025319389696 controller.py:291]  eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
I0719 04:33:13.600398 47025319389696 controller.py:291]  eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
I0719 05:10:26.761108 47025319389696 controller.py:291]  eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
I0719 05:47:24.072698 47025319389696 controller.py:291]  eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
I0719 06:24:23.891359 47025319389696 controller.py:291]  eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
I0719 07:01:34.794637 47025319389696 controller.py:291]  eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
I0719 07:38:47.115363 47025319389696 controller.py:291]  eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
I0719 08:15:57.504423 47025319389696 controller.py:291]  eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
I0719 08:53:14.727922 47025319389696 controller.py:291]  eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
I0719 09:30:20.804430 47025319389696 controller.py:291]  eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
I0719 10:07:20.642869 47025319389696 controller.py:291]  eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
I0719 10:46:23.875730 47025319389696 controller.py:291]  eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
I0719 11:23:17.199512 47025319389696 controller.py:291]  eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
I0719 12:00:04.367842 47025319389696 controller.py:291]  eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
I0719 12:37:04.218576 47025319389696 controller.py:291]  eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
I0719 13:14:02.169136 47025319389696 controller.py:291]  eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
I0719 13:51:13.950588 47025319389696 controller.py:291]  eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
I0719 14:28:19.452734 47025319389696 controller.py:291]  eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
I0719 15:05:49.973363 47025319389696 controller.py:291]  eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
I0719 15:42:51.213348 47025319389696 controller.py:291]  eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
I0719 16:19:54.159197 47025319389696 controller.py:291]  eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
I0719 16:57:06.514332 47025319389696 controller.py:291]  eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
I0719 17:34:12.212647 47025319389696 controller.py:291]  eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
I0719 18:11:14.854687 47025319389696 controller.py:291]  eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
I0719 18:48:31.401537 47025319389696 controller.py:291]  eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
I0719 19:25:37.033689 47025319389696 controller.py:291]  eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
 eval | step:  10009 | eval time:   74.7 sec | output: {'test_accuracy': 0.08882, 'test_loss': 1.2578498}
 eval | step:  20018 | eval time:   47.9 sec | output: {'test_accuracy': 0.18752, 'test_loss': 1.0495228}
 eval | step:  30027 | eval time:   48.0 sec | output: {'test_accuracy': 0.2584, 'test_loss': 0.9102226}
 eval | step:  40036 | eval time:   47.3 sec | output: {'test_accuracy': 0.33168, 'test_loss': 0.77455497}
 eval | step:  50045 | eval time:   48.6 sec | output: {'test_accuracy': 0.32974, 'test_loss': 0.7866249}
 eval | step:  60054 | eval time:   48.0 sec | output: {'test_accuracy': 0.32832, 'test_loss': 0.7928696}
 eval | step:  70063 | eval time:   47.4 sec | output: {'test_accuracy': 0.36952, 'test_loss': 0.7357325}
 eval | step:  80072 | eval time:   49.6 sec | output: {'test_accuracy': 0.33716, 'test_loss': 0.7839137}
 eval | step:  90081 | eval time:   51.2 sec | output: {'test_accuracy': 0.32314, 'test_loss': 0.82485163}
 eval | step:  100090 | eval time:   49.1 sec | output: {'test_accuracy': 0.36102, 'test_loss': 0.7599265}
 eval | step:  110099 | eval time:   49.5 sec | output: {'test_accuracy': 0.42732, 'test_loss': 0.6477533}
 eval | step:  120108 | eval time:   49.1 sec | output: {'test_accuracy': 0.3769, 'test_loss': 0.745124}
 eval | step:  130117 | eval time:  196.2 sec | output: {'test_accuracy': 0.43222, 'test_loss': 0.63742775}
 eval | step:  140126 | eval time:   49.1 sec | output: {'test_accuracy': 0.43536, 'test_loss': 0.63410264}
 eval | step:  150135 | eval time:   46.7 sec | output: {'test_accuracy': 0.43782, 'test_loss': 0.6309778}
 eval | step:  160144 | eval time:   49.5 sec | output: {'test_accuracy': 0.44036, 'test_loss': 0.6212317}
 eval | step:  170153 | eval time:   47.7 sec | output: {'test_accuracy': 0.4211, 'test_loss': 0.6638591}
 eval | step:  180162 | eval time:   48.7 sec | output: {'test_accuracy': 0.41044, 'test_loss': 0.6806779}
 eval | step:  190171 | eval time:   48.9 sec | output: {'test_accuracy': 0.4227, 'test_loss': 0.6592407}
 eval | step:  200180 | eval time:   47.1 sec | output: {'test_accuracy': 0.41614, 'test_loss': 0.6689654}
 eval | step:  210189 | eval time:   49.0 sec | output: {'test_accuracy': 0.43086, 'test_loss': 0.6471556}
 eval | step:  220198 | eval time:   49.0 sec | output: {'test_accuracy': 0.43288, 'test_loss': 0.651372}
 eval | step:  230207 | eval time:   48.6 sec | output: {'test_accuracy': 0.44876, 'test_loss': 0.61514467}
 eval | step:  240216 | eval time:   47.8 sec | output: {'test_accuracy': 0.39994, 'test_loss': 0.6896438}
 eval | step:  250225 | eval time:   49.3 sec | output: {'test_accuracy': 0.43974, 'test_loss': 0.63887787}
 eval | step:  260234 | eval time:   48.7 sec | output: {'test_accuracy': 0.42148, 'test_loss': 0.65387714}
 eval | step:  270243 | eval time:   48.0 sec | output: {'test_accuracy': 0.42988, 'test_loss': 0.64366835}
 eval | step:  280252 | eval time:   49.1 sec | output: {'test_accuracy': 0.4292, 'test_loss': 0.64544094}
 eval | step:  290261 | eval time:   47.7 sec | output: {'test_accuracy': 0.43688, 'test_loss': 0.633886}
 eval | step:  300270 | eval time:   47.0 sec | output: {'test_accuracy': 0.4378, 'test_loss': 0.6427796}
 eval | step:  310279 | eval time:   47.1 sec | output: {'test_accuracy': 0.60604, 'test_loss': 0.41581097}
 eval | step:  320288 | eval time:   47.6 sec | output: {'test_accuracy': 0.60958, 'test_loss': 0.41247183}
 eval | step:  330297 | eval time:   47.5 sec | output: {'test_accuracy': 0.62418, 'test_loss': 0.3925642}
 eval | step:  340306 | eval time:   49.3 sec | output: {'test_accuracy': 0.60878, 'test_loss': 0.41140762}
 eval | step:  350315 | eval time:   46.3 sec | output: {'test_accuracy': 0.61416, 'test_loss': 0.40241298}
 eval | step:  360324 | eval time:   48.7 sec | output: {'test_accuracy': 0.61666, 'test_loss': 0.40103617}
 eval | step:  370333 | eval time:   47.6 sec | output: {'test_accuracy': 0.64238, 'test_loss': 0.3717267}
 eval | step:  380342 | eval time:   46.9 sec | output: {'test_accuracy': 0.63504, 'test_loss': 0.37905875}
 eval | step:  390351 | eval time:   46.7 sec | output: {'test_accuracy': 0.62588, 'test_loss': 0.38983408}
 eval | step:  400360 | eval time:   47.5 sec | output: {'test_accuracy': 0.62752, 'test_loss': 0.39042756}
 eval | step:  410369 | eval time:   47.6 sec | output: {'test_accuracy': 0.63086, 'test_loss': 0.38619643}
 eval | step:  420378 | eval time:   46.4 sec | output: {'test_accuracy': 0.61492, 'test_loss': 0.40776753}
 eval | step:  430387 | eval time:   47.1 sec | output: {'test_accuracy': 0.61482, 'test_loss': 0.40698445}
 eval | step:  440396 | eval time:   46.6 sec | output: {'test_accuracy': 0.61064, 'test_loss': 0.40885204}
 eval | step:  450405 | eval time:   47.2 sec | output: {'test_accuracy': 0.6101, 'test_loss': 0.41232613}
 eval | step:  460414 | eval time:   50.2 sec | output: {'test_accuracy': 0.62022, 'test_loss': 0.39717302}
 eval | step:  470423 | eval time:   47.8 sec | output: {'test_accuracy': 0.63738, 'test_loss': 0.3755849}
 eval | step:  480432 | eval time:   48.6 sec | output: {'test_accuracy': 0.63156, 'test_loss': 0.38420147}
 eval | step:  490441 | eval time:   47.9 sec | output: {'test_accuracy': 0.61762, 'test_loss': 0.40265194}
 eval | step:  500450 | eval time:   47.4 sec | output: {'test_accuracy': 0.63422, 'test_loss': 0.3809521}
 eval | step:  510459 | eval time:   50.4 sec | output: {'test_accuracy': 0.62098, 'test_loss': 0.39487627}
 eval | step:  520468 | eval time:   47.2 sec | output: {'test_accuracy': 0.61512, 'test_loss': 0.40484685}
 eval | step:  530477 | eval time:   47.3 sec | output: {'test_accuracy': 0.63776, 'test_loss': 0.37440324}
 eval | step:  540486 | eval time:  203.9 sec | output: {'test_accuracy': 0.62722, 'test_loss': 0.39110208}
 eval | step:  550495 | eval time:   47.9 sec | output: {'test_accuracy': 0.63562, 'test_loss': 0.38335004}
 eval | step:  560504 | eval time:   46.5 sec | output: {'test_accuracy': 0.61826, 'test_loss': 0.40100682}
 eval | step:  570513 | eval time:   46.1 sec | output: {'test_accuracy': 0.63496, 'test_loss': 0.3817627}
 eval | step:  580522 | eval time:   47.2 sec | output: {'test_accuracy': 0.61354, 'test_loss': 0.40860355}
 eval | step:  590531 | eval time:   48.7 sec | output: {'test_accuracy': 0.62766, 'test_loss': 0.3902482}
 eval | step:  600540 | eval time:   47.3 sec | output: {'test_accuracy': 0.62992, 'test_loss': 0.38722458}
 eval | step:  610549 | eval time:   49.5 sec | output: {'test_accuracy': 0.6785, 'test_loss': 0.3319877}
 eval | step:  620558 | eval time:   47.3 sec | output: {'test_accuracy': 0.68068, 'test_loss': 0.3297141}
 eval | step:  630567 | eval time:   47.5 sec | output: {'test_accuracy': 0.67544, 'test_loss': 0.33620217}
 eval | step:  640576 | eval time:   47.8 sec | output: {'test_accuracy': 0.67902, 'test_loss': 0.32960042}
 eval | step:  650585 | eval time:   47.2 sec | output: {'test_accuracy': 0.68312, 'test_loss': 0.32697994}
 eval | step:  660594 | eval time:   49.1 sec | output: {'test_accuracy': 0.68692, 'test_loss': 0.3218242}
 eval | step:  670603 | eval time:   47.5 sec | output: {'test_accuracy': 0.68098, 'test_loss': 0.33020604}
 eval | step:  680612 | eval time:   47.8 sec | output: {'test_accuracy': 0.68784, 'test_loss': 0.32333767}
 eval | step:  690621 | eval time:   47.1 sec | output: {'test_accuracy': 0.68598, 'test_loss': 0.32332215}
 eval | step:  700630 | eval time:   48.9 sec | output: {'test_accuracy': 0.68566, 'test_loss': 0.3250208}
 eval | step:  710639 | eval time:   47.2 sec | output: {'test_accuracy': 0.68696, 'test_loss': 0.32350788}
 eval | step:  720648 | eval time:   52.0 sec | output: {'test_accuracy': 0.68832, 'test_loss': 0.3217886}
 eval | step:  730657 | eval time:   49.9 sec | output: {'test_accuracy': 0.6852, 'test_loss': 0.32565758}
 eval | step:  740666 | eval time:   49.2 sec | output: {'test_accuracy': 0.68326, 'test_loss': 0.33065847}
 eval | step:  750675 | eval time:   48.0 sec | output: {'test_accuracy': 0.68372, 'test_loss': 0.32719776}
 eval | step:  760684 | eval time:   47.9 sec | output: {'test_accuracy': 0.68192, 'test_loss': 0.32962936}
 eval | step:  770693 | eval time:   47.2 sec | output: {'test_accuracy': 0.68262, 'test_loss': 0.32934195}
 eval | step:  780702 | eval time:   46.9 sec | output: {'test_accuracy': 0.67706, 'test_loss': 0.3367395}
 eval | step:  790711 | eval time:   49.9 sec | output: {'test_accuracy': 0.68592, 'test_loss': 0.32515442}
 eval | step:  800720 | eval time:   49.5 sec | output: {'test_accuracy': 0.6774, 'test_loss': 0.33403492}
 eval | step:  810729 | eval time:   48.3 sec | output: {'test_accuracy': 0.68796, 'test_loss': 0.32154444}
 eval | step:  820738 | eval time:   47.6 sec | output: {'test_accuracy': 0.68378, 'test_loss': 0.32751894}
 eval | step:  830747 | eval time:   47.5 sec | output: {'test_accuracy': 0.6856, 'test_loss': 0.3268724}
 eval | step:  840756 | eval time:   46.4 sec | output: {'test_accuracy': 0.68698, 'test_loss': 0.32407942}
 eval | step:  850765 | eval time:   48.8 sec | output: {'test_accuracy': 0.6875, 'test_loss': 0.32387343}
 eval | step:  860774 | eval time:   48.9 sec | output: {'test_accuracy': 0.6877, 'test_loss': 0.32420784}
 eval | step:  870783 | eval time:   48.5 sec | output: {'test_accuracy': 0.688, 'test_loss': 0.3242049}
 eval | step:  880792 | eval time:   48.6 sec | output: {'test_accuracy': 0.6882, 'test_loss': 0.324394}
 eval | step:  890801 | eval time:   49.7 sec | output: {'test_accuracy': 0.68516, 'test_loss': 0.32722312}
 eval | step:  900810 | eval time:   48.7 sec | output: {'test_accuracy': 0.68694, 'test_loss': 0.32629344}
@south-ocean
Copy link
Author

@jeffdaily @keryell @pfultz2 @whchung It had been three weeks, can someone help me to solve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@south-ocean