Training model fails with exception in VS Code

[seank-com]

Repro Steps

1. Follow steps here to install vs code, python (anaconda) and the Tools for AI extension
2. When VS Code launches use the install link to setup the environment
3. From the Tools for AI Start Page, click the HERE link to download this sample
4. Press F5

Expected

Successful execution

Actual

Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting input/train-images-idx3-ubyte.gz
Extracting input/train-labels-idx1-ubyte.gz
Extracting input/t10k-images-idx3-ubyte.gz
Extracting input/t10k-labels-idx1-ubyte.gz
Initialized!
Step 0 (epoch 0.00), 3.0 ms
Minibatch loss: 8.334, learning rate: 0.010000
Minibatch error: 85.9%
Validation error: 84.6%
Step 100 (epoch 0.12), 124.2 ms
Minibatch loss: 3.254, learning rate: 0.010000
Minibatch error: 7.8%
Validation error: 7.7%
Step 200 (epoch 0.23), 127.0 ms
Minibatch loss: 3.382, learning rate: 0.010000
Minibatch error: 10.9%
Validation error: 4.3%
Step 300 (epoch 0.35), 127.2 ms
Minibatch loss: 3.157, learning rate: 0.010000
Minibatch error: 4.7%
Validation error: 2.9%
Step 400 (epoch 0.47), 128.8 ms
Minibatch loss: 3.201, learning rate: 0.010000
Minibatch error: 6.2%
Validation error: 2.6%
Step 500 (epoch 0.58), 128.1 ms
Minibatch loss: 3.179, learning rate: 0.010000
Minibatch error: 6.2%
Validation error: 2.6%
Step 600 (epoch 0.70), 128.0 ms
Minibatch loss: 3.119, learning rate: 0.010000
Minibatch error: 3.1%
Validation error: 2.2%
Step 700 (epoch 0.81), 128.1 ms
Minibatch loss: 2.991, learning rate: 0.010000
Minibatch error: 3.1%
Validation error: 2.3%
Step 800 (epoch 0.93), 127.8 ms
Minibatch loss: 3.048, learning rate: 0.010000
Minibatch error: 4.7%
Validation error: 2.1%
Step 900 (epoch 1.05), 131.8 ms
Minibatch loss: 2.905, learning rate: 0.009500
Minibatch error: 1.6%
Validation error: 1.7%
Step 1000 (epoch 1.16), 127.7 ms
Minibatch loss: 2.876, learning rate: 0.009500
Minibatch error: 1.6%
Validation error: 1.8%
Step 1100 (epoch 1.28), 145.9 ms
Minibatch loss: 2.823, learning rate: 0.009500
Minibatch error: 0.0%
Validation error: 1.5%
Step 1200 (epoch 1.40), 136.1 ms
Minibatch loss: 2.930, learning rate: 0.009500
Minibatch error: 3.1%
Validation error: 1.5%
Step 1300 (epoch 1.51), 143.6 ms
Minibatch loss: 2.815, learning rate: 0.009500
Minibatch error: 3.1%
Validation error: 1.6%
Step 1400 (epoch 1.63), 142.0 ms
Minibatch loss: 2.814, learning rate: 0.009500
Minibatch error: 3.1%
Validation error: 1.5%
Step 1500 (epoch 1.75), 132.6 ms
Minibatch loss: 2.860, learning rate: 0.009500
Minibatch error: 3.1%
Validation error: 1.3%
Step 1600 (epoch 1.86), 134.6 ms
Minibatch loss: 2.724, learning rate: 0.009500
Minibatch error: 1.6%
Validation error: 1.4%
Step 1700 (epoch 1.98), 135.9 ms
Minibatch loss: 2.663, learning rate: 0.009500
Minibatch error: 1.6%
Validation error: 1.6%
Step 1800 (epoch 2.09), 124.8 ms
Minibatch loss: 2.663, learning rate: 0.009025
Minibatch error: 1.6%
Validation error: 1.4%
Step 1900 (epoch 2.21), 124.4 ms
Minibatch loss: 2.638, learning rate: 0.009025
Minibatch error: 1.6%
Validation error: 1.3%
Step 2000 (epoch 2.33), 124.5 ms
Minibatch loss: 2.633, learning rate: 0.009025
Minibatch error: 3.1%
Validation error: 1.2%
Step 2100 (epoch 2.44), 129.6 ms
Minibatch loss: 2.571, learning rate: 0.009025
Minibatch error: 0.0%
Validation error: 1.1%
Step 2200 (epoch 2.56), 133.8 ms
Minibatch loss: 2.575, learning rate: 0.009025
Minibatch error: 1.6%
Validation error: 1.1%
Step 2300 (epoch 2.68), 127.3 ms
Minibatch loss: 2.581, learning rate: 0.009025
Minibatch error: 1.6%
Validation error: 1.1%
Step 2400 (epoch 2.79), 122.5 ms
Minibatch loss: 2.517, learning rate: 0.009025
Minibatch error: 1.6%
Validation error: 1.3%
Step 2500 (epoch 2.91), 133.5 ms
Minibatch loss: 2.474, learning rate: 0.009025
Minibatch error: 0.0%
Validation error: 1.2%
Step 2600 (epoch 3.03), 124.5 ms
Minibatch loss: 2.470, learning rate: 0.008574
Minibatch error: 1.6%
Validation error: 1.6%
Step 2700 (epoch 3.14), 120.9 ms
Minibatch loss: 2.477, learning rate: 0.008574
Minibatch error: 1.6%
Validation error: 1.1%
Step 2800 (epoch 3.26), 120.6 ms
Minibatch loss: 2.500, learning rate: 0.008574
Minibatch error: 4.7%
Validation error: 1.1%
Step 2900 (epoch 3.37), 122.4 ms
Minibatch loss: 2.504, learning rate: 0.008574
Minibatch error: 1.6%
Validation error: 1.0%
Step 3000 (epoch 3.49), 121.3 ms
Minibatch loss: 2.415, learning rate: 0.008574
Minibatch error: 3.1%
Validation error: 0.9%
Step 3100 (epoch 3.61), 121.8 ms
Minibatch loss: 2.396, learning rate: 0.008574
Minibatch error: 3.1%
Validation error: 1.0%
Step 3200 (epoch 3.72), 120.6 ms
Minibatch loss: 2.348, learning rate: 0.008574
Minibatch error: 0.0%
Validation error: 1.1%
Step 3300 (epoch 3.84), 121.2 ms
Minibatch loss: 2.351, learning rate: 0.008574
Minibatch error: 1.6%
Validation error: 1.1%
Step 3400 (epoch 3.96), 127.5 ms
Minibatch loss: 2.303, learning rate: 0.008574
Minibatch error: 1.6%
Validation error: 1.2%
Step 3500 (epoch 4.07), 138.8 ms
Minibatch loss: 2.286, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 0.9%
Step 3600 (epoch 4.19), 159.9 ms
Minibatch loss: 2.251, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 0.9%
Step 3700 (epoch 4.31), 149.2 ms
Minibatch loss: 2.230, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 1.0%
Step 3800 (epoch 4.42), 145.7 ms
Minibatch loss: 2.215, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 0.9%
Step 3900 (epoch 4.54), 140.0 ms
Minibatch loss: 2.228, learning rate: 0.008145
Minibatch error: 1.6%
Validation error: 1.0%
Step 4000 (epoch 4.65), 135.8 ms
Minibatch loss: 2.235, learning rate: 0.008145
Minibatch error: 3.1%
Validation error: 1.1%
Step 4100 (epoch 4.77), 135.8 ms
Minibatch loss: 2.168, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 0.9%
Step 4200 (epoch 4.89), 136.9 ms
Minibatch loss: 2.159, learning rate: 0.008145
Minibatch error: 0.0%
Validation error: 1.1%
Step 4300 (epoch 5.00), 165.6 ms
Minibatch loss: 2.209, learning rate: 0.007738
Minibatch error: 3.1%
Validation error: 0.9%
Step 4400 (epoch 5.12), 172.2 ms
Minibatch loss: 2.148, learning rate: 0.007738
Minibatch error: 1.6%
Validation error: 1.1%
Step 4500 (epoch 5.24), 148.4 ms
Minibatch loss: 2.193, learning rate: 0.007738
Minibatch error: 4.7%
Validation error: 0.9%
Step 4600 (epoch 5.35), 134.1 ms
Minibatch loss: 2.085, learning rate: 0.007738
Minibatch error: 0.0%
Validation error: 1.0%
Step 4700 (epoch 5.47), 130.8 ms
Minibatch loss: 2.070, learning rate: 0.007738
Minibatch error: 0.0%
Validation error: 1.0%
Step 4800 (epoch 5.59), 130.4 ms
Minibatch loss: 2.055, learning rate: 0.007738
Minibatch error: 0.0%
Validation error: 0.9%
Step 4900 (epoch 5.70), 142.3 ms
Minibatch loss: 2.080, learning rate: 0.007738
Minibatch error: 1.6%
Validation error: 1.1%
Step 5000 (epoch 5.82), 127.7 ms
Minibatch loss: 2.071, learning rate: 0.007738
Minibatch error: 3.1%
Validation error: 0.9%
Step 5100 (epoch 5.93), 147.5 ms
Minibatch loss: 2.024, learning rate: 0.007738
Minibatch error: 1.6%
Validation error: 1.1%
Step 5200 (epoch 6.05), 148.0 ms
Minibatch loss: 2.056, learning rate: 0.007351
Minibatch error: 4.7%
Validation error: 1.0%
Step 5300 (epoch 6.17), 141.4 ms
Minibatch loss: 1.970, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 0.8%
Step 5400 (epoch 6.28), 134.5 ms
Minibatch loss: 1.958, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 1.0%
Step 5500 (epoch 6.40), 131.6 ms
Minibatch loss: 1.961, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 1.0%
Step 5600 (epoch 6.52), 140.4 ms
Minibatch loss: 1.927, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 0.9%
Step 5700 (epoch 6.63), 163.3 ms
Minibatch loss: 1.913, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 0.8%
Step 5800 (epoch 6.75), 146.6 ms
Minibatch loss: 1.897, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 0.9%
Step 5900 (epoch 6.87), 134.0 ms
Minibatch loss: 1.890, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 0.9%
Step 6000 (epoch 6.98), 135.4 ms
Minibatch loss: 1.889, learning rate: 0.007351
Minibatch error: 0.0%
Validation error: 1.0%
Step 6100 (epoch 7.10), 141.3 ms
Minibatch loss: 1.868, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6200 (epoch 7.21), 143.3 ms
Minibatch loss: 1.844, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6300 (epoch 7.33), 158.5 ms
Minibatch loss: 1.837, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6400 (epoch 7.45), 147.4 ms
Minibatch loss: 1.839, learning rate: 0.006983
Minibatch error: 1.6%
Validation error: 0.9%
Step 6500 (epoch 7.56), 145.9 ms
Minibatch loss: 1.810, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6600 (epoch 7.68), 142.3 ms
Minibatch loss: 1.821, learning rate: 0.006983
Minibatch error: 1.6%
Validation error: 0.9%
Step 6700 (epoch 7.80), 131.6 ms
Minibatch loss: 1.784, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6800 (epoch 7.91), 131.9 ms
Minibatch loss: 1.774, learning rate: 0.006983
Minibatch error: 0.0%
Validation error: 0.9%
Step 6900 (epoch 8.03), 133.5 ms
Minibatch loss: 1.757, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 0.7%
Step 7000 (epoch 8.15), 129.1 ms
Minibatch loss: 1.753, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 0.9%
Step 7100 (epoch 8.26), 132.9 ms
Minibatch loss: 1.734, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 0.8%
Step 7200 (epoch 8.38), 133.1 ms
Minibatch loss: 1.736, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 1.0%
Step 7300 (epoch 8.49), 136.3 ms
Minibatch loss: 1.749, learning rate: 0.006634
Minibatch error: 3.1%
Validation error: 0.8%
Step 7400 (epoch 8.61), 150.9 ms
Minibatch loss: 1.700, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 0.7%
Step 7500 (epoch 8.73), 142.8 ms
Minibatch loss: 1.699, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 0.9%
Step 7600 (epoch 8.84), 136.3 ms
Minibatch loss: 1.740, learning rate: 0.006634
Minibatch error: 1.6%
Validation error: 0.8%
Step 7700 (epoch 8.96), 135.7 ms
Minibatch loss: 1.666, learning rate: 0.006634
Minibatch error: 0.0%
Validation error: 1.0%
Step 7800 (epoch 9.08), 151.3 ms
Minibatch loss: 1.659, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Step 7900 (epoch 9.19), 138.0 ms
Minibatch loss: 1.646, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.8%
Step 8000 (epoch 9.31), 146.6 ms
Minibatch loss: 1.646, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Step 8100 (epoch 9.43), 130.6 ms
Minibatch loss: 1.634, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Step 8200 (epoch 9.54), 129.0 ms
Minibatch loss: 1.618, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Step 8300 (epoch 9.66), 131.6 ms
Minibatch loss: 1.607, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.8%
Step 8400 (epoch 9.77), 142.1 ms
Minibatch loss: 1.597, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.8%
Step 8500 (epoch 9.89), 153.5 ms
Minibatch loss: 1.606, learning rate: 0.006302
Minibatch error: 0.0%
Validation error: 0.9%
Test error: 0.7%
Backend MacOSX is interactive backend. Turning interactive mode on.

Then the debugger broke at line 409 of mnist.py with the following:

Exception has occurred: SystemExit
exception: no description
  File "/Users/seank/Desktop/MNIST/mnist.py", line 409, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)

[TobeyQin]

Hi @seank-com , thank you for your report.

This exception is caused by the _sys.exit() in TensorFlow's build-in function "tensorflow.app.run". __sys.exit() will trigger an SystemExit exception to exit python, and your shell captured this exception. It will occur if you use anaconda or ipython as your python shell. It's not a real problem and won't influence on the program's result. You can just ignore it.

Hope this can help you.

[seank-com]

Should the documentation be updated not point to anaconda in the Prerequisite section when they say "Install Python 3.5 or greater."? If the failure is expected, it seems like a poor first user experience.

[TobeyQin]

Please follow this document in vscode tools for ai github. It is the newest document. Thanks.