[Help Wanted] Is CifarCryptoNets Functional? Low Accuracy with Small Test

[mcsnguyen]

I was able to reproduce the output provided for CifarCryptoNets. I believe they are the prediction values for the 10 categories in Cifar-10 and that the max value is the prediction.

I tested the model separately with the full Cifar-10 dataset and with 1 test image from the dataset and produced the same output which, I believe, is because the code doesn't loop through the network with the GetNext() method like the other provided models.

After iterating through 10 test images and took the max value as the prediction as displayed below, I noticed that the accuracy based on my approach is roughly around 10% instead of the higher accuracy expected of LoLaCryptoNet.

Is the CifarCryptoNet fully functional? Please advise.

# Prediction output for the first 10 images of the Cifar-10 dataset. 
0: -14.9947
1: -13.1457
2: 12.7913
3: 8.6813
4: 12.8255
5: -3.2621
6: 0.7135
7: 4.8243
8: -14.3768
9: -7.1885
Prediction 4 Label 3
0: -8.2010
1: -3.6772
2: 0.1281
3: 10.1785
4: -10.3159
5: -13.5987
6: -7.4372
7: -14.5098
8: -15.4177
9: -5.1588
Prediction 3 Label 8
0: 4.5273
1: 11.4319
2: -8.0989
3: 11.5048
4: -12.1562
5: 14.9259
6: 15.8742
7: -4.1455
8: 9.8287
9: 10.6513
Prediction 6 Label 8
0: 14.2459
1: -11.5474
2: 8.2095
3: -13.5229
4: -7.3236
5: 4.1504
6: -14.8993
7: -12.6799
8: -9.5187
9: -11.7253
Prediction 0 Label 0
0: -2.7235
1: -7.0029
2: 13.5247
3: -13.9630
4: 15.9141
5: 6.9853
6: -13.9143
7: -7.6527
8: -0.7338
9: 10.5105
Prediction 4 Label 6
0: 4.4299
1: -5.5504
2: -8.3946
3: 2.4063
4: 6.1031
5: -10.7999
6: -9.1229
7: 3.5830
8: -5.5245
9: -5.6314
Prediction 4 Label 6
0: -14.3356
1: 4.2231
2: 15.7832
3: 5.4311
4: 3.8572
5: 5.8699
6: -13.2955
7: -11.3057
8: 8.0676
9: -2.4369
Prediction 2 Label 1
0: 11.9822
1: 5.2481
2: -12.5903
3: 5.4623
4: 2.8625
5: 8.3463
6: -3.1007
7: 0.5379
8: -11.2521
9: -8.9174
Prediction 0 Label 6
0: 13.6397
1: -14.8832
2: -9.6858
3: 5.0069
4: -7.6892
5: 2.1170
6: 4.9055
7: 9.6919
8: -4.4410
9: 2.3507
Prediction 0 Label 3
0: 0.9694
1: -13.4114
2: -3.8784
3: -8.0430
4: 3.8281
5: -2.8454
6: 13.5963
7: 2.3001
8: 0.8784
9: -1.7873
Prediction 6 Label 1
Accuracy: 1/10

[rang-microsoft]

We are working on this issue, sorry about the delay

[ohsai]

Me too!

Generating encryption keys 2019-08-07 오전 8:41:46
Encryption keys ready 2019-08-07 오전 8:41:50
Preparing
Prepare LLDenseLayer computed in 0.025961 seconds (08:42:27.449 -- 08:42:27.475)
Current Label : 3
Layer LLDenseLayer computed in 2.002351 seconds (08:55:49.482 -- 08:55:51.484) layer width (4075,1)
-14.9947
-13.1457
12.7913
8.6813
12.8255
-3.2621
0.7135
4.8243
-14.3768
-7.1885
Prediction : 4
Elapsed For single input processing=00:13:24.1783536
Current Label : 8
Layer LLDenseLayer computed in 1.9348759 seconds (09:09:13.487 -- 09:09:15.422) layer width (4075,1)
-8.2010
-3.6772
0.1281
10.1785
-10.3159
-13.5987
-7.4372
-14.5098
-15.4177
-5.1588
Prediction : 3
Elapsed For single input processing=00:26:48.0918934
Current Label : 8
Layer LLDenseLayer computed in 1.9563049 seconds (09:22:40.264 -- 09:22:42.221) layer width (4075,1)
4.5273
11.4319
-8.0989
11.5048
-12.1562
14.9259
15.8742
-4.1455
9.8287
10.6513
Prediction : 6
Elapsed For single input processing=00:40:14.8978885
Current Label : 0
Layer LLDenseLayer computed in 1.9577387 seconds (09:35:21.083 -- 09:35:23.040) layer width (4075,1)
14.2459
-11.5474
8.2095
-13.5229
-7.3236
4.1504
-14.8993
-12.6799
-9.5187
-11.7253
Prediction : 0
Elapsed For single input processing=01:21:37.9993481
Current Label : 6
Layer LLDenseLayer computed in 1.9603099 seconds (10:17:17.184 -- 10:17:19.145) layer width (4075,1)
-2.7235
-7.0029
13.5247
-13.9630
15.9141
6.9853
-13.9143
-7.6527
-0.7338
10.5105
Prediction : 4
Elapsed For single input processing=02:42:04.3713496
Current Label : 6
Layer LLDenseLayer computed in 1.9926721 seconds (11:37:44.916 -- 11:37:46.908) layer width (4075,1)
4.4299
-5.5504
-8.3946
2.4063
6.1031
-10.7999
-9.1229
3.5830
-5.5245
-5.6314
Prediction : 4
Elapsed For single input processing=02:55:19.6033491
Current Label : 1
Layer LLDenseLayer computed in 1.9508309 seconds (11:51:04.017 -- 11:51:05.968) layer width (4075,1)
-14.3356
4.2231
15.7832
5.4311
3.8572
5.8699
-13.2955
-11.3057
8.0676
-2.4369
Prediction : 2
Elapsed For single input processing=03:08:38.6519574
Current Label : 6
Layer LLDenseLayer computed in 2.0191416 seconds (12:04:18.878 -- 12:04:20.897) layer width (4075,1)
11.9822
5.2481
-12.5903
5.4623
2.8625
8.3463
-3.1007
0.5379
-11.2521
-8.9174
Prediction : 0
Elapsed For single input processing=03:21:53.6001682
Current Label : 3
Layer LLDenseLayer computed in 2.013621 seconds (12:17:35.786 -- 12:17:37.799) layer width (4075,1)
13.6397
-14.8832
-9.6858
5.0069
-7.6892
2.1170
4.9055
9.6919
-4.4410
2.3507
Prediction : 0
Elapsed For single input processing=03:35:10.5065900
Current Label : 1
Layer LLDenseLayer computed in 2.0356489 seconds (12:30:54.785 -- 12:30:56.821) layer width (4075,1)
0.9694
-13.4114
-3.8784
-8.0430
3.8281
-2.8454
13.5963
2.3001
0.8784
-1.7873
Prediction : 6
Elapsed For single input processing=03:48:29.5062656

By the way, It REALLY takes long for cifar10..

[rang-microsoft]

We are working on resolving this issue. Sorry it is taking so long.

[ranigb]

I just pushed a fix to this issue. The main problem was DataPreprocessing but during the fix we generated a new model for CIFAR that has higher accuracy (76.31%) and runs in 750 seconds. Please check the readme.md for small changes in the command line arguments.

We also started documenting the process of converting a model trained with Keras to work with this package.