Weird errors raised by running the new accelerated BNN test() method

[henrysky]

System information

• Have I written custom code?: Irrelevant
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04 or Windows 10 v1709 x64): Win10 v1706 x64, CentOS 7.4
• astroNN (Build or Version): 0.9.2.8dev
• Did you try the latest astroNN commit?: Yes
• TensorFlow installed from (source or binary, official build?): official GPU build for Windows and CPU for CentOS
• TensorFlow version: 1.7.0rc-1 for Windows, 1.7.0 for CentOS
• Keras version: 2.1.5
• Python version: 3.6
• CUDA/cuDNN version (Only neccessary if you are using Tensorflow-gpu): 9.0/7.0
• GPU model and memory (Only neccessary if you are using Tensorflow-gpu): Irrelevant
• Exact command/script to reproduce (optional): Running BNN test() mulitple times in a row

Describe the problem

Running BNN test() mulitple times (the 7th time??) in a row will raise a weird error complaining shape not right, or dimension not right, can be reproduced on both CPU and GPU on my Windows and astro department linux server.

This bug is initially discovered by doing open/globular clusters benchmark, because I need to run BNN test() method for every cluster by stopped by this bug

Source code / logs

Variation 1:

ValueError                                Traceback (most recent call last)
<ipython-input-7-c36beede380f> in <module>()
     57         print(np.sum(np.isnan(spec)))
     58     print(name, ' and number of stars: ', indices.shape[0])
---> 59     pred, pred_var = bcnn.test(spec[1:])
     60     means = np.mean(pred, axis=0)
     61     mad_stds = mad_std(pred, axis=0)

d:\university\ast425\astronn\astroNN\models\BayesianCNNBase.py in test(self, input_data, inputs_err)
    210                                                                                           inputs_err[data_gen_shape:])
    211             remainder_result = np.asarray(new.predict_generator(remainder_generator, steps=1))
--> 212             result = np.concatenate((result, remainder_result))
    213 
    214         if result.ndim < 3:  # in case only 1 test data point, in such case we need to add a dimension

ValueError: all the input arrays must have same number of dimensions

Variation 2

ValueError                                Traceback (most recent call last)
<ipython-input-8-b4056e9283f7> in <module>()
     55         print(np.sum(np.isnan(spec)))
     56     print(name, ' and number of stars: ', indices.shape[0])
---> 57     pred, pred_var = bcnn.test(spec[1:])
     58     means = np.mean(pred, axis=0)
     59     mad_stds = mad_std(pred, axis=0)

d:\university\ast425\astronn\astroNN\models\BayesianCNNBase.py in test(self, input_data, inputs_err)
    218 
    219         predictions = result[:, :half_first_dim, 0]  # mean prediction
--> 220         mc_dropout_uncertainty = result[:, :half_first_dim, 1] * (self.labels_std ** 2)  # model uncertainty
    221         predictions_var = np.exp(result[:, half_first_dim:, 0]) * (self.labels_std ** 2)  # predictive uncertainty
    222 

ValueError: operands could not be broadcast together with shapes (1,5075) (25,) 

Suggestion

The cause is unknown but BNN test_old() method is unaffected

[henrysky]

Just checked with TF 1.8.0 cuda 9.0 cudnn 7.1 Keras 2.1.6, its still an issue...need further investigation

[henrysky]

The root cause is batch size not handling correctly, fixed in # 4be8d5a