Can't run mnist_100_conv_gamma

[anna1969]

I can run mnist_100_full and got similar result as in the paper. But when I ran mnist_100_conv_gamma, it hangs after "INFO:main.utils:e 0, i 0:V_C_class nan, V_E 90, V_C_de 1" as following. Could you please help? Your help is greatly appreciated.
By the way, I had commented the function pool_2d() in nn.py.
-Anna

$THEANO_FLAGS='floatX=float32' python run.py train --encoder-layers convf:32:5:1:1-maxpool:2:2-convv:64:3:1:1-convf:64:3:1:1-maxpool:2:2-convv:128:3:1:1-convv:10:1:1:1-globalmeanpool:6:6-fc:10 --decoder-spec 0-0-0-0-0-0-0-0-0-gauss --denoising-cost-x 0,0,0,0,0,0,0,0,0,1 --labeled-samples 100 --unlabeled-samples 50000 --seed 1 -- mnist_100_conv_gamma
ERROR:main:Subprocess returned fatal: Not a git repository (or any parent up to mount point /nfs/home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
INFO:main:Logging into results/mnist_100_conv_gamma11/log.txt
INFO:main:== COMMAND LINE ==
INFO:main:run.py train --encoder-layers convf:32:5:1:1-maxpool:2:2-convv:64:3:1:1-convf:64:3:1:1-maxpool:2:2-convv:128:3:1:1-convv:10:1:1:1-globalmeanpool:6:6-fc:10 --decoder-spec 0-0-0-0-0-0-0-0-0-gauss --denoising-cost-x 0,0,0,0,0,0,0,0,0,1 --labeled-samples 100 --unlabeled-samples 50000 --seed 1 -- mnist_100_conv_gamma
INFO:main:== PARAMETERS ==
INFO:main: zestbn : bugfix
INFO:main: dseed : 1
INFO:main: top_c : 1
INFO:main: super_noise_std : 0.3
INFO:main: batch_size : 100
INFO:main: dataset : mnist
INFO:main: valid_set_size : 10000
INFO:main: num_epochs : 150
INFO:main: whiten_zca : 0
INFO:main: unlabeled_samples : 50000
INFO:main: decoder_spec : ('0', '0', '0', '0', '0', '0', '0', '0', '0', 'gauss')
INFO:main: valid_batch_size : 100
INFO:main: denoising_cost_x : (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0)
INFO:main: f_local_noise_std : 0.3
INFO:main: cmd : train
INFO:main: act : relu
INFO:main: lrate_decay : 0.67
INFO:main: seed : 1
INFO:main: lr : 0.002
INFO:main: save_to : mnist_100_conv_gamma
INFO:main: save_dir : results/mnist_100_conv_gamma11
INFO:main: commit :
INFO:main: contrast_norm : 0
INFO:main: encoder_layers : ('convf:32:5:1:1', 'maxpool:2:2', 'convv:64:3:1:1', 'convf:64:3:1:1', 'maxpool:2:2', 'convv:128:3:1:1', 'convv:10:1:1:1', 'globalmeanpool:6:6', 'fc:10')
INFO:main: labeled_samples : 100
INFO:main:Using 10000 examples for validation
INFO:main.model:Encoder: clean, labeled
INFO:main.model: 0: noise 0
/nfs/home/yan/PycharmProjects/ladder-master_aicurious2/ladder.py:454: UserWarning: The method getOutputShape is deprecated useget_conv_output_shape instead.
stride, bm))
INFO:main.model: f1: convf, relu, BN, noise 0.00, params [32, 5, 1, 1], dim (1, 28, 28) -> (32, 32, 32)
/nfs/home/yan/PycharmProjects/ladder-master_aicurious2/nn.py:288: UserWarning: pool_2d() will have the parameter ignore_border default value changed to True (currently False). To have consistent behavior with all Theano version, explicitly add the parameter ignore_border=True. On the GPU, using ignore_border=True is needed to use cuDNN. When using ignore_border=False and not using cuDNN, the only GPU combination supported is when ds == st and padding == (0, 0) and mode == 'max'. Otherwise, the convolution will be executed on CPU.
z = pool_2d(z, ds=poolsize, st=poolstride)
INFO:main.model: f2: maxpool, linear, BN, noise 0.00, params [2, 2], dim (32, 32, 32) -> (32, 16, 16)
INFO:main.model: f3: convv, relu, BN, noise 0.00, params [64, 3, 1, 1], dim (32, 16, 16) -> (64, 14, 14)
INFO:main.model: f4: convf, relu, BN, noise 0.00, params [64, 3, 1, 1], dim (64, 14, 14) -> (64, 16, 16)
INFO:main.model: f5: maxpool, linear, BN, noise 0.00, params [2, 2], dim (64, 16, 16) -> (64, 8, 8)
INFO:main.model: f6: convv, relu, BN, noise 0.00, params [128, 3, 1, 1], dim (64, 8, 8) -> (128, 6, 6)
INFO:main.model: f7: convv, relu, BN, noise 0.00, params [10, 1, 1, 1], dim (128, 6, 6) -> (10, 6, 6)
INFO:main.model: f8: globalmeanpool, linear, BN, noise 0.00, params [6, 6], dim (10, 6, 6) -> (10, 1, 1)
INFO:main.model: f9: fc, softmax, BN, noise 0.00, params 10, dim (10, 1, 1) -> (10,)
INFO:main.model:Encoder: corr, labeled
INFO:main.model: 0: noise 0.3
INFO:main.model: f1: convf, relu, BN, noise 0.30, params [32, 5, 1, 1], dim (1, 28, 28) -> (32, 32, 32)
INFO:main.model: f2: maxpool, linear, BN, noise 0.30, params [2, 2], dim (32, 32, 32) -> (32, 16, 16)
INFO:main.model: f3: convv, relu, BN, noise 0.30, params [64, 3, 1, 1], dim (32, 16, 16) -> (64, 14, 14)
INFO:main.model: f4: convf, relu, BN, noise 0.30, params [64, 3, 1, 1], dim (64, 14, 14) -> (64, 16, 16)
INFO:main.model: f5: maxpool, linear, BN, noise 0.30, params [2, 2], dim (64, 16, 16) -> (64, 8, 8)
INFO:main.model: f6: convv, relu, BN, noise 0.30, params [128, 3, 1, 1], dim (64, 8, 8) -> (128, 6, 6)
INFO:main.model: f7: convv, relu, BN, noise 0.30, params [10, 1, 1, 1], dim (128, 6, 6) -> (10, 6, 6)
INFO:main.model: f8: globalmeanpool, linear, BN, noise 0.30, params [6, 6], dim (10, 6, 6) -> (10, 1, 1)
INFO:main.model: f9: fc, softmax, BN, noise 0.30, params 10, dim (10, 1, 1) -> (10,)
INFO:main.model:Decoder: z_corr -> z_est
INFO:main.model: g9: gauss, denois 1.00, dim None -> (10,)
INFO:main.model: g8: 0, , dim (10,) -> (10, 1, 1)
INFO:main.model: g7: 0, , dim (10, 1, 1) -> (10, 6, 6)
INFO:main.model: g6: 0, , dim (10, 6, 6) -> (128, 6, 6)
INFO:main.model: g5: 0, , dim (128, 6, 6) -> (64, 8, 8)
INFO:main.model: g4: 0, , dim (64, 8, 8) -> (64, 16, 16)
INFO:main.model: g3: 0, , dim (64, 16, 16) -> (64, 14, 14)
INFO:main.model: g2: 0, , dim (64, 14, 14) -> (32, 16, 16)
INFO:main.model: g1: 0, , dim (32, 16, 16) -> (32, 32, 32)
INFO:main.model: g0: 0, , dim (32, 32, 32) -> (1, 28, 28)
INFO:main:Found the following parameters: [f_7_b, f_6_b, f_4_b, f_3_b, f_1_b, g_9_a5, f_9_c, f_9_b, g_9_a4, g_9_a3, g_9_a2, g_9_a1, g_9_a10, g_9_a9, g_9_a8, g_9_a7, g_9_a6, f_1_W, f_3_W, f_4_W, f_6_W, f_7_W, f_9_W]
INFO:blocks.algorithms:Taking the cost gradient
INFO:blocks.algorithms:The cost gradient computation graph is built
INFO:main:Balancing 100 labels...
INFO:main.nn:Batch norm parameters: f_1_bn_mean_clean, f_1_bn_var_clean, f_2_bn_mean_clean, f_2_bn_var_clean, f_3_bn_mean_clean, f_3_bn_var_clean, f_4_bn_mean_clean, f_4_bn_var_clean, f_5_bn_mean_clean, f_5_bn_var_clean, f_6_bn_mean_clean, f_6_bn_var_clean, f_7_bn_mean_clean, f_7_bn_var_clean, f_8_bn_mean_clean, f_8_bn_var_clean, f_9_bn_mean_clean, f_9_bn_var_clean
INFO:main:Balancing 100 labels...
INFO:main.nn:Batch norm parameters: f_1_bn_mean_clean, f_1_bn_var_clean, f_2_bn_mean_clean, f_2_bn_var_clean, f_3_bn_mean_clean, f_3_bn_var_clean, f_4_bn_mean_clean, f_4_bn_var_clean, f_5_bn_mean_clean, f_5_bn_var_clean, f_6_bn_mean_clean, f_6_bn_var_clean, f_7_bn_mean_clean, f_7_bn_var_clean, f_8_bn_mean_clean, f_8_bn_var_clean, f_9_bn_mean_clean, f_9_bn_var_clean
INFO:blocks.main_loop:Entered the main loop
/nfs/home/yan/.conda/envs/ladder2/lib/python2.7/site-packages/pandas/core/generic.py:939: PerformanceWarning:
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->mixed-integer,key->block0_values] [items->[0]]

return pytables.to_hdf(path_or_buf, key, self, **kwargs)
INFO:blocks.algorithms:Initializing the training algorithm
INFO:blocks.algorithms:The training algorithm is initialized
INFO:blocks.extensions.monitoring:Monitoring on auxiliary data started
INFO:blocks.extensions.monitoring:Monitoring on auxiliary data finished
INFO:main.utils:e 0, i 0:V_C_class nan, V_E 90, V_C_de 1