Error when training network using GammaLatencyRegularizer

[VashishtMadhavan]

Hi, I've been trying to train a simple model on MNIST to test the out the GammaLatencyRegularizer. Since I'm testing this on CPU I'm passing FLOP_LATENCY as the hardware argument. However, it seems to give me the following error:

Traceback (most recent call last):
  File "gamma_prune.py", line 43, in <module>
    network_regularizer = latency_regularizer.GammaLatencyRegularizer([out.op], gamma_threshold=1e-3, hardware='FLOP_LATENCY')
  File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/network_regularizers/latency_regularizer.py", line 92, in __init__
    force_group=force_group, regularizer_blacklist=regularizer_blacklist)
  File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_regularizer_manager.py", line 132, in __init__
    ['%s (%s)' % (o.name, o.type) for o in self._op_deque])
RuntimeError: OpRegularizerManager could not handle ops: ['base/Flatten/flatten/Reshape (Reshape)']

However, when I train the same model using GammaFlopsRegularizer it seems to work fine. Any comment on this issue?

Also, here is my network def'n

def base_model(x_ph, is_training_ph, scope, channels=[32, 64, 64], reuse=False):
    norm_params = {'is_training': False, 'scale': True, 'center': False}
    # Network Definition
    with tf.variable_scope(scope, reuse=reuse):
        with slim.arg_scope([slim.conv2d, slim.fully_connected],
                      normalizer_fn=slim.batch_norm,
                      normalizer_params=norm_params,
                      weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
                      weights_regularizer=slim.l2_regularizer(0.0005)):
            conv1 = slim.conv2d(x_ph, num_outputs=channels[0], kernel_size=3, scope='conv1')
            pool1 = slim.max_pool2d(conv1, kernel_size=2, scope='pool1')
            conv2 = slim.conv2d(pool1, num_outputs=channels[1], kernel_size=3, scope='conv2')
            pool2 = slim.max_pool2d(conv2, kernel_size=2, scope='pool2')
            conv3 = slim.conv2d(pool2, num_outputs=channels[2], kernel_size=3, scope='conv3')
            conv3_flat = slim.flatten(conv3)
            out = slim.fully_connected(conv3_flat, num_outputs=10, normalizer_fn=None, normalizer_params=None,
                activation_fn=None, scope='output')
    pred = tf.argmax(out, axis=1)
    return out, pred

[yairmov]

Thanks for trying out our framework :)

Some thoughts:

1. I assume you are running it on LeNet just to try out the method, correct? LeNet is really small as it is, and there really isn't a benefit for using shrinking on it unless it is for instructional reasons.

2. FLOP_LATENCY is a way to simulate hardware that has infinite memory, so in terms of regularization it behaves identically to a FLOPS regularizer. It is useful mostly as a debugging tool (we should more clearly comment that), but in your case you should just use the GammaFlopsRegularizer.

3. In terms of the error both should behave the same: Ignoring the flatten step as it can not be controlled. Thanks for finding this bug, we'll submit a fix soon.

4. As a workaround (besides just using GammaFlopsRegularizer) and for future reference, most modern convolutional networks forgo the flatten/fully_connected pattern, and instead use 1x1conv/reduce_mean. In your case you can replace the code with:

def base_model(x_ph, is_training_ph, scope, channels=[32, 64, 64], reuse=False):
  norm_params = {'is_training': False, 'scale': True, 'center': False}
  # Network Definition
  with tf.variable_scope(scope, reuse=reuse):
    with slim.arg_scope([slim.conv2d, slim.fully_connected],
                        normalizer_fn=slim.batch_norm,
                        normalizer_params=norm_params,
                        weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
                        weights_regularizer=slim.l2_regularizer(0.0005)):
      conv1 = slim.conv2d(x_ph, num_outputs=channels[0], kernel_size=3, scope='conv1')
      pool1 = slim.max_pool2d(conv1, kernel_size=2, scope='pool1')
      conv2 = slim.conv2d(pool1, num_outputs=channels[1], kernel_size=3, scope='conv2')
      pool2 = slim.max_pool2d(conv2, kernel_size=2, scope='pool2')
      conv3 = slim.conv2d(pool2, num_outputs=channels[2], kernel_size=3, scope='conv3')
      out = slim.conv2d(
              conv3, 10, [1, 1], activation_fn=None, normalizer_fn=None, scope='output_conv')
      out = tf.reduce_mean(out, [1, 2], name='output', keepdims=False)
      pred = tf.argmax(out, axis=1)
      return out, pred

and it should be fully supported by MorphNet.

[VashishtMadhavan]

Cool thanks for the response. I figured that this was the issue and tried the same solution. It worked! Thanks for the help and for this framework

[yairmov]

Glad it works!