Are other datasets than object recognition in images supported?

[Michelvl92]

Hi All!

I have a neural network in tensorflow for classical channel estimation which I want to compress. This network is a simple Fully Connected 'denoising' network consisting of ten layers and 48x1 inputs and 48x1 outputs. This network easily solves the classical problem x=H*y with mean square error, where x is known and put in the NN (can be seen as image data) and y should be the optimal result (can be seen as a label that suits a particular x). So the NN should solve y with a small as possible mse (when x is input). (i hope this is clear, else let me know).

I have this network trained in TF, with optimizer='rmsprop',loss='mse'.
My training schedule looks like this with 2000 training samples;
epochs=100,batch_size=3211
epochs=100,batch_size=3221
epochs=100,batch_size=3241
epochs=100,batch_size=3281
epochs=100,batch_size=32161

Now I want to know if it is possible to compress this network with my dataset with this Framework?

I have read the 'self-defined models' section. From here it is easy to define a 'Network definition' file (as my model is made in TF. But I think I should change the loss function? As it is especially for the softmax of CNN's? (I only use mse to minimize diff between x and y). And how about the accuracy? I have never used such a measurement. How should I change the setup_lrn_rate function to have similar training as above?

Furthermore the "network definition" doesn't look hard to change as well. I have loaded my data, then parsed as numpy arrays (both x and y are the same). But now comes the tricky part. First of all, I don't have any classes... Furthermore, how should I choose my training samples, validation samples, and evaluation samples? I only trained with 2000 samples before. Furthermore, how should I choose batch_size and batch_size_eval? I always used 1000 sample to do a prediction. Do I need to change more things?

At the moment if have done the above, and it runs without any error. But the loss looks totally different from what i had in TF. So i really don't know if the performance is correctly measured, and if even correctly internally applied in the framework to change all the weights and biases? And maybe it is even not possible to use this framework with other data sets than bject recognition in images?

Hope to hear from anyone.

[jiaxiang-wu]

1. This framework can be applied to other data sets, as long as your data input pipeline and network architecture is correctly defined.
2. You should change the loss function from softmax to mse to fit your problem. Accuracy is not required and you can simply set it to "- loss" (the larger, the better).
3. What do you mean by "batch_size=32_16_1"?

[Michelvl92]

1. This makes everything clear. Maybe mention it explicitly in the
2. This makes the loss function clear as well.
3. I train my model consecutively with 5 different batch sizes, and 100 epochs each: 1; (32x1x1), 2;
   (32x2x1), 3; (32x4x1), 2; (32x8x1), 2; (32x16x1). Although the "Network definition" file only allows to set
   one epoch and one batch size.
4. What should I set for the num of classes? Should I set this to the number of training data? As
   obviously, my problem is not based on classes.
5. For both 'Channel Pruning' methods, it is mentioned in the wiki, that pruning is done for the
   convolutional layers only. Is this correct? So the framework doesn't support channel pruning for fully
   connected layers?

[jiaxiang-wu]

1. If you need to vary the batch size during the training process, then you should implement it in your self-defined data input pipeline. You can omit AbstractDataset and create the data input pipeline from scratch with variable batch size, as long as the input features and output labels is correctly set.
2. The number of classes is not required since you are solving a regression problem instead of classification. However, currently DistillationHelper only supports distillation on classification logits, so FLAGS.enbl_dst should be disabled.
3. Yes, your understanding is correct that ChannelPrunedLearner does not support channel pruning for fully-connected layers.