Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compose Custom Loss with Softmax #2677

Closed
RishabGargeya opened this issue Jul 12, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@RishabGargeya
Copy link

commented Jul 12, 2016

Hello!

I'm trying to create a custom loss function using symbolic variables. The loss function is the sum of the softmax error and other values computed from internal layers. I know I cannot achieve this with SoftmaxOutput so I am using SoftmaxActivation as my activation function after the final fc layer and softmax_cross_entropy as the function to compute softmax loss stacked with MakeLoss. However, I am unable to run this pipeline without getting an error:

Check failed: (this->shape_.Size()) == (shape.Size()) TBlob.get_with_shape: new and old shape do not match total elements

I know there is a shape mismatch but I'm not sure what the issue is

Here is some sample code showing the basic structure I want to achieve with the network:

data = mx.symbol.Variable(name='data')
fc1 = mx.symbol.FullyConnected(data=data, num_hidden=num_hidden, name='fc1')
act = mx.symbol.Activation(data=fc1, act_type='sigmoid')
fc2 = mx.symbol.FullyConnected(data=act, num_hidden=2, name='fc2')
label = mx.symbol.Variable(name='softmax_label')
softmax_activation = mx.symbol.SoftmaxActivation(data=fc2)
loss_1 = mx.symbol.softmax_cross_entropy(lhs=softmax_activation, rhs=label)
loss_2 = mx.symbol.sum(ms.symbol.square(fc1 - 0.5))
return mx.symbol.MakeLoss(data=loss_1+loss_2, name='makeloss')

Does this make sense? I'm pretty sure this is possible, but I'm not sure where I went wrong.

On its own, the following does not compile:

data = mx.symbol.Variable(name='data')
fc1 = mx.symbol.FullyConnected(data=act, num_hidden=2, name='fc1')
label = mx.symbol.Variable(name='softmax_label')
softmax_activation = mx.symbol.SoftmaxActivation(data=fc1)
loss_1 = mx.symbol.softmax_cross_entropy(lhs=softmax_activation, rhs=label)
return mx.symbol.MakeLoss(data=loss_1, name='makeloss')

while this does:

data = mx.symbol.Variable(name='data')
fc1 = mx.symbol.FullyConnected(data=act, num_hidden=2, name='fc1')
label = mx.symbol.Variable(name='softmax_label')
return mx.symbol.SoftmaxOutput(data=fc1, label=label)
@piiswrong

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2016

You can use softmaxoutput for this. Multiple outputs are summed during backward

@RishabGargeya

This comment has been minimized.

Copy link
Author

commented Jul 13, 2016

Hello,

Thank you for the response! However, I am still a bit confused.

My aimed objective function is : softmax loss + custom value computed on internal fc layer

SoftmaxOutput does not output the raw loss value, so I cannot add this to another custom value and run MakeLoss on that? Or are you saying I don't need to explicitly add these symbols together

@piiswrong

This comment has been minimized.

Copy link
Contributor

commented Jul 13, 2016

You don't need makeloss for simple addition between objectives. Use mx.sym.Group and they are implicitly added during backprop.
You just need to compute metric correctly for logging

@Liuxg16

This comment has been minimized.

Copy link

commented Aug 25, 2016

@RishabGargeya have you figure out this problem? I have the same question....

@yajiedesign

This comment has been minimized.

Copy link
Contributor

commented Sep 28, 2017

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.