Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wrong output while using combination of DummyData and Dropout layer #4031

Closed
bharatsau opened this issue Apr 22, 2016 · 4 comments
Closed

Wrong output while using combination of DummyData and Dropout layer #4031

bharatsau opened this issue Apr 22, 2016 · 4 comments

Comments

@bharatsau
Copy link

bharatsau commented Apr 22, 2016 •
edited

I got some strange output while trying the below combination of layers for some purpose:

layer{
name: "dummy_ones"
type: "DummyData"
top: "dummy_ones"
dummy_data_param{
data_filler{
type:"constant" # This is happenning only when using "constant".
value: 1
}
shape:{
dim: 16
dim: 1
}
}
}

layer{
name: "drop1"
type: "Dropout"
bottom: "dummy_ones"
top: "dummy_ones"
dropout_param {
dropout_ratio: 0.5
}
}

Expected output should be combination of 1s and 0s. But I got the below result:

I0423 04:33:47.016577 26114 solver.cpp:228] Iteration 0, loss = 2.34377
I0423 04:33:47.016935 26114 solver.cpp:244] Train net output #0: dummy_ones = 2
I0423 04:33:47.016971 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.017001 26114 solver.cpp:244] Train net output #2: dummy_ones = 2
I0423 04:33:47.017027 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.017053 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.017082 26114 solver.cpp:244] Train net output #5: dummy_ones = 2
I0423 04:33:47.017108 26114 solver.cpp:244] Train net output #6: dummy_ones = 2
I0423 04:33:47.017135 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.017160 26114 solver.cpp:244] Train net output #8: dummy_ones = 2
I0423 04:33:47.017189 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.017216 26114 solver.cpp:244] Train net output #10: dummy_ones = 2
I0423 04:33:47.017256 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.017293 26114 solver.cpp:244] Train net output #12: dummy_ones = 2
I0423 04:33:47.017320 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.017349 26114 solver.cpp:244] Train net output #14: dummy_ones = 2
I0423 04:33:47.017376 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.017475 26114 sgd_solver.cpp:106] Iteration 0, lr = 0.0002
I0423 04:33:47.023126 26114 solver.cpp:228] Iteration 1, loss = 2.30396
I0423 04:33:47.023555 26114 solver.cpp:244] Train net output #0: dummy_ones = 4
I0423 04:33:47.023591 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.023619 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.023644 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.023674 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.023702 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.023731 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.023758 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.023784 26114 solver.cpp:244] Train net output #8: dummy_ones = 4
I0423 04:33:47.023811 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.023836 26114 solver.cpp:244] Train net output #10: dummy_ones = 4
I0423 04:33:47.023862 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.023892 26114 solver.cpp:244] Train net output #12: dummy_ones = 4
I0423 04:33:47.023919 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.023944 26114 solver.cpp:244] Train net output #14: dummy_ones = 4
I0423 04:33:47.023983 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.024073 26114 sgd_solver.cpp:106] Iteration 1, lr = 0.0002
I0423 04:33:47.028427 26114 solver.cpp:228] Iteration 2, loss = 2.36808
I0423 04:33:47.028853 26114 solver.cpp:244] Train net output #0: dummy_ones = 0
I0423 04:33:47.028888 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.028916 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.028942 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.028991 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.029019 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.029044 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.029069 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.029095 26114 solver.cpp:244] Train net output #8: dummy_ones = 8
I0423 04:33:47.029124 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.029150 26114 solver.cpp:244] Train net output #10: dummy_ones = 0
I0423 04:33:47.029176 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.029203 26114 solver.cpp:244] Train net output #12: dummy_ones = 8
I0423 04:33:47.029229 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.029258 26114 solver.cpp:244] Train net output #14: dummy_ones = 0
I0423 04:33:47.029285 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.029393 26114 sgd_solver.cpp:106] Iteration 2, lr = 0.0002
I0423 04:33:47.033825 26114 solver.cpp:228] Iteration 3, loss = 2.27006
I0423 04:33:47.034241 26114 solver.cpp:244] Train net output #0: dummy_ones = 0
I0423 04:33:47.034276 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.034315 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.034343 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.034370 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.034396 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.034423 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.034449 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.034476 26114 solver.cpp:244] Train net output #8: dummy_ones = 16
I0423 04:33:47.034504 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.034531 26114 solver.cpp:244] Train net output #10: dummy_ones = 0
I0423 04:33:47.034557 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.034584 26114 solver.cpp:244] Train net output #12: dummy_ones = 16
I0423 04:33:47.034610 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.034637 26114 solver.cpp:244] Train net output #14: dummy_ones = 0
I0423 04:33:47.034663 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.034751 26114 sgd_solver.cpp:106] Iteration 3, lr = 0.0002
I0423 04:33:47.039185 26114 solver.cpp:228] Iteration 4, loss = 2.17809
I0423 04:33:47.039542 26114 solver.cpp:244] Train net output #0: dummy_ones = 0
I0423 04:33:47.039577 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.039604 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.039631 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.039659 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.039685 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.039711 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.039737 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.039764 26114 solver.cpp:244] Train net output #8: dummy_ones = 32
I0423 04:33:47.039791 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.039819 26114 solver.cpp:244] Train net output #10: dummy_ones = 0
I0423 04:33:47.039844 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.039871 26114 solver.cpp:244] Train net output #12: dummy_ones = 0
I0423 04:33:47.039897 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.039923 26114 solver.cpp:244] Train net output #14: dummy_ones = 0
I0423 04:33:47.039949 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.040038 26114 sgd_solver.cpp:106] Iteration 4, lr = 0.0002
I0423 04:33:47.044420 26114 solver.cpp:228] Iteration 5, loss = 2.17145
I0423 04:33:47.044841 26114 solver.cpp:244] Train net output #0: dummy_ones = 0
I0423 04:33:47.044874 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.044903 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.044929 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.044956 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.044983 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.045017 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.045045 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.045073 26114 solver.cpp:244] Train net output #8: dummy_ones = 0
I0423 04:33:47.045099 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.045125 26114 solver.cpp:244] Train net output #10: dummy_ones = 0
I0423 04:33:47.045151 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.045177 26114 solver.cpp:244] Train net output #12: dummy_ones = 0
I0423 04:33:47.045204 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.045230 26114 solver.cpp:244] Train net output #14: dummy_ones = 0
I0423 04:33:47.045256 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
I0423 04:33:47.045344 26114 sgd_solver.cpp:106] Iteration 5, lr = 0.0002
I0423 04:33:47.049790 26114 solver.cpp:228] Iteration 6, loss = 2.0659
I0423 04:33:47.050205 26114 solver.cpp:244] Train net output #0: dummy_ones = 0
I0423 04:33:47.050240 26114 solver.cpp:244] Train net output #1: dummy_ones = 0
I0423 04:33:47.050267 26114 solver.cpp:244] Train net output #2: dummy_ones = 0
I0423 04:33:47.050303 26114 solver.cpp:244] Train net output #3: dummy_ones = 0
I0423 04:33:47.050331 26114 solver.cpp:244] Train net output #4: dummy_ones = 0
I0423 04:33:47.050359 26114 solver.cpp:244] Train net output #5: dummy_ones = 0
I0423 04:33:47.050385 26114 solver.cpp:244] Train net output #6: dummy_ones = 0
I0423 04:33:47.050412 26114 solver.cpp:244] Train net output #7: dummy_ones = 0
I0423 04:33:47.050439 26114 solver.cpp:244] Train net output #8: dummy_ones = 0
I0423 04:33:47.050465 26114 solver.cpp:244] Train net output #9: dummy_ones = 0
I0423 04:33:47.050493 26114 solver.cpp:244] Train net output #10: dummy_ones = 0
I0423 04:33:47.050519 26114 solver.cpp:244] Train net output #11: dummy_ones = 0
I0423 04:33:47.050545 26114 solver.cpp:244] Train net output #12: dummy_ones = 0
I0423 04:33:47.050572 26114 solver.cpp:244] Train net output #13: dummy_ones = 0
I0423 04:33:47.050598 26114 solver.cpp:244] Train net output #14: dummy_ones = 0
I0423 04:33:47.050624 26114 solver.cpp:244] Train net output #15: dummy_ones = 0
@seanbell
Copy link

No, dropout scales up activations so that the total energy of the layer is the same on average.

@seanbell
Copy link

seanbell commented Apr 22, 2016 •
edited

Also, you can't place an in-place layer after DummyData. Only some layers can support that.

See #2015 (comment)

@bharatsau
Copy link
Author

bharatsau commented Apr 22, 2016 •
edited

Thank you sean for providing the details.
I am getting correct output while using not in-place layer after DummyData. But for "gaussian" dummy values, I am getting correct output even after using in-place layer. So, should not it work for "constant" values also ?

@seanbell
Copy link

No for constant outputs there is an optimization that assumes that the
output is not modified in place.

Look through the implementation of DummyData and you will see.

On Friday, April 22, 2016, bharatsau notifications@github.com wrote:

Thank you seanbell for providing the details.
I am getting correct output while using not in-place layer after
DummyData. But for "gaussian" dummy values, I am getting correct output
even after using in-place layer. So, should not it work for "constant"
values also ?

—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
#4031 (comment)

Cheers,
Sean

(Sent from my phone)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@seanbell @bharatsau