Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DL4J: RNN masks sometimes not being applied? #7115

Closed
AlexDBlack opened this issue Feb 5, 2019 · 4 comments

Comments

@AlexDBlack
Copy link
Contributor

commented Feb 5, 2019

As reported in gitter by @C4N4D4M4N

@AlexDBlack I made a test model for the issue I posted earlier where RNN model ignored the masks. It's quite configurable, and works with multiple inputs/labels. Here's the Gist: https://gist.github.com/C4N4D4M4N/7c0596bb751c2144df246d8b375d8334.
The results of the first two tests are identical, but test #1 has no masked data and test #2 has all masked data. Test #3 is with all values masked, and all values set to 0, and it shows that the value of the data, masked or not, is still being considered by the training algorithm.
These tests were done in both beta-2 and beta-3.

AlexDBlack added a commit that referenced this issue Feb 15, 2019
@AlexDBlack

This comment has been minimized.

Copy link
Contributor Author

commented Feb 15, 2019

@C4N4D4M4N Turns out that yes there was an issue here with LastTimeStep layer and masks
I've done two things:
(a) reproduced and fixed the issue
(b) I'de also added a check for the "all masked out" case - an exception will now be thrown if mask is all 0s for a layer (I think that's the only reasonable approach here - there is no last time step if there is no non-masked input data)

Fix is here and will be merged soon: #7169

@AlexDBlack AlexDBlack closed this Feb 15, 2019

@C4N4D4M4N

This comment has been minimized.

Copy link

commented Feb 15, 2019

@AlexDBlack Awesome, thanks a lot for looking into this. How will I know when the fix is applied, will it just be whenever the next DL4J version is released?

@AlexDBlack

This comment has been minimized.

Copy link
Contributor Author

commented Feb 15, 2019

Once merged, the fix will be available on snapshots (it usually takes a few hours to build and upload).
https://deeplearning4j.org/docs/latest/deeplearning4j-config-snapshots

Otherwise yes, it will be in the next release (not sure on the date yet, but more than a few weeks for that most likely).

AlexDBlack added a commit that referenced this issue Feb 15, 2019
[WIP] Misc DL4J/ND4J Fixes (#7169)
* #7152 Yolo2OutputLayer label shape validation

* #7143 empty array checks

* #7123 Update old URLs

* #7115 Fix issue with LastTimeStep and masks

* #6809 Fix updater state view array layout when dynamically changing LR with updaters like Adam

* Delete old/unmaintained VideoRecordReader

* #7104 Fix memory info reporting when activations shape/length can't be inferred (some RNNs etc)
@lock

This comment has been minimized.

Copy link

commented Mar 17, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 17, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.