Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SequenceRecordReaderDataSetIterator handles negative labelIndex incorrectly #5966

Closed
alrutten opened this issue Jul 25, 2018 · 2 comments

Comments

@alrutten
Copy link

commented Jul 25, 2018

in this if statement (line 197-216 in SequenceRecordReaderDataSetIterator.java) the if (labelIndex < 0) will never be evaluated:

 if (labelIndex == 0 || labelIndex == totalSizeF - 1) {
                //Labels are first or last -> one input in underlying
                int inputFrom;
                int inputTo;
                **if (labelIndex < 0) {**
                    //No label
                    inputFrom = 0;
                    inputTo = totalSizeF - 1;
                } else if (labelIndex == 0) {
                    inputFrom = 1;
                    inputTo = totalSizeF - 1;
                } else {
                    inputFrom = 0;
                    inputTo = labelIndex - 1;
                }

                builder.addInput(READER_KEY, inputFrom, inputTo);

                underlyingIsDisjoint = false;
            }

which i think means that the builder will .addInput(readername,0,-2), which seems to lead to a shape error (reader is the same CSVSequenceRecordReader in both code snippets):

val iterator = new SequenceRecordReaderDataSetIterator(reader,1,13,-1)
iterator.next(1)
Exception in thread "main" org.nd4j.linalg.exception.ND4JIllegalStateException: Invalid shape: Requested INDArray shape [1, 0, 50] contains dimension size values < 1 (all dimensions must be 1 or more)
	at org.nd4j.linalg.factory.Nd4j.checkShapeValues(Nd4j.java:5593)
	at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:5563)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesSequence(RecordReaderMultiDataSetIterator.java:598)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:368)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:326)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:212)
	at org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator.next(SequenceRecordReaderDataSetIterator.java:361)

val iterator = new RecordReaderMultiDataSetIterator.Builder(1)
		                    .addSequenceReader("features",reader)
		                    .addInput("features",0,-2)
	    	                   .build()
iterator.next(1)
Exception in thread "main" org.nd4j.linalg.exception.ND4JIllegalStateException: Invalid shape: Requested INDArray shape [1, 0, 50] contains dimension size values < 1 (all dimensions must be 1 or more)
	at org.nd4j.linalg.factory.Nd4j.checkShapeValues(Nd4j.java:5593)
	at org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:5563)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesSequence(RecordReaderMultiDataSetIterator.java:598)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:368)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:326)
	at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:212)

expected behaviour:

val iterator = new RecordReaderMultiDataSetIterator.Builder(1)
		                    .addSequenceReader("features",reader)
		                    .addInput("features",0,0)
	    	                   .build()
iterator.next(1)
MultiDataSet: 1 input arrays, 0 label arrays, 0 input masks, 0 label masks
=== INPUT 0 ===
[[[    1.0000,   82.0000,   48.0000,   10.0000,   10.0000,    3.0000,   82.0000,    1.0000,   10.0000,   82.0000,    1.0000,   48.0000,   48.0000,   82.0000,    1.0000,   10.0000,    1.0000,   82.0000,   48.0000,    1.0000,   82.0000,   10.0000, 2872.0000,   82.0000,    3.0000,   10.0000,    1.0000,   48.0000,    1.0000,   82.0000,    3.0000,   10.0000,    3.0000,   82.0000,   10.0000,   48.0000,    1.0000,   10.0000,    3.0000,   48.0000,    1.0000,   82.0000,   10.0000,   48.0000,   82.0000,    1.0000,   82.0000,    3.0000,   48.0000,    1.0000]]]

@AlexDBlack AlexDBlack added this to the DL4J/Arbiter/DataVec Next Steps milestone Jul 25, 2018

@AlexDBlack AlexDBlack self-assigned this Jul 25, 2018

AlexDBlack added a commit that referenced this issue Jul 26, 2018
AlexDBlack added a commit that referenced this issue Jul 26, 2018
@AlexDBlack

This comment has been minimized.

Copy link
Contributor

commented Jul 26, 2018

Fixed here: #5970
Thanks for reporting.

@lock

This comment has been minimized.

Copy link

commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
2 participants
You can’t perform that action at this time.