Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialize exampleMetaData by calling DataSet.save(...) & MultiDataSet.save(...) #4557

Closed
amoellmann opened this issue Jan 26, 2018 · 1 comment

Comments

@amoellmann
Copy link

commented Jan 26, 2018

Feature request

Serialize exampleMetaData by calling DataSet.save(...) and MultiDataSet.save(...)

  @Test
    public void testDataSetMetaDataSerialization() throws IOException {
        System.out.println("testDataSetMetaDataSerialization");
        // create simple data set with meta data object
        INDArray f = Nd4j.create(1);
        INDArray l = Nd4j.create(1);
        List<String> metaData = new ArrayList<>();
        metaData.add("1");
        DataSet ds = new DataSet(f,l);
        ds.setExampleMetaData(metaData);

        // check if the meta data was serialized and deserialized
        Path tmp = Files.createTempFile(Paths.get("/tmp"), "", "");
        ds.save(tmp.toFile());
        DataSet loaded = new DataSet();
        loaded.load(tmp.toFile());
        Files.deleteIfExists(tmp);

        /*
        fails: serialization not supported
        https://github.com/deeplearning4j/nd4j/blob/6eb6081ed9c4a3b65d578b5cdb1c7eb4f8d960d5/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/dataset/DataSet.java#L275-L274
        */
        // assertNotNull(loaded.getExampleMetaData());
    }

    @Test
    public void testMultiDataSetMetaDataSerialization() throws IOException {
        System.out.println("testMultiDataSetMetaDataSerialization");
        // create simple data set with meta data object
        INDArray f = Nd4j.create(1);
        INDArray l = Nd4j.create(1);
        List<String> metaData = new ArrayList<>();
        metaData.add("1");
        MultiDataSet ds = new MultiDataSet(f,l);
        ds.setExampleMetaData(metaData);

        // check if the meta data was serialized and deserialized
        Path tmp = Files.createTempFile(Paths.get("/tmp"), "", "");
        ds.save(tmp.toFile());
        DataSet loaded = new DataSet();
        loaded.load(tmp.toFile());
        Files.deleteIfExists(tmp);

        /*
        fails: serialization not supported
        https://github.com/deeplearning4j/nd4j/blob/6eb6081ed9c4a3b65d578b5cdb1c7eb4f8d960d5/nd4j-backends/nd4j-api-parent/nd4j-api/src/main/java/org/nd4j/linalg/dataset/DataSet.java#L275-L274
        */
        // assertNotNull(loaded.getExampleMetaData());
    }

Version Information

  • Deeplearning4j version: 0.9.1
AlexDBlack added a commit that referenced this issue Jul 25, 2018
AlexDBlack added a commit that referenced this issue Jul 25, 2018
DL4J/ND4J Various Fixes (#5963)
* #5913 ND4J CPU OOM - throw OutOfMemoryError instead of IllegalStateException

* #5960 EmbeddingSequenceLayer now supports rank 3 [mb,1,tsLength] input

* #5205 Handle batchSize config >> total examples condition to avoid OOM on list size

* Deprecated functionality cleanup

* Clean up some more long-deprecated and unused functionality; deprecated javadoc improvements

* #4866 CUDNN conv dilation > 2 fix

* Minor test polish

* #5056 implement Yolo2OutputLayer.computeScoreForExamples

* #4557 save metadata in (Multi)DataSet.save(...)
@lock

This comment has been minimized.

Copy link

commented Sep 21, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot unassigned nkwoh01 Sep 21, 2018

@lock lock bot locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
3 participants
You can’t perform that action at this time.