New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SameDiff: Serialization (/loading flatbuffers in Java) #5759

Closed
AlexDBlack opened this Issue Jun 30, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@AlexDBlack
Copy link
Member

AlexDBlack commented Jun 30, 2018

Use case: I want to save/serialize a SameDiff instance graph/op structure - without the arrays - and in a format that won't break on every version upgrade.

(Why: this would allow what I would call "true" lambda layers in DL4J: a neat but non-critical feature... defining no-params layers as a lambda class should be possible now (pending a PR merge) but the JSON serialization clearly won't work here. However, if I can write a custom JSON serializer that handles the SameDiff instance serialization, even in a binary format - that would be fine).

Currently, the only serialization format we support is FlatBuffers - via SameDiff.asFlatBuffers.
However, this is inadequate for the above use case for 2 reasons:

  1. Arrays are always included
  2. As far as I can see: there is no way to create a new SameDiff instance from a serialized FlatBuffers version.
@AlexDBlack

This comment has been minimized.

Copy link
Member

AlexDBlack commented Jun 30, 2018

Discussed in dev channel: adding support to load the FlatBuffers format we already have seems like a reasonable way forward here.

Adding support to not serialize the arrays should also be easy.

@AlexDBlack AlexDBlack changed the title SameDiff: Serialization SameDiff: Serialization (/loading flatbuffers in Java) Jun 30, 2018

@raver119

This comment has been minimized.

Copy link
Contributor

raver119 commented Jun 30, 2018

Arrays are always included mostly due to const existence. Arrays, which are not outputs of existing ops must be preserved.

@lock

This comment has been minimized.

Copy link

lock bot commented Nov 2, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Nov 2, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.