Skip to content

Add decode_json support#695

Merged
terrytangyuan merged 1 commit intotensorflow:masterfrom
yongtang:decode_json
Dec 18, 2019
Merged

Add decode_json support#695
terrytangyuan merged 1 commit intotensorflow:masterfrom
yongtang:decode_json

Conversation

@yongtang
Copy link
Copy Markdown
Member

This PR adds decode_json support so that it is possible to use

values = tfio.experimental.serialization.decode_json(json, specs)

to decode a JSON string into a structured Tensors.

The specs field is the structured tf.TensorSpec that is necessary
to reconstruct the JSON values. The reason being JSON could be intepreted
differently in lists:

{
"v": [1, 2, 3]
}

could be considered as one Tensor of shape (3), or a 3 Tensor structure within dict "v".

Note structured tf.TensorSpecs need to change list [] into tuple ()
as list is not supported in structures in TensorFlow.

This PR fixes #692.

Signed-off-by: Yong Tang yong.tang.github@outlook.com

This PR adds decode_json support so that it is possible to use
```
values = tfio.experimental.serialization.decode_json(json, specs)
```
to decode a JSON string into a structured Tensors.

The specs field is the structured tf.TensorSpec that is necessary
to reconstruct the JSON values. The reason being JSON could be intepreted
differently in lists:
```
{
"v": [1, 2, 3]
}
```
could be considered as one Tensor of shape `(3)`, or a 3 Tensor structure within dict "v".

Note structured tf.TensorSpecs need to change list `[]` into tuple `()`
as list is not supported in structures in TensorFlow.

This PR fixes 692.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@yongtang
Copy link
Copy Markdown
Member Author

/cc @noahtren

@terrytangyuan terrytangyuan merged commit 3ebe2d1 into tensorflow:master Dec 18, 2019
@noahtren
Copy link
Copy Markdown

Thanks so much! This is a huge help to my pipeline which includes JSON alongside a lot of other data types. So helpful to have a function to turn these into tensors.

@yongtang yongtang deleted the decode_json branch December 21, 2019 18:25
@djl11
Copy link
Copy Markdown

djl11 commented Aug 29, 2020

The following example does not parse the json correctly:

core_ops.io_decode_json(tf.constant('{"x":[[1.0]]}'), ['/x'], [tf.float32])

Gives:

[<tf.Tensor: shape=(1,), dtype=float32, numpy=array([4.2949673e+09], dtype=float32)>]

However, 1-D works, for example:

core_ops.io_decode_json(tf.constant('{"x":[1.0, 2.0]}'), ['/x'], [tf.float32])

Gives:

[<tf.Tensor: shape=(2,), dtype=float32, numpy=array([1., 2.], dtype=float32)>]

Is this desired behaviour?
I am not sure how to load a multi-dimensional tensor stored in a json file as a nested list.

@yongtang
Copy link
Copy Markdown
Member Author

@djl11 A PR #1101 has been opened to support the 2-D tensor in your example.

@djl11
Copy link
Copy Markdown

djl11 commented Aug 31, 2020

I think with recursion the multi-dimensional case shouldn't be too hard, unless I'm missing something. If I find the time I'll try and implement this myself. Thanks for putting in the new PR @yongtang!

i-ony pushed a commit to i-ony/io that referenced this pull request Feb 8, 2021
This PR adds decode_json support so that it is possible to use
```
values = tfio.experimental.serialization.decode_json(json, specs)
```
to decode a JSON string into a structured Tensors.

The specs field is the structured tf.TensorSpec that is necessary
to reconstruct the JSON values. The reason being JSON could be intepreted
differently in lists:
```
{
"v": [1, 2, 3]
}
```
could be considered as one Tensor of shape `(3)`, or a 3 Tensor structure within dict "v".

Note structured tf.TensorSpecs need to change list `[]` into tuple `()`
as list is not supported in structures in TensorFlow.

This PR fixes 692.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support decode_json to decode json into Tensor

4 participants