[Data] Support JSONL file format #37637

c21 · 2023-07-21T06:43:59Z

Why are these changes needed?

This is to add supporting for JSONL file format. Our read_json and write_json already supports JSONL file format. The actual code change is to support ".jsonl" file extension in read_json, and update documentation and example.

Related issue number

Closes #37611

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Cheng Su <scnju13@gmail.com>

bveeramani

Seems like there's some misalignment in our APIs:

read_json can only read JSONL files
write_json always writes to JSON files

To avoid confusion, maybe we should rename read_json to read_jsonl?

bveeramani · 2023-07-21T17:38:41Z

python/ray/data/read_api.py

@@ -933,10 +933,10 @@ def read_json(
    ignore_missing_paths: bool = False,
    **arrow_json_args,
 ) -> Dataset:
-    """Creates a :class:`~ray.data.Dataset` from JSON files.
+    """Creates a :class:`~ray.data.Dataset` from JSON and JSONL files.


PyArrow's read_json only support JSONL, not normal JSON.

Currently only the line-delimited JSON format is supported.

Suggested change

"""Creates a :class:`~ray.data.Dataset` from JSON and JSONL files.

"""Creates a :class:`~ray.data.Dataset` from JSONL files.

I think it supports both JSONL and JSON format. Essentially JSON is just a line of JSONL.

Tried with the example in https://json.org/example.html:

>>> import ray >>> ds = ray.data.read_json("/Users/chengsu/try/json/input.json") >>> ds Dataset( num_blocks=20, num_rows=1, schema={ glos...: struct<title: string, GlossDiv: struct<title: string, GlossList: struct<GlossEntry: struct<ID: string, SortAs: string, GlossTerm: string, Acronym: string, Abbrev: string, GlossDef: struct<para: string, GlossSeeAlso: list<item: string>>, GlossSee: string>>>> } )

We can document this: For an input json file, the whole file will be read as one row. For an input jsonl file, each line will be read as one row.

Yes, updated.

bveeramani · 2023-07-21T17:40:07Z

python/ray/data/read_api.py


    Examples:
-        Read a file in remote storage.
+        Read a JSON file in remote storage.


logs.json is actually a JSONL file with the JSON extension. To avoid confusion, maybe we should rename logs.json to logs.jsonl? We also don't need to keep both examples because they both read JSONL.

Yeah good catch. I will add a file with a single line of JSON, to represent JSON file.

python/ray/data/tests/test_json.py

bveeramani · 2023-07-21T17:45:38Z

python/ray/data/datasource/json_datasource.py

@@ -13,7 +13,7 @@

 @PublicAPI
 class JSONDatasource(FileBasedDatasource):
-    """JSON datasource, for reading and writing JSON files.
+    """JSON datasource, for reading and writing JSON and JSONL files.


Suggested change

"""JSON datasource, for reading and writing JSON and JSONL files.

"""JSON datasource, for reading JSONL and writing JSON files.

bveeramani · 2023-07-21T17:47:01Z

python/ray/data/dataset.py

@@ -2535,6 +2535,11 @@ def write_json(
            >>> ds = ray.data.range(100)
            >>> ds.write_json("local:///tmp/data")

+            Write the dataset as JSONL files to a local directory.


I'm not sure if we need both of these examples. They both write to JSON format files.

Updated to have two examples, one is to write a JSON file, another is to write multiple JSONL files.

bveeramani · 2023-07-21T17:49:35Z

python/ray/data/dataset.py

@@ -2513,7 +2513,7 @@ def write_json(
        ray_remote_args: Dict[str, Any] = None,
        **pandas_json_args,
    ) -> None:
-        """Writes the :class:`~ray.data.Dataset` to JSON files.
+        """Writes the :class:`~ray.data.Dataset` to JSON and JSONL files.


Looks like we use Pandas' to_json, and that always writes to JSON (not JSONL)?

Suggested change

"""Writes the :class:`~ray.data.Dataset` to JSON and JSONL files.

"""Writes the :class:`~ray.data.Dataset` to JSON files.

I tried ds.write_json("xxx.jsonl") locally, and it did write jsonl files. But I'm not sure it's because the ".jsonl" suffix or something else that makes it to write jsonl files. I think we need to figure it out and also make it clear in the doc.

It's always writing in JSONL format. If it only has a single line, it becomes a JSON file. More context in #37637 (comment) .

Got it. let's update the doc to reflect this?

Sure, updated.

c21 · 2023-07-21T18:43:09Z

@bveeramani, @raulchen, just to add some context. JSONL (https://jsonlines.org/examples/) format is essentially lines of JSON format (https://json.org/example.html).

A JSONL file (multiple lines of JSON):

{"name": "foo", ...}
{"name": "bar", ...}
...

A JSON file (a single line of JSON):

{"name": "foo", ...}

We use Arrow read_json, which supports both JSON and JSONL files. JSON file is a special case of JSONL file anyway - JSON file is a JSONL file with a single line.

We use Pandas to_json to write JSON and JSONL files. We set lines=True, so if there're multiple lines of JSON, each JSON will be written in separate line (JSONL format).

c21 · 2023-07-21T18:52:41Z

To avoid confusion, maybe we should rename read_json to read_jsonl?

Given all other libraries (Arrow, Pandas) are using read_json, to_json. I would love us to stick with read_json and avoid breaking API naming change as well. Does it make sense?

Signed-off-by: Cheng Su <scnju13@gmail.com>

raulchen · 2023-07-21T20:15:47Z

python/ray/data/dataset.py

@@ -2513,7 +2513,7 @@ def write_json(
        ray_remote_args: Dict[str, Any] = None,
        **pandas_json_args,
    ) -> None:
-        """Writes the :class:`~ray.data.Dataset` to JSON files.
+        """Writes the :class:`~ray.data.Dataset` to JSON and JSONL files.


Got it. let's update the doc to reflect this?

raulchen · 2023-07-21T20:20:14Z

python/ray/data/read_api.py

@@ -933,10 +933,10 @@ def read_json(
    ignore_missing_paths: bool = False,
    **arrow_json_args,
 ) -> Dataset:
-    """Creates a :class:`~ray.data.Dataset` from JSON files.
+    """Creates a :class:`~ray.data.Dataset` from JSON and JSONL files.


We can document this: For an input json file, the whole file will be read as one row. For an input jsonl file, each line will be read as one row.

Signed-off-by: Cheng Su <scnju13@gmail.com>

bveeramani · 2023-07-21T22:56:40Z

python/ray/data/dataset.py

+        When the dataset has multiple rows, the output file is in JSONL format.
+        Otherwise, the output file is in JSON format.


Nit: IMO it's better to exclude this information, as well the example of writing to a single JSON file. How often will users work with single-row datasets? I'm concerned this would introduce confusion, and it'd be easier to understand "write_json always writes JSONL".

okay I agree single-row dataset would be very rare. Removed these two lines.

Signed-off-by: Cheng Su <scnju13@gmail.com>

This is to add supporting for JSONL file format. Our `read_json` and `write_json` already supports JSONL file format. The actual code change is to support ".jsonl" file extension in `read_json`, and update documentation and example. Signed-off-by: Cheng Su <scnju13@gmail.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>

This is to add supporting for JSONL file format. Our `read_json` and `write_json` already supports JSONL file format. The actual code change is to support ".jsonl" file extension in `read_json`, and update documentation and example. Signed-off-by: Cheng Su <scnju13@gmail.com> Signed-off-by: harborn <gangsheng.wu@intel.com>

This is to add supporting for JSONL file format. Our `read_json` and `write_json` already supports JSONL file format. The actual code change is to support ".jsonl" file extension in `read_json`, and update documentation and example. Signed-off-by: Cheng Su <scnju13@gmail.com>

This is to add supporting for JSONL file format. Our `read_json` and `write_json` already supports JSONL file format. The actual code change is to support ".jsonl" file extension in `read_json`, and update documentation and example. Signed-off-by: Cheng Su <scnju13@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

This is to add supporting for JSONL file format. Our `read_json` and `write_json` already supports JSONL file format. The actual code change is to support ".jsonl" file extension in `read_json`, and update documentation and example. Signed-off-by: Cheng Su <scnju13@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>

Support JSONL file format

20690d5

Signed-off-by: Cheng Su <scnju13@gmail.com>

c21 requested review from ericl, scv119, amogkam, scottjlee, bveeramani and raulchen as code owners July 21, 2023 06:43

c21 assigned raulchen and bveeramani Jul 21, 2023

bveeramani reviewed Jul 21, 2023

View reviewed changes

Address comments for examples

831a06e

Signed-off-by: Cheng Su <scnju13@gmail.com>

raulchen approved these changes Jul 21, 2023

View reviewed changes

Address comments

6800f45

Signed-off-by: Cheng Su <scnju13@gmail.com>

bveeramani approved these changes Jul 21, 2023

View reviewed changes

Address comment

2a7b8a4

Signed-off-by: Cheng Su <scnju13@gmail.com>

c21 merged commit 2bc97f0 into ray-project:master Jul 22, 2023
59 of 62 checks passed

c21 deleted the jsonl branch July 22, 2023 21:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Data] Support JSONL file format #37637

[Data] Support JSONL file format #37637

c21 commented Jul 21, 2023

bveeramani left a comment

bveeramani Jul 21, 2023

c21 Jul 21, 2023

raulchen Jul 21, 2023

c21 Jul 21, 2023

bveeramani Jul 21, 2023

c21 Jul 21, 2023

bveeramani Jul 21, 2023 •

edited

bveeramani Jul 21, 2023

c21 Jul 21, 2023 •

edited

bveeramani Jul 21, 2023

raulchen Jul 21, 2023

c21 Jul 21, 2023

raulchen Jul 21, 2023

c21 Jul 21, 2023

c21 commented Jul 21, 2023

c21 commented Jul 21, 2023

raulchen Jul 21, 2023

raulchen Jul 21, 2023

bveeramani Jul 21, 2023

c21 Jul 21, 2023

	"""Creates a :class:`~ray.data.Dataset` from JSON and JSONL files.
	"""Creates a :class:`~ray.data.Dataset` from JSONL files.

	"""JSON datasource, for reading and writing JSON and JSONL files.
	"""JSON datasource, for reading JSONL and writing JSON files.

	"""Writes the :class:`~ray.data.Dataset` to JSON and JSONL files.
	"""Writes the :class:`~ray.data.Dataset` to JSON files.

		When the dataset has multiple rows, the output file is in JSONL format.
		Otherwise, the output file is in JSON format.

[Data] Support JSONL file format #37637

[Data] Support JSONL file format #37637

Conversation

c21 commented Jul 21, 2023

Why are these changes needed?

Related issue number

Checks

bveeramani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bveeramani Jul 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c21 Jul 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c21 commented Jul 21, 2023

c21 commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bveeramani Jul 21, 2023 •

edited

c21 Jul 21, 2023 •

edited