JSON loader #4067
JSON loader #4067
Conversation
There was a problem hiding this comment.
nice, thanks @avsolatorio for adding! could we add a brief demo notebook (see docs/modules/indexes/document_loaders/examples) and integration test (tests/integration_tests/document_loaders)?
|
|
||
| for i, text in enumerate(body_text, 1): | ||
| metadata = dict( | ||
| source=self.file_path.as_posix(), |
There was a problem hiding this comment.
If it's as posix, how would that impact systems running on Windows? Any reason to choose that vs. str(self.file_path)?
There was a problem hiding this comment.
@vowelparrot good point! Will update the PR. Thanks!
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Hi, @dev2049 , I have added these in the current version of the PR. I also implemented a few functionalities allowing the user a more expressive and explicit way to add metadata from the JSON file. |
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
| # and prevent the user from getting a cryptic error later on. | ||
| if self._content_key is not None: | ||
| sample = data.first() | ||
| assert isinstance(sample, dict) |
There was a problem hiding this comment.
nit-picky point, but convention in package so far has been to more explicitly validate and raise errors (usually ValueErrors). so something like
if not isinstance(sample, dict):
raise ValueError(f'expected foo bar but received baz: {sample}')Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
| import jq # noqa:F401 | ||
| except ImportError: | ||
| raise ValueError( | ||
| "jq package not found, please install it with " "`pipenv install jq`" |
There was a problem hiding this comment.
nit: combine strings and would just do pip (instead of pipenv, which is bit less universal)
There was a problem hiding this comment.
Oh, right! I also had to fix some linting issues so I just included this change in the latest commit. Thanks!
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
This implements a loader of text passages in JSON format. The
jqsyntax is used to define a schema for accessing the relevant contents from the JSON file. This requires dependency on thejqpackage: https://pypi.org/project/jq/.