Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] POC for reading multi-line JSON in string columns #15520

Draft
wants to merge 1 commit into
base: branch-24.06
Choose a base branch
from

Conversation

shrshi
Copy link
Contributor

@shrshi shrshi commented Apr 12, 2024

Description

Addresses #15277
Given a JSON lines buffer with records separated by a delimiter defined at runtime, the idea is as follows -

  1. Modify the JSON tokenization FST to consider the passed delimiter to generate EOL token instead of the newline character currently hard-coded.
  2. Modify the whitespace normalization FST to strip out unquoted \n and \r. Invoke the FST as a pre-processing step.

Note that this is not a multi-object JSON reader since we are not using the offsets data in the string column, and hence there is no resetting of the start state at every row offset.

Current status:

  • Example FST that implements runtime configurable transition characters
  • Adding transitions to handle unquoted newlines and carriage returns in whitespace normalization FST

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Apr 12, 2024
@shrshi shrshi added feature request New feature or request non-breaking Non-breaking change labels Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CMake CMake build issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

1 participant