Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reading in multiple JSON files using glob (wildcard) syntax #3450

Closed
EvanZ opened this issue Sep 20, 2021 · 6 comments · Fixed by #6873
Closed

Reading in multiple JSON files using glob (wildcard) syntax #3450

EvanZ opened this issue Sep 20, 2021 · 6 comments · Fixed by #6873
Labels
External Pull requests and issues from people who do not regularly contribute to modin new feature/request 💬 Requests and pull requests for new features P2 Minor bugs or low-priority feature requests pandas.io

Comments

@EvanZ
Copy link

EvanZ commented Sep 20, 2021

There was a similar feature request (#2662) that solved this problem for CSV files. For my use case I am trying to read hundreds or thousands of jsonlines files spit out by a Spark ETL process. In some cases we could convert to csv, but a lot of our datasets have array or vectorized features, which are typically easier to deal with as json.

Discussed here:

https://discuss.modin.org/t/read-multiple-csv-files/157/4?u=evanz
@EvanZ EvanZ added the new feature/request 💬 Requests and pull requests for new features label Sep 20, 2021
@modin-bot
Copy link

This issue has been mentioned on Modin Discuss. There might be relevant details there:

https://discuss.modin.org/t/read-multiple-csv-files/157/8

@devin-petersohn
Copy link
Collaborator

Thanks @EvanZ for the request!

cc @anmyachev. I don't think it would be too complicated or too much of a deviation from the CSV (or maybe even pickle) versions.

@anmyachev
Copy link
Collaborator

@devin-petersohn agree. I will take over.

@anmyachev anmyachev self-assigned this Sep 21, 2021
@anmyachev anmyachev mentioned this issue Nov 15, 2021
Closed
7 tasks
anmyachev added a commit to anmyachev/modin that referenced this issue Nov 17, 2021
@anmyachev
FEAT-modin-project#3450: reading multiple JSON files using glob
6b99c1f 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
anmyachev added a commit to anmyachev/modin that referenced this issue Nov 23, 2021
@anmyachev
FEAT-modin-project#3450: address review comments
f4643c1 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
@RehanSD
Copy link
Collaborator

RehanSD commented Oct 12, 2022

@anmyachev has this issue been resolved?

@mvashishtha mvashishtha added P2 Minor bugs or low-priority feature requests P3 Very minor bugs, or features we can hopefully add some day. and removed P3 Very minor bugs, or features we can hopefully add some day. labels Oct 12, 2022
@anmyachev
Copy link
Collaborator

@anmyachev has this issue been resolved?

@RehanSD not yet

@mvashishtha mvashishtha added pandas.io P3 Very minor bugs, or features we can hopefully add some day. and removed P2 Minor bugs or low-priority feature requests labels Oct 12, 2022
@anmyachev anmyachev added the External Pull requests and issues from people who do not regularly contribute to modin label Apr 19, 2023
@anmyachev anmyachev added P2 Minor bugs or low-priority feature requests and removed P3 Very minor bugs, or features we can hopefully add some day. labels Dec 20, 2023
@anmyachev
Copy link
Collaborator

Also requested in https://modin-project.slack.com/archives/C02MY2YV5MK/p1703075320557569

anmyachev added a commit to anmyachev/modin that referenced this issue Jan 22, 2024
@anmyachev
FEAT-modin-project#3450: Implement read_json_glob and to_json_glob
10594c9 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
@anmyachev anmyachev mentioned this issue Jan 22, 2024
Merged
7 tasks
anmyachev added a commit to anmyachev/modin that referenced this issue Jan 22, 2024
@anmyachev
FEAT-modin-project#3450: Implement read_json_glob and to_json_glob
9e77b2e 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
anmyachev added a commit to anmyachev/modin that referenced this issue Jan 22, 2024
@anmyachev
FEAT-modin-project#3450: Implement read_json_glob and to_json_glob
b949408 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
YarShev pushed a commit that referenced this issue Jan 23, 2024
@anmyachev
FEAT-#3450: Implement read_json_glob and to_json_glob (#6873)
23ee584 
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
External Pull requests and issues from people who do not regularly contribute to modin new feature/request 💬 Requests and pull requests for new features P2 Minor bugs or low-priority feature requests pandas.io
Projects
None yet
Milestone
No milestone
6 participants
@EvanZ @mvashishtha @devin-petersohn @RehanSD @anmyachev @modin-bot