-
Notifications
You must be signed in to change notification settings - Fork 674
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat/ enable partition_json to partition any json #1038
Comments
any news? |
Hey @lironrisk ! |
Over 180 days old but keeping open due to addition to roadmap, as mentioned by @Coniferish |
Hello unstructured Team. Would be interested in the partition_json. |
We, so far, have implemented a helper function that parses the json, wraps it into a list and feeds it Unstructured.IO, so it could be parsed with the current version. utility function
the main function where we receive json and feed to to Unstructured
|
thank you. And the content_for_unstructured can then be parsed by the unstructured partition function in the unstructured pipeline? `runner = LocalRunner(
|
@apmavrin @adrianruchti can you say a little more about your use case for this? It's not clear to me yet how a useful JSON partitioner would behave. In particular:
Any help you can give characterizing the use cases will help in developing a spec for something like this. Grateful for whatever help you can provide :) |
@scanny the use case: 60 mb law documents (xml format). Was trying unstructured xml parser and azure document intelligence. The extraction and recognition of different types is not satisfying with the xml parser. So I parsed the xml with lxml etree and saved it as json. This worked well with azure document intelligence that can partition and chunk json. Would be nice to get the same from unstructured as this would be a cheaper solution. By the way: unstructured converts the documents to json afte Partitioning so why not accepting this format in your partitioner. Do you have any platform I could share the big xml and converted json file in private with you Steve? (If you are interested) |
Hi @adrianruchti you can DM me on the Unstructured Slack channel or reach me at the email on my GitHub profile. No need for a huge document just yet but a modest sized one might be helpful. The big question for me is schema information, like should an XML partitioner take some schema descriptors to determine what to partition and how, or should it be made to do the best it can without any schema information (like partition all text it finds) or possibly both. Just glancing at some LegalXML documents online, it looks like there is a lot of metadata and only a little narrative text. So if you can help me understand what a successful outcome in your case would be maybe that will help us noodle this a bit further. Also if you can give a sense of the diversity of XML-vocabularies/schemas you need to partition, that would help in reasoning about it. One aspect of an approach that occurs to me is applying a legal-document-type-specific XSLT transform to produce a "standardized" XML document that partitions in a well-known way, including adding whatever "extra" metadata you might want on the partitioned elements. Not sure how that fits in but thought I'd mention in the interest of brainstorming :) |
Currently
partition_json
is intended only for deserializing the unstructured JSON outputs/elements and is not included as a file format we accept for partitioning (see here).The goal of this issue is to make
partition_json
work for any JSON file (probably similar to howpartition_xml
works)The text was updated successfully, but these errors were encountered: