langchain: adds recursive json splitter

[joelsprunger]

• Description: This adds a recursive json splitter class to the existing text_splitters as well as unit tests
• Issue: splitting text from structured data can cause issues if you have a large nested json object and you split it as regular text you may end up losing the structure of the json. To mitigate against this you can split the nested json into large chunks and overlap them, but this causes unnecessary text processing and there will still be times where the nested json is so big that the chunks get separated from the parent keys.

As an example you wouldn't want the following to be split in half:

{'val0': 'DFWeNdWhapbR',
 'val1': {'val10': 'QdJo',
          'val11': 'FWSDVFHClW',
          'val12': 'bkVnXMMlTiQh',
          'val13': 'tdDMKRrOY',
          'val14': 'zybPALvL',
          'val15': 'JMzGMNH',
          'val16': {'val160': 'qLuLKusFw',
                    'val161': 'DGuotLh',
                    'val162': 'KztlcSBropT',
-----------------------------------------------------------------------split-----
                    'val163': 'YlHHDrN',
                    'val164': 'CtzsxlGBZKf',
                    'val165': 'bXzhcrWLmBFp',
                    'val166': 'zZAqC',
                    'val167': 'ZtyWno',
                    'val168': 'nQQZRsLnaBhb',
                    'val169': 'gSpMbJwA'},
          'val17': 'JhgiyF',
          'val18': 'aJaqjUSFFrI',
          'val19': 'glqNSvoyxdg'}}

Any llm processing the second chunk of text may not have the context of val1, and val16 reducing accuracy. Embeddings will also lack this context and this makes retrieval less accurate.

Instead you want it to be split into chunks that retain the json structure.

{'val0': 'DFWeNdWhapbR',
 'val1': {'val10': 'QdJo',
          'val11': 'FWSDVFHClW',
          'val12': 'bkVnXMMlTiQh',
          'val13': 'tdDMKRrOY',
          'val14': 'zybPALvL',
          'val15': 'JMzGMNH',
          'val16': {'val160': 'qLuLKusFw',
                    'val161': 'DGuotLh',
                    'val162': 'KztlcSBropT',
                    'val163': 'YlHHDrN',
                    'val164': 'CtzsxlGBZKf'}}}

and

{'val1':{'val16':{
                    'val165': 'bXzhcrWLmBFp',
                    'val166': 'zZAqC',
                    'val167': 'ZtyWno',
                    'val168': 'nQQZRsLnaBhb',
                    'val169': 'gSpMbJwA'},
          'val17': 'JhgiyF',
          'val18': 'aJaqjUSFFrI',
          'val19': 'glqNSvoyxdg'}}

This recursive json text splitter does this. Values that contain a list can be converted to dict first by using split(... convert_lists=True) otherwise long lists will not be split and you may end up with chunks larger than the max chunk.

In my testing large json objects could be split into small chunks with
✅ Increased question answering accuracy
✅ The ability to split into smaller chunks meant retrieval queries can use fewer tokens

• Dependencies: json import added to text_splitter.py, and random added to the unit test
• Twitter handle: @joelsprunger

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 8, 2024 9:44pm

[joelsprunger]

I've added an ipynb notebook and I'm trying to get the documentation built locally.

[joelsprunger]

I think we are good here. I was able to run the unit tests, linting, and see the documentation.

Let me know if I need to do anything else. :-)

[hwchase17]

really really like this - one large-ish comment/refactor to make more consistent with other classes

[hwchase17]

i dont think this class should be stateful - pretty inconsistent with other classes

[joelsprunger]

I get that. I think I initially coded it up for my own use. And with the chunks passed to 3 of the functions in the class I was thinking it made more sense to just have those be a class member. I can change that.

[joelsprunger]

I'll try to have it inherit from TextSplitter, and change split to split_text so that create_documents() can work.

[joelsprunger]

The issue I am seeing is with create_documents()

The super is expecting List[str]
I wanted to keep the loaded json as a Dict rather than expecting str because the recursion for preprocessing and splitting is on Dict objects. And anyone with json in python probably has it in a Dict format to start. I think I will stop inheriting from TextSplitter, and instead just shaddow the functionality with split_text, and create_documents, with that said I'll add split_json that returns List[dict] so that people can get the output in json if they want. Then split_text and create_docs can be wrappers around split_json.

[joelsprunger]

@hwchase17 I refactored this and updated the docs and unit tests.

[hwchase17]

thanks @joelsprunger ! will highlight this when its out in next release. really like it!

[joelsprunger]

@hwchase17 great! If you want me to demo some of the benefits with example json using langsmith I can make a screen-cast or something.

[funkymonkeymonk]

So I just found this in the documentation and it took me a bit of spelunking to find that this was not yet released because it's already in the documentation. Thanks for making this just in time for me to need it :-)

[joelsprunger]

@hwchase17 were you planning to highlight this feature in some way?