feat (documents): add a source code loader based on AST manipulation

[cristobalcl]

Summary

A new approach to loading source code is implemented:

Each top-level function and class in the code is loaded into separate documents. Then, an additional document is created with the top-level code, but without the already loaded functions and classes.

This could improve the accuracy of QA chains over source code.

For instance, having this script:

class MyClass:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, {self.name}!")

def main():
    name = input("Enter your name: ")
    obj = MyClass(name)
    obj.greet()

if __name__ == '__main__':
    main()

The loader will create three documents with this content:

First document:

class MyClass:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, {self.name}!")

Second document:

def main():
    name = input("Enter your name: ")
    obj = MyClass(name)
    obj.greet()

Third document:

# Code for: class MyClass:

# Code for: def main():

if __name__ == '__main__':
    main()

A threshold parameter is added to control whether small scripts are split in this way or not.

At this moment, only Python and JavaScript are supported. The appropriate parser is determined by examining the file extension.

Tests

This PR adds:

• Unit tests
• Integration tests

Dependencies

Only one dependency was added as optional (needed for the JavaScript parser).

Documentation

A notebook is added showing how the loader can be used.

Who can review?

@eyurtsev @hwchase17

[vercel]

@cristobalcl is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

[hwchase17]

@rlancemartin we have the concept of parsers - how does this relate to that?

[hwchase17]

cc @eyurtsev for his thoughts as well

[cristobalcl]

@rlancemartin we have the concept of parsers - how does this relate to that?

AFAIK parsers are used in LangChain to process the output of a model response and convert it to a Python struct.

In the context of this PR, parser refers to the manipulation of a source code written in a programming language, in order to separate chunks of text in a meaningful way.

Any suggestion about naming is welcome 😄

[dev2049]

@rlancemartin we have the concept of parsers - how does this relate to that?

@hwchase17 you're referring to BlobParsers here, right (not OutputParser)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)			Jun 27, 2023 9:38pm

[rlancemartin]

Nice work! Interesting PR.

Suggestion: we can re-organize the code slightly to fit into a loader -> parser workflow as we do here.

(1) You should be able to use existing FileSystemBlobLoader here to load the .js or .py files. No new code should be required for loading.

(2) Then, just move your parsing logic into a new parser file in the directory here.

The UX is simple and we reduce code duplication (no new loading code). Specifically, the UX can follow what we do here where the loader and parser are called:

# Transcribe the videos to text
loader = GenericLoader(YoutubeAudioLoader(urls, save_dir), OpenAIWhisperParser())
docs = loader.load()

In this case, of course it will look like:

# Parse js or py to docs
loader = GenericLoader(FileSystemBlobLoader(path="",glob=""), <New_Code_Parser>())
docs = loader.load()

Thoughts?

[cristobalcl]

Oh, right! That makes much more sense. I wasn't familiar with GenericLoader, and I wasn't sure how parsers were used. Now that's clear.

Let me refactor the code.

[cristobalcl]

Done! It's much better now, in my opinion.

One thing to note: I've reused langchain.text_splitter.Language to avoid duplicate things, but maybe that class should be moved to a global module.

More suggestions are welcome!

[rlancemartin]

Done! It's much better now, in my opinion.

One thing to note: I've reused langchain.text_splitter.Language to avoid duplicate things, but maybe that class should be moved to a global module.

More suggestions are welcome!

Nice! Yes, clean UX:

loader = GenericLoader.from_filesystem(
    "./example_data/languages",
    glob="*",
    suffixes=[".py", ".js"],
    parser=LanguageParser()
)
docs = loader.load()

Small clarification in the docs:

1. it will default split on classes and functions? can this be configured (e.g., only split on functions)?
2. will the top-level code, but without the already loaded functions and classes doc always be generated?

This is looking good. I think we only need to clarify usage a bit.

In short, it simply looks like this will by default split any js or py based on class and function definitions?

[cristobalcl]

Exactly, it only splits, or segments, the source code based on top-level classes and functions, and the remaining code.

I've edited the documentation trying to make it more straightforward. Also, the docstring mentions the only parameters that can be configured: language and parser_threshold.

Should I add an example in the notebook like the one I mention in the first post in this PR? I thought that the code and the results would be self-explanatory, but perhaps it's not clear enough.

[rlancemartin]

Exactly, it only splits, or segments, the source code based on top-level classes and functions, and the remaining code.

I've edited the documentation trying to make it more straightforward. Also, the docstring mentions the only parameters that can be configured: language and parser_threshold.

Should I add an example in the notebook like the one I mention in the first post in this PR? I thought that the code and the results would be self-explanatory, but perhaps it's not clear enough.

Nice work! No need to add more examples. I'm running tests now and will plan to merge this once they all pass.

[cristobalcl]

Nice, thanks for your help, and for that last fixes! 😃