Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust)

[ByronHsu]

As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each splitter.
Let me know if any concerns.

Fixes # (issue)
#5170

Who can review?

Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested:
@eyurtsev @hwchase17

[dev2049]

could we add some unit tests for all these :) may be easier to have sep pr for each, but will let you decide

[ByronHsu]

@dev2049 ok i will add tests and split the pr :)

[ByronHsu]

Added tests! plz take a look again. Thanks!

@dev2049

btw, may i ask why i run into mypy error locally, but not on github action?

~/learn-repo/langchain more-code-splitter !2 ❯ make lint                                                                                              4s  langchain byhsu@byhsu-ld1 22:50:19
poetry run mypy .
langchain/evaluation/loading.py:5: error: Incompatible import of "load_dataset" (imported name has type "Callable[[str, Optional[str], Optional[str], Union[str, Sequence[str], Mapping[str, Union[str, Sequence[str]]], None], Union[str, Split, None], Optional[str], Optional[Features], Optional[DownloadConfig], Optional[GenerateMode], bool, Optional[bool], bool, Union[str, Version, None], Union[bool, str, None], Union[str, TaskTemplate, None], bool, Any, KwArg(Any)], Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]]", local name has type "Callable[[str], List[Dict[Any, Any]]]")  [assignment]
langchain/evaluation/loading.py:8: error: No overload variant of "__getitem__" of "list" matches argument type "str"  [call-overload]
langchain/evaluation/loading.py:8: note: Possible overload variants:
langchain/evaluation/loading.py:8: note:     def __getitem__(self, SupportsIndex, /) -> Dict[Any, Any]
langchain/evaluation/loading.py:8: note:     def __getitem__(self, slice, /) -> List[Dict[Any, Any]]
langchain/document_loaders/hugging_face_dataset.py:81: error: Item "Dataset" of "Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]" has no attribute "keys"  [union-attr]
langchain/document_loaders/hugging_face_dataset.py:81: error: Item "IterableDataset" of "Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]" has no attribute "keys"  [union-attr]
langchain/document_loaders/hugging_face_dataset.py:82: error: Value of type "Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]" is not indexable  [index]
Found 5 errors in 2 files (checked 1024 source files)
make: *** [lint] Error 1

[ByronHsu]

@eyurtsev Can you review? Thanks!

[eyurtsev]

Left some minor suggestion with enum. Let's update the exception to a ValueError rather than throwing a generic exception.

[eyurtsev]

This looks good to me.

Have you considered using a union of literals instead of an enum?

Language = Union[Literal['go'], Literal['js'], ...]

Gives the same static typing guarantees and makes it such that a user does not have to import an extra enum.

I think the only downside is that it's harder to list all the options in the set.

[ByronHsu]

Hmm I would think prompting available languages is more important to users

[corleyma]

I believe VSCode/PyLance would still prompt appropriately for a union of literals, but the enum is more foolproof.

[eyurtsev]

IDE auto-completion will work in either case

I meant that with enums it may be more convenient for a developer to import the enum and then do something like looping through all the possibilities.

In [1]: from enum import Enum

In [2]: class A(Enum):
   ...:     foo = 'foo'
   ...:     bar = 'bar'
   ...: 

In [4]: list(A)
Out[4]: [<A.foo: 'foo'>, <A.bar: 'bar'>]

---

Anyway this works for now. I think we could propagate the file name through in the document metadata via the source field, then the code splitter will be able to automatically split based on the language potentially

[eyurtsev]

@ByronHsu i can take a look at mypy error on monday. i assume you checked from master?

[ByronHsu]

@dev2049 i've added notebook examples and improved the tests. Could you plz review again? Thanks!