Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev2049/from tiktoken fix #4361

Merged
merged 3 commits into from
May 8, 2023
Merged

Dev2049/from tiktoken fix #4361

merged 3 commits into from
May 8, 2023

Conversation

dev2049
Copy link
Contributor

@dev2049 dev2049 commented May 8, 2023

Fix TextSplitter.from_tiktoken

Thanks to @danb27 for the fix! Minor update

Fixes #4357

danb27 and others added 3 commits May 8, 2023 14:12
…encoder (#4358)

# Fix model name not being passed to __init__ when using
from_tiktoken_encoder

<!--
Thank you for contributing to LangChain! Your PR will appear in our next
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.
-->

<!-- Remove if not applicable -->

Fixes #4357


## Before submitting

<!-- If you're adding a new integration, include an integration test and
an example notebook showing its use! -->

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

        @hwchase17 - project lead

        Tracing / Callbacks
        - @agola11

        Async
        - @agola11

        DataLoader Abstractions
        - @eyurtsev

        LLM/Chat Wrappers
        - @hwchase17
        - @agola11

        Tools / Toolkits
        - @vowelparrot
 -->
@hwchase17
@dev2049 dev2049 merged commit 02ebb15 into master May 8, 2023
9 checks passed
@dev2049 dev2049 deleted the dev2049/from_tiktoken_fix branch May 8, 2023 23:36
EandrewJones pushed a commit to Oogway-Technologies/langchain that referenced this pull request May 9, 2023
Thanks to @danb27 for the fix! Minor update

Fixes langchain-ai#4357

---------

Co-authored-by: Dan Bianchini <42096328+danb27@users.noreply.github.com>
jpzhangvincent pushed a commit to jpzhangvincent/langchain that referenced this pull request May 12, 2023
Thanks to @danb27 for the fix! Minor update

Fixes langchain-ai#4357

---------

Co-authored-by: Dan Bianchini <42096328+danb27@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TokenTextSplitter ignores model_name when using from_tiktoken_encoder
2 participants