Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add content to Strip decoder to allow decoding mid tokens. #1199

Merged
merged 3 commits into from
Mar 24, 2023
Merged

Add content to Strip decoder to allow decoding mid tokens. #1199

merged 3 commits into from
Mar 24, 2023

Conversation

Narsil
Copy link
Collaborator

@Narsil Narsil commented Mar 24, 2023 •
edited

Allows for more flexible stripping.

Strip("_", left=1).decode("_Hello") -> "Hello"
Strip("_", left=1).decode("Hello") -> "Hello"

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Mar 24, 2023 •
edited

The documentation is not available anymore as the PR was closed or merged.

@Narsil Narsil merged commit 3aaf494 into main Mar 24, 2023
@Narsil Narsil deleted the strip_decoder_content branch March 24, 2023 09:14
@sgugger
Copy link
Contributor

sgugger commented Mar 24, 2023

Looked good to me 😅

@Narsil
Copy link
Collaborator Author

Narsil commented Mar 24, 2023

Sorry I was getting impatient to get things working on the final PR :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger Awaiting requested review from sgugger

@ArthurZucker ArthurZucker Awaiting requested review from ArthurZucker

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Narsil @HuggingFaceDocBuilderDev @sgugger