[mypyc] Add efficient librt.base64.b64decode #20263
Merged
+297
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
p-sawicki
approved these changes
Nov 19, 2025
mypyc/lib-rt/librt_base64.c
Outdated
Comment on lines
73
to
74
| return ((c >= 'A' && c <= 'Z') | (c >= 'a' && c <= 'z') | | ||
| (c >= '0' && c <= '9') | (c == '+') | (c == '/') | (allow_padding && c == '=')); |
Collaborator
There was a problem hiding this comment.
is there a reason to mix logical && and bitwise |?
Collaborator
Author
There was a problem hiding this comment.
No particularly good reason, mainly to highlight that these don't need branches at runtime (we don't want many mispredicted branches). I think this was from ChatGPT output and I thought it was okay in this use case. The semantics are identical so it's probably better to use && consistently though.
Collaborator
Author
There was a problem hiding this comment.
I did some experiments, and using || might help compilers generate faster code, so I will use it.
Contributor
|
According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The performance can be 10x faster than stdlib if input is valid base64, or if input has extra non-base64 characters only at the end of input. Similar to the base64 encode implementation I added recently, this uses SIMD instructions when available.
The implementation first tries to decode the input optimistically assuming valid base64. If this fails, we'll perform a slow path with a preprocessing step that removes extra characters, and we'll perform a strict base64 decode on the cleaned up input.
The semantics aren't 100% compatible with stdlib. First, we raise ValueError on invalid padding instead of
binascii.Error, since I don't want a runtime dependency on the unrelated abinasciimodule. This needs to be documented, but stdlib can already raise ValueError on other conditions, so the deviation is not huge. Also, some invalid inputs are checked more strictly for padding violations. The stdlib implementation has some mysterious behaviors with invalid inputs that didn't seem worth replicating.The function only accepts a single ASCII str or bytes argument for now, since that seems to be by the far the most common use case. The stdlib function also accepts buffer objects and a
validateargument.The slow path is still somewhat faster than stdlib (on the order of 1.3x to 2x for longer inputs), at least if the input is much smaller than L1 cache size.
Got the initial fast path implementation from ChatGPT, but did a bunch of manual edits afterwards and reviewed carefully.