-
-
Notifications
You must be signed in to change notification settings - Fork 31.6k
bytes.hex(sep, bytes_per_sep) is many times slower than manually inserting the separators #84493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Consider the following example, linewrapping 10^4 bytes in hex form to 128 characters per line, on Py 3.8.2 (Arch Linux repo package):
(the last line checks the validity of the code.) It appears that a naive manual wrap is nearly 3x faster than the builtin functionality. |
I replicated this behavior. This looks like the relevant loop in pystrhex.c: for (i=j=0; i < arglen; ++i) {
assert((j + 1) < resultlen);
unsigned char c;
c = (argbuf[i] >> 4) & 0x0f;
retbuf[j++] = Py_hexdigits[c];
c = argbuf[i] & 0x0f;
retbuf[j++] = Py_hexdigits[c];
if (bytes_per_sep_group && i < arglen - 1) {
Py_ssize_t anchor;
anchor = (bytes_per_sep_group > 0) ? (arglen - 1 - i) : (i + 1);
if (anchor % abs_bytes_per_sep == 0) {
retbuf[j++] = sep_char;
}
}
} It looks like this can be refactored a bit for a tighter inner loop with fewer if-tests. I can work on a PR. |
========== Master ========== .\python.bat -m pyperf timeit -s "import random, math; data=random.getrandbits(8*10_000_000).to_bytes(10_000_000, 'big')" "temp = data.hex(); '\n'.join(temp[n:n+128] for n in range(0, len(temp), 128))" Mean +- std dev: 74.3 ms +- 1.1 ms .\python.bat -m pyperf timeit -s "import random; data=random.getrandbits(8*10_000_000).to_bytes(10_000_000, 'big')" "data.hex('\n', -64)" Mean +- std dev: 44.0 ms +- 0.3 ms ========== PR 19594 ========== .\python.bat -m pyperf timeit -s "import random, math; data=random.getrandbits(8*10_000_000).to_bytes(10_000_000, 'big')" "temp = data.hex(); '\n'.join(temp[n:n+128] for n in range(0, len(temp), 128))" Mean +- std dev: 65.2 ms +- 0.6 ms .\python.bat -m pyperf timeit -s "import random; data=random.getrandbits(8*10_000_000).to_bytes(10_000_000, 'big')" "data.hex('\n', -64)" Mean +- std dev: 18.1 ms +- 0.1 ms |
Thanks Dennis for the optimization! FYI I also pushed another optimization recently: commit 455df97
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: