base64: switch to libsodium implementation #1786
Conversation
libsodium provides a base64 implementation that can replace libutil/base64.[ch]. It was added in version 1.0.14.
Codecov Report
@@ Coverage Diff @@
## master #1786 +/- ##
==========================================
- Coverage 79.66% 79.58% -0.08%
==========================================
Files 187 186 -1
Lines 34724 34560 -164
==========================================
- Hits 27662 27504 -158
+ Misses 7062 7056 -6
|
|
would it be worthwhile putting the |
|
Yeah... thoughts on where? Is that appropriate for // maximum size of buffer needed to decode base64 string of length 'x'
#define BASE64_DECODE_SIZE(x) ((((x) + 3) / 4) * 3) |
Seems like as good a place as any to me. You might want to keep @dun's nice explanation from the original source (and also make a note about not including space for terminating NUL) |
|
Ah this Yes, I'll add something to that effect. |
garlick
force-pushed
the
base64_sodium
branch
from
544b6b6
to
6036bf6
Compare
|
OK I made that change and forced a push. Macro is documented as follows. @dun, I pilfered some of your words, hope you don't mind. /* Maximum size of buffer needed to decode a base64 string of length 'x',
* where 4 characters are decoded into 3 bytes. Add 3 bytes to ensure a
* partial 4-byte chunk will be accounted for during integer division.
* This size is safe for use with all (4) libsodium base64 variants.
* N.B. unlike @dun's base64 implementation from the munge project,
* this size does not include room for a \0 string terminator.
*/
#define BASE64_DECODE_SIZE(x) ((((x) + 3) / 4) * 3) |
|
@garlick Looks good to me. 👻 |
src/common/libsubprocess/remote.c
Outdated
| @@ -587,7 +588,7 @@ static int remote_output (flux_subprocess_t *p, flux_future_t *f, | |||
|
|
|||
| if (tmp != len) { | |||
| flux_log_error (p->h, "channel buffer error: rank = %d pid = %d, stream = %s, len = %d", | |||
| rank, pid, stream, len); | |||
| rank, pid, stream, (int)len); | |||
There was a problem hiding this comment.
instead of casting, perhaps should just use %zu (i think) specifier?
There was a problem hiding this comment.
Ok, fixed and forced a push.
src/common/libsubprocess/server.c
Outdated
| @@ -427,7 +429,7 @@ static int write_subprocess (flux_subprocess_server_t *s, flux_subprocess_t *p, | |||
|
|
|||
| if (tmp != len) { | |||
| flux_log_error (s->h, "channel buffer error: rank = %d pid = %d, stream = %s, len = %d", | |||
| s->rank, flux_subprocess_pid (p), name, len); | |||
| s->rank, flux_subprocess_pid (p), name, (int)len); | |||
garlick
force-pushed
the
base64_sodium
branch
from
6036bf6
to
6ecba49
Compare
|
I think this is ok to merge, although I'm unsure why codecov says "CI" failed? diff coverage seems acceptable, as some error paths weren't hit. Should it just be re-run? |
|
Hmm that is odd. I did restart a builder that had hung without much to go on. about a half hour ago. Coverage was OK before those minor changes so prob its OK to merge (IMHO) |
|
oh that's weird, the "diff coverage" stat also disappeared, it was like 76% out of 79% a minute ago. codecov's twitter says there's some issues, so I'll assume that's just it. Hitting the button. |
This PR switches users of the base64 implementation that we borrowed from munge over to the one in libsodium per discussion in #1775.
One thing that's slightly different about the libsodium implementation is that on decoding (base64 to bin), it does not implicitly pad with a
\0byte. The munge version did this, and did not include it in the length. We were relying on that in one spot inlibkvs/treeobj.c. I will open a bug on fixing our reliance on that (did not want to get distracted with it here).Another difference is that there is no function in libsodium for computing the decoded size in advance so the exact buffer size can be allocated. I used a value of
((srclen + 3) / 4) * 3(math borrowed from the munge base64.c, without its +1 for the aforementioned\0).Finally error handling is a bit different. The encode function always succeeds, which is expected since the buffer is pre-allocated of a size computed by libsodium. If you manage to pass a buffer that is too small, then libsodium internally calls its
sodium_misuse()function, which callsabort().Docs and configure are updated for the new external dependency. The Docker builds for travis were already installing it.
Fixes #1775