Skip to content

Base64 update with absl::Base64#12703

Closed
ping-sun wants to merge 3 commits into
envoyproxy:masterfrom
ping-sun:Base64-update
Closed

Base64 update with absl::Base64#12703
ping-sun wants to merge 3 commits into
envoyproxy:masterfrom
ping-sun:Base64-update

Conversation

@ping-sun
Copy link
Copy Markdown
Contributor

@ping-sun ping-sun commented Aug 18, 2020

Related Issue: #12694

Description:
This pull request helps to replace the current Base64::encode/decode with absl::Base64.
The current code base provide base64 encoding and decoding with or without padding. Abseil has supported to do that. To be more specific:
Base64::encode(src_data, src_len, padding=true) can be replaced with absl::Base64Escape(std::string src).
Base64::encode(src_data, src_len, padding=false) can be replaced with absl::Base64WebSafeEscape(std::string src).
Base64Url::encode(src_data, src_len) can be replaced with absl::Base64WebSafeEscape(std::string src).

Base64::decode(std::string src) can be replaced with absl::Base64Unescape(src, *dst).
Base64::decodeWithoutPadding(std::string src) can be replaced with absl::Base64WebSafeUnescape(src, *dst).
Base64Url::decode(std::string src) can be replaced with absl::Base64WebSafeEscape(std::string src).

The doc of absl::Base64: https://github.com/abseil/abseil-cpp/blob/master/absl/strings/escaping.h

Signed-off-by: pingsun <pingsun@google.com>
Signed-off-by: pingsun <pingsun@google.com>
Signed-off-by: pingsun <pingsun@google.com>
@lizan
Copy link
Copy Markdown
Member

lizan commented Aug 18, 2020

If we can't completely replace and remove Base64::encode/decode, it might not worth for the replacement. Can you just replace the implementation of Base64::encode/decode?

@ping-sun
Copy link
Copy Markdown
Contributor Author

If we can't completely replace and remove Base64::encode/decode, it might not worth for the replacement. Can you just replace the implementation of Base64::encode/decode?

Sure, Thanks for the great idea. I previously thought that we will finally remove the lib of the current Base64 so I just updated the code where it has been used directly. I think your idea is a even better, I will try to replace the implementation of Base64.

@ping-sun
Copy link
Copy Markdown
Contributor Author

I have some trouble implementing the Base64::decode

org = "Zh==";
success = absl::Base64Unescape(org, &decoded);
EXPECT_EQ(decoded, "");    // "f"
success = absl::WebSafeBase64Unescape(org, &decoded); 
EXPECT_EQ(decoded, "");    // "f"

The expected value for decode("Zh==") in the test is EMPTY_STRING.
However, I can not get that result neither with a absl::Base64Unescape nor a absl::WebSafeBase64Unescape

@ping-sun
Copy link
Copy Markdown
Contributor Author

The current implementation:

std::string Base64::decode(const std::string& input) {
  if (input.length() % 4) {
    return EMPTY_STRING;
  }
  return decodeWithoutPadding(input);
}

std::string Base64::decodeWithoutPadding(absl::string_view input) {
  if (input.empty()) {
    return EMPTY_STRING;
  }
  std::string decoded;
  bool success = absl::Base64Unescape(input, &decoded);
  if (success) {
    return decoded;
  } else {
    return EMPTY_STRING;
  }
}

std::string Base64Url::decode(const std::string& input) {
  if (input.empty()) {
    return EMPTY_STRING;
  }
  std::string decoded;
  bool success = absl::WebSafeBase64Unescape(input, &decoded);
  if (success) {
    return decoded;
  } else {
    return EMPTY_STRING;
  }
}

@ping-sun
Copy link
Copy Markdown
Contributor Author

ping-sun commented Aug 24, 2020

Hi @lizan, I think it might not be possible to remove the original implementation of Base64 Encode/Decode, especially considering the usage of encodeBase/encodeLast and decodeBase/decodeLast for const Buffer::Instance& buffer object. So probably it is not a good idea to replace the current implementation directly with the abseil. What do you think?

@mattklein123
Copy link
Copy Markdown
Member

@ASOPVII if we can't replace it yeah I would say let's just close this for now?

@ping-sun
Copy link
Copy Markdown
Contributor Author

@ASOPVII if we can't replace it yeah I would say let's just close this for now?

Sure, I agree~

@ping-sun ping-sun closed this Aug 24, 2020
RyanTheOptimist pushed a commit that referenced this pull request May 21, 2026
Commit Message: replaces base64 implementation with Abseil's base64
implementation.

Additional Description: Replaces the existing base64 implementation with
Abseil’s base64 one. This change maintains identical encoding/decoding
behavior.

Benchmarks show 
~2x at 8 bytes
~5x at 64 bytes
~8-10x at 512 bytes and above
<details>
<summary>Full Benchmark</summary>
<pre>

----------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...

----------------------------------------------------------------------------------------------------------------
BM_AbslBase64Encode/8 20.5 ns 20.4 ns 33893373
bytes_per_second=374.723Mi/s
BM_AbslBase64Encode/64 56.1 ns 55.8 ns 12443880
bytes_per_second=1.06846Gi/s
BM_AbslBase64Encode/512 248 ns 247 ns 2822896
bytes_per_second=1.93045Gi/s
BM_AbslBase64Encode/4096 1777 ns 1770 ns 395332
bytes_per_second=2.15477Gi/s
BM_AbslBase64Encode/32768 14170 ns 14108 ns 50054
bytes_per_second=2.16317Gi/s
BM_AbslBase64Encode/262144 110850 ns 110328 ns 6340
bytes_per_second=2.21287Gi/s
BM_AbslBase64Encode/1048576 442523 ns 440467 ns 1589
bytes_per_second=2.21711Gi/s
BM_LegacyBase64Encode/8 42.7 ns 42.5 ns 16462454
bytes_per_second=179.434Mi/s
BM_LegacyBase64Encode/64 310 ns 308 ns 2324094
bytes_per_second=197.93Mi/s
BM_LegacyBase64Encode/512 2202 ns 2193 ns 322767
bytes_per_second=222.7Mi/s
BM_LegacyBase64Encode/4096 17763 ns 17687 ns 40018
bytes_per_second=220.852Mi/s
BM_LegacyBase64Encode/32768 141403 ns 140443 ns 5038
bytes_per_second=222.51Mi/s
BM_LegacyBase64Encode/262144 1127638 ns 1121530 ns 633
bytes_per_second=222.91Mi/s
BM_LegacyBase64Encode/1048576 4445197 ns 4424678 ns 155
bytes_per_second=226.005Mi/s
BM_AbslBase64Decode/8 25.9 ns 25.8 ns 26666600
bytes_per_second=443.357Mi/s
BM_AbslBase64Decode/64 80.2 ns 79.8 ns 8809031
bytes_per_second=1.02672Gi/s
BM_AbslBase64Decode/512 398 ns 396 ns 1774014 bytes_per_second=1.607Gi/s
BM_AbslBase64Decode/4096 3000 ns 2985 ns 237276
bytes_per_second=1.70479Gi/s
BM_AbslBase64Decode/32768 23209 ns 23118 ns 30188
bytes_per_second=1.76019Gi/s
BM_AbslBase64Decode/262144 185230 ns 184336 ns 3801
bytes_per_second=1.76593Gi/s
BM_AbslBase64Decode/1048576 742042 ns 738630 ns 953
bytes_per_second=1.76284Gi/s
BM_LegacyBase64Decode/8 59.7 ns 59.5 ns 11753923
bytes_per_second=192.427Mi/s
BM_LegacyBase64Decode/64 431 ns 428 ns 1568483
bytes_per_second=195.924Mi/s
BM_LegacyBase64Decode/512 3209 ns 3196 ns 216661
bytes_per_second=204.13Mi/s
BM_LegacyBase64Decode/4096 25388 ns 25295 ns 27671
bytes_per_second=206.001Mi/s
BM_LegacyBase64Decode/32768 203233 ns 202417 ns 3500
bytes_per_second=205.852Mi/s
BM_LegacyBase64Decode/262144 1615847 ns 1608513 ns 437
bytes_per_second=207.232Mi/s
BM_LegacyBase64Decode/1048576 6483992 ns 6455343 ns 109
bytes_per_second=206.548Mi/s
BM_AbslBase64BufferInstanceEncode/8 35.3 ns 35.1 ns 19569259
bytes_per_second=217.294Mi/s
BM_AbslBase64BufferInstanceEncode/64 70.5 ns 70.1 ns 9924410
bytes_per_second=870.793Mi/s
BM_AbslBase64BufferInstanceEncode/512 269 ns 268 ns 2601418
bytes_per_second=1.77916Gi/s
BM_AbslBase64BufferInstanceEncode/4096 1836 ns 1826 ns 383958
bytes_per_second=2.0896Gi/s
BM_AbslBase64BufferInstanceEncode/32768 14150 ns 14086 ns 49749
bytes_per_second=2.16657Gi/s
BM_AbslBase64BufferInstanceEncode/262144 111975 ns 111408 ns 6288
bytes_per_second=2.19141Gi/s
BM_AbslBase64BufferInstanceEncode/1048576 446107 ns 443873 ns 1580
bytes_per_second=2.20009Gi/s
BM_LegacyBase64BufferInstanceEncode/8 54.1 ns 53.9 ns 13240954
bytes_per_second=141.619Mi/s
BM_LegacyBase64BufferInstanceEncode/64 335 ns 334 ns 2099860
bytes_per_second=183.004Mi/s
BM_LegacyBase64BufferInstanceEncode/512 2355 ns 2346 ns 299256
bytes_per_second=208.131Mi/s
BM_LegacyBase64BufferInstanceEncode/4096 18410 ns 18336 ns 38182
bytes_per_second=213.041Mi/s
BM_LegacyBase64BufferInstanceEncode/32768 147610 ns 147060 ns 4762
bytes_per_second=212.498Mi/s
BM_LegacyBase64BufferInstanceEncode/262144 1179349 ns 1174398 ns 596
bytes_per_second=212.875Mi/s
BM_LegacyBase64BufferInstanceEncode/1048576 4742301 ns 4721762 ns 149
bytes_per_second=211.785Mi/s
BM_AbslBase64BufferInstanceEncodeMultiSlice/4096 1911 ns 1903 ns 369925
bytes_per_second=2.00461Gi/s
BM_AbslBase64BufferInstanceEncodeMultiSlice/32768 15227 ns 15165 ns
45886 bytes_per_second=2.01239Gi/s
BM_AbslBase64BufferInstanceEncodeMultiSlice/262144 124408 ns 123841 ns
5669 bytes_per_second=1.9714Gi/s
BM_AbslBase64BufferInstanceEncodeMultiSlice/1048576 499894 ns 497611 ns
1407 bytes_per_second=1.9625Gi/s
BM_LegacyBase64BufferInstanceEncodeMultiSlice/4096 18588 ns 18516 ns
37684 bytes_per_second=210.969Mi/s
BM_LegacyBase64BufferInstanceEncodeMultiSlice/32768 147629 ns 147052 ns
4634 bytes_per_second=212.51Mi/s
BM_LegacyBase64BufferInstanceEncodeMultiSlice/262144 1183149 ns 1177438
ns 594 bytes_per_second=212.325Mi/s
BM_LegacyBase64BufferInstanceEncodeMultiSlice/1048576 4724186 ns 4702789
ns 149 bytes_per_second=212.64Mi/s
BM_AbslBase64UrlEncode/8 19.9 ns 19.8 ns 35318734
bytes_per_second=385.207Mi/s
BM_AbslBase64UrlEncode/64 54.6 ns 54.4 ns 12961587
bytes_per_second=1.09557Gi/s
BM_AbslBase64UrlEncode/512 252 ns 252 ns 2792499
bytes_per_second=1.89564Gi/s
BM_AbslBase64UrlEncode/4096 1760 ns 1753 ns 399001
bytes_per_second=2.17577Gi/s
BM_AbslBase64UrlEncode/32768 14038 ns 13986 ns 50051
bytes_per_second=2.18199Gi/s
BM_AbslBase64UrlEncode/262144 110773 ns 110276 ns 6348
bytes_per_second=2.2139Gi/s
BM_AbslBase64UrlEncode/1048576 442787 ns 440739 ns 1589
bytes_per_second=2.21574Gi/s
BM_LegacyBase64UrlEncode/8 41.5 ns 41.3 ns 16950384
bytes_per_second=184.617Mi/s
BM_LegacyBase64UrlEncode/64 297 ns 296 ns 2355033
bytes_per_second=206.009Mi/s
BM_LegacyBase64UrlEncode/512 2136 ns 2128 ns 328029
bytes_per_second=229.444Mi/s
BM_LegacyBase64UrlEncode/4096 16885 ns 16823 ns 41566
bytes_per_second=232.199Mi/s
BM_LegacyBase64UrlEncode/32768 134893 ns 134392 ns 5212
bytes_per_second=232.529Mi/s
BM_LegacyBase64UrlEncode/262144 1083154 ns 1078276 ns 650
bytes_per_second=231.852Mi/s
BM_LegacyBase64UrlEncode/1048576 4356684 ns 4336895 ns 161
bytes_per_second=230.58Mi/s
BM_AbslBase64UrlDecode/8 25.3 ns 25.2 ns 27737820
bytes_per_second=415.973Mi/s
BM_AbslBase64UrlDecode/64 78.3 ns 78.0 ns 8924236
bytes_per_second=1.02696Gi/s
BM_AbslBase64UrlDecode/512 397 ns 395 ns 1767540
bytes_per_second=1.60968Gi/s
BM_AbslBase64UrlDecode/4096 2967 ns 2952 ns 242422
bytes_per_second=1.7233Gi/s
BM_AbslBase64UrlDecode/32768 23697 ns 23524 ns 29233
bytes_per_second=1.72977Gi/s
BM_AbslBase64UrlDecode/262144 185443 ns 184584 ns 3795
bytes_per_second=1.76354Gi/s
BM_AbslBase64UrlDecode/1048576 746481 ns 742547 ns 953
bytes_per_second=1.75354Gi/s
BM_LegacyBase64UrlDecode/8 60.3 ns 60.0 ns 11947521
bytes_per_second=174.801Mi/s
BM_LegacyBase64UrlDecode/64 427 ns 425 ns 1636120
bytes_per_second=193.1Mi/s
BM_LegacyBase64UrlDecode/512 3163 ns 3151 ns 219959
bytes_per_second=206.698Mi/s
BM_LegacyBase64UrlDecode/4096 25836 ns 25510 ns 27914
bytes_per_second=204.193Mi/s
BM_LegacyBase64UrlDecode/32768 203179 ns 202069 ns 3471
bytes_per_second=206.201Mi/s
BM_LegacyBase64UrlDecode/262144 1621516 ns 1614225 ns 435
bytes_per_second=206.498Mi/s
BM_LegacyBase64UrlDecode/1048576 6539881 ns 6506282 ns 107
bytes_per_second=204.93Mi/s
</pre>
</details>

Risk Level: Medium (encoding/decoding is core utils logic and should
preserve behavior if tests pass).

Testing: Existing unit tests for base64 encoding/decoding pass. Added
differential fuzz testing (test/common/common/base64_fuzz_test.cc) and
benchmarks (test/common/common/base64_speed_test.cc) comparing the new
base64 implementation against the legacy implementation.

Docs Changes: None

Release Notes: Base64 implementation updated to use Abseil utilities. No
behavioral changes expected.

Platform Specific Features: N/A

[Optional Fixes #Issue]: Fixes #12694.

[Optional Fixes commit #PR or SHA]: Related to PR #29360 and PR #12703.

---------

Signed-off-by: Ethan Marantz <ebmarantz@gmail.com>
Signed-off-by: ethan <ethan@debian-server.home.arpa>
Co-authored-by: ethan <ethan@debian-server.home.arpa>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants