Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Avoid double serialization of message data
Prior to this commit, messages with metadata were always serialized in the following way: ```ruby Base64.strict_encode64( ActiveSupport::JSON.encode({ "_rails" => { "message" => Base64.strict_encode64( serializer.dump(data) ), "pur" => "the purpose", "exp" => "the expiration" }, }) ) ``` in which the message data is serialized and URL-encoded twice. This commit changes message serialization such that, when possible, the data is serialized and URL-encoded only once: ```ruby Base64.strict_encode64( serializer.dump({ "_rails" => { "data" => data, "pur" => "the purpose", "exp" => "the expiration" }, }) ) ``` This improves performance in proportion to the size of the data: **Benchmark** ```ruby # frozen_string_literal: true require "benchmark/ips" require "active_support/all" verifier = ActiveSupport::MessageVerifier.new("secret", serializer: JSON) payloads = [ { "content" => "x" * 100 }, { "content" => "x" * 2000 }, { "content" => "x" * 1_000_000 }, ] if ActiveSupport::Messages::Metadata.respond_to?(:use_message_serializer_for_metadata) ActiveSupport::Messages::Metadata.use_message_serializer_for_metadata = true end Benchmark.ips do |x| payloads.each do |payload| x.report("generate ~#{payload["content"].size}B") do $generated_message = verifier.generate(payload, purpose: "x") end x.report("verify ~#{payload["content"].size}B") do verifier.verify($generated_message, purpose: "x") end end end puts puts "Message size:" payloads.each do |payload| puts " ~#{payload["content"].size} bytes of data => " \ "#{verifier.generate(payload, purpose: "x").size} byte message" end ``` **Before** ``` Warming up -------------------------------------- generate ~100B 1.578k i/100ms verify ~100B 2.506k i/100ms generate ~2000B 447.000 i/100ms verify ~2000B 1.409k i/100ms generate ~1000000B 1.000 i/100ms verify ~1000000B 6.000 i/100ms Calculating ------------------------------------- generate ~100B 15.807k (± 1.8%) i/s - 80.478k in 5.093161s verify ~100B 25.240k (± 2.1%) i/s - 127.806k in 5.066096s generate ~2000B 4.530k (± 2.4%) i/s - 22.797k in 5.035398s verify ~2000B 14.136k (± 2.3%) i/s - 71.859k in 5.086267s generate ~1000000B 11.673 (± 0.0%) i/s - 59.000 in 5.060598s verify ~1000000B 64.372 (± 6.2%) i/s - 324.000 in 5.053304s Message size: ~100 bytes of data => 306 byte message ~2000 bytes of data => 3690 byte message ~1000000 bytes of data => 1777906 byte message ``` **After** ``` Warming up -------------------------------------- generate ~100B 4.689k i/100ms verify ~100B 3.183k i/100ms generate ~2000B 2.722k i/100ms verify ~2000B 2.066k i/100ms generate ~1000000B 12.000 i/100ms verify ~1000000B 11.000 i/100ms Calculating ------------------------------------- generate ~100B 46.984k (± 1.2%) i/s - 239.139k in 5.090540s verify ~100B 32.043k (± 1.2%) i/s - 162.333k in 5.066903s generate ~2000B 27.163k (± 1.2%) i/s - 136.100k in 5.011254s verify ~2000B 20.726k (± 1.7%) i/s - 105.366k in 5.085442s generate ~1000000B 125.600 (± 1.6%) i/s - 636.000 in 5.064607s verify ~1000000B 122.039 (± 4.1%) i/s - 616.000 in 5.058386s Message size: ~100 bytes of data => 234 byte message ~2000 bytes of data => 2770 byte message ~1000000 bytes of data => 1333434 byte message ``` This optimization is only applied for recognized serializers that are capable of serializing a `Hash`. Additionally, because the optimization changes the message format, a `config.active_support.use_message_serializer_for_metadata` option has been added to disable it. The optimization is disabled by default, but enabled with `config.load_defaults 7.1`. Regardless of whether the optimization is enabled, messages using either format can still be read. In the case of a rolling deploy of a Rails upgrade, wherein servers that have not yet been upgraded must be able to read messages from upgraded servers, the optimization can be disabled on first deploy, then safely enabled on a subsequent deploy.
- Loading branch information
1 parent
ebc3b66
commit 91bb5da
Showing
10 changed files
with
184 additions
and
70 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,83 +1,101 @@ | ||
# frozen_string_literal: true | ||
|
||
require "time" | ||
require "active_support/json" | ||
|
||
module ActiveSupport | ||
module Messages # :nodoc: | ||
class Metadata # :nodoc: | ||
def initialize(message, expires_at = nil, purpose = nil) | ||
@message, @purpose = message, purpose | ||
@expires_at = expires_at.is_a?(String) ? parse_expires_at(expires_at) : expires_at | ||
end | ||
module Metadata # :nodoc: | ||
singleton_class.attr_accessor :use_message_serializer_for_metadata | ||
|
||
def as_json(options = {}) | ||
{ _rails: { message: @message, exp: @expires_at, pur: @purpose } } | ||
end | ||
ENVELOPE_SERIALIZERS = [ | ||
::JSON, | ||
ActiveSupport::JSON, | ||
ActiveSupport::JsonWithMarshalFallback, | ||
Marshal, | ||
] | ||
|
||
class << self | ||
def wrap(message, expires_at: nil, expires_in: nil, purpose: nil) | ||
if expires_at || expires_in || purpose | ||
JSON.encode new(encode(message), pick_expiry(expires_at, expires_in), purpose) | ||
private | ||
def serialize_with_metadata(data, **metadata) | ||
has_metadata = metadata.any? { |k, v| v } | ||
|
||
if has_metadata && !use_message_serializer_for_metadata? | ||
data_string = serialize_to_json_safe_string(data) | ||
envelope = wrap_in_metadata_envelope({ "message" => data_string }, **metadata) | ||
ActiveSupport::JSON.encode(envelope) | ||
else | ||
message | ||
data = wrap_in_metadata_envelope({ "data" => data }, **metadata) if has_metadata | ||
serializer.dump(data) | ||
end | ||
end | ||
|
||
def verify(message, purpose) | ||
extract_metadata(message).verify(purpose) | ||
end | ||
|
||
private | ||
def pick_expiry(expires_at, expires_in) | ||
if expires_at | ||
expires_at.utc.iso8601(3) | ||
elsif expires_in | ||
Time.now.utc.advance(seconds: expires_in).iso8601(3) | ||
end | ||
end | ||
|
||
def extract_metadata(message) | ||
begin | ||
data = JSON.decode(message) if message.start_with?('{"_rails":') | ||
rescue ::JSON::JSONError | ||
end | ||
|
||
if data | ||
new(decode(data["_rails"]["message"]), data["_rails"]["exp"], data["_rails"]["pur"]) | ||
def deserialize_with_metadata(message, **expected_metadata) | ||
if dual_serialized_metadata_envelope_json?(message) | ||
envelope = ActiveSupport::JSON.decode(message) | ||
extracted = extract_from_metadata_envelope(envelope, **expected_metadata) | ||
deserialize_from_json_safe_string(extracted["message"]) if extracted | ||
else | ||
deserialized = serializer.load(message) | ||
if metadata_envelope?(deserialized) | ||
extracted = extract_from_metadata_envelope(deserialized, **expected_metadata) | ||
extracted["data"] if extracted | ||
else | ||
new(message) | ||
deserialized if expected_metadata.none? { |k, v| v } | ||
end | ||
end | ||
end | ||
|
||
def encode(message) | ||
::Base64.strict_encode64(message) | ||
end | ||
def use_message_serializer_for_metadata? | ||
Metadata.use_message_serializer_for_metadata && Metadata::ENVELOPE_SERIALIZERS.include?(serializer) | ||
end | ||
|
||
def decode(message) | ||
::Base64.strict_decode64(message) | ||
end | ||
end | ||
def wrap_in_metadata_envelope(hash, expires_at: nil, expires_in: nil, purpose: nil) | ||
expiry = pick_expiry(expires_at, expires_in) | ||
hash["exp"] = expiry if expiry | ||
hash["pur"] = purpose.to_s if purpose | ||
{ "_rails" => hash } | ||
end | ||
|
||
def verify(purpose) | ||
@message if match?(purpose) && fresh? | ||
end | ||
def extract_from_metadata_envelope(envelope, purpose: nil) | ||
hash = envelope["_rails"] | ||
return if hash["exp"] && Time.now.utc >= parse_expiry(hash["exp"]) | ||
return if hash["pur"] != purpose&.to_s | ||
hash | ||
end | ||
|
||
private | ||
def match?(purpose) | ||
@purpose.to_s == purpose.to_s | ||
def metadata_envelope?(object) | ||
object.is_a?(Hash) && object.key?("_rails") | ||
end | ||
|
||
def dual_serialized_metadata_envelope_json?(string) | ||
string.start_with?('{"_rails":{"message":') | ||
end | ||
|
||
def fresh? | ||
@expires_at.nil? || Time.now.utc < @expires_at | ||
def pick_expiry(expires_at, expires_in) | ||
if expires_at | ||
expires_at.utc.iso8601(3) | ||
elsif expires_in | ||
Time.now.utc.advance(seconds: expires_in).iso8601(3) | ||
end | ||
end | ||
|
||
def parse_expires_at(expires_at) | ||
if ActiveSupport.use_standard_json_time_format | ||
def parse_expiry(expires_at) | ||
if !expires_at.is_a?(String) | ||
expires_at | ||
elsif ActiveSupport.use_standard_json_time_format | ||
Time.iso8601(expires_at) | ||
else | ||
Time.parse(expires_at) | ||
end | ||
end | ||
|
||
def serialize_to_json_safe_string(data) | ||
::Base64.strict_encode64(serializer.dump(data)) | ||
end | ||
|
||
def deserialize_from_json_safe_string(string) | ||
serializer.load(::Base64.strict_decode64(string)) | ||
end | ||
end | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.