Add a encryption option to support previous data encrypted non-determ…

…inistically with a SHA1 hash digest (#48530)

* Make sure active record encryption configuration happens after initializers have run

Co-authored-by: Cadu Ribeiro <mail@cadu.dev>

* Add a new option to support previous data encrypted non-deterministically with a hash digest of SHA1

There is currently a problem with Active Record encryption for users updating from 7.0 to 7.1 Before
#44873, data encrypted with non-deterministic encryption was always using SHA-1. The reason is that
`ActiveSupport::KeyGenerator.hash_digest_class` is set in an after_initialize block in the railtie config,
but encryption config was running before that, so it was effectively using the previous default SHA1. That
means that existing users are using SHA256 for non deterministic encryption, and SHA1 for deterministic
encryption.

This adds a new option `use_sha1_digest_for_non_deterministic_data` that
users can enable to support for SHA1 and SHA256 when decrypting existing data.

* Set a default value of true for `support_sha1_for_non_deterministic_encryption` and proper initializer values.

We want to enable the flag existing versions (< 7.1), and we want it to be false moving by
default moving forward.

* Make sure the system to auto-filter params supports different initialization orders

This reworks the system to auto-filter params so that it works when encrypted
attributes are declared before the encryption configuration logic runs.

Co-authored-by: Cadu Ribeiro <mail@cadu.dev>

---------

Co-authored-by: Cadu Ribeiro <mail@cadu.dev>

activerecord/CHANGELOG.md

@@ -1,3 +1,19 @@
+*   Support decrypting data encrypted non-deterministically with a SHA1 hash digest.
+
+    This adds a new Active Record encryption option to support decrypting data encrypted
+    non-deterministically with a SHA1 hash digest:
+
+    ```
+    Rails.application.config.active_record.encryption.support_sha1_for_non_deterministic_encryption = true
+    ```
+
+    The new option addresses a problem when upgrading from 7.0 to 7.1. Due to a bug in how Active Record
+    Encryption was getting initialized, the key provider used for non-deterministic encryption were using
+    SHA-1 as its digest class, instead of the one configured globally by Rails via
+    `Rails.application.config.active_support.key_generator_hash_digest_class`.
+
+    *Cadu Ribeiro and Jorge Manrubia*
+
 *   Apply scope to association subqueries. (belongs_to/has_one/has_many)
 
     Given: `has_many :welcome_posts, -> { where(title: "welcome") }`

activerecord/lib/active_record/encryption.rb

@@ -8,6 +8,7 @@ module Encryption
     extend ActiveSupport::Autoload
 
     eager_autoload do
+      autoload :AutoFilteredParameters
       autoload :Cipher
       autoload :Config
       autoload :Configurable

activerecord/lib/active_record/encryption/auto_filtered_parameters.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module ActiveRecord
+  module Encryption
+    class AutoFilteredParameters
+      def initialize(app)
+        @app = app
+        @attributes_by_class = Concurrent::Map.new
+        @collecting = true
+
+        install_collecting_hook
+      end
+
+      def enable
+        apply_collected_attributes
+        @collecting = false
+      end
+
+      private
+        attr_reader :app
+
+        def install_collecting_hook
+          ActiveRecord::Encryption.on_encrypted_attribute_declared do |klass, attribute|
+            attribute_was_declared(klass, attribute)
+          end
+        end
+
+        def attribute_was_declared(klass, attribute)
+          if collecting?
+            collect_for_later(klass, attribute)
+          else
+            apply_filter(klass, attribute)
+          end
+        end
+
+        def apply_collected_attributes
+          @attributes_by_class.each do |klass, attributes|
+            attributes.each do |attribute|
+              apply_filter(klass, attribute)
+            end
+          end
+        end
+
+        def collecting?
+          @collecting
+        end
+
+        def collect_for_later(klass, attribute)
+          @attributes_by_class[klass] ||= Concurrent::Array.new
+          @attributes_by_class[klass] << attribute
+        end
+
+        def apply_filter(klass, attribute)
+          filter = [("#{klass.model_name.element}" if klass.name), attribute.to_s].compact.join(".")
+          unless excluded_from_filter_parameters?(filter)
+            app.config.filter_parameters << filter unless app.config.filter_parameters.include?(filter)
+            klass.filter_attributes += [ attribute ]
+          end
+        end
+
+        def excluded_from_filter_parameters?(filter_parameter)
+          ActiveRecord::Encryption.config.excluded_from_filter_parameters.find { |excluded_filter| excluded_filter.to_s == filter_parameter }
+        end
+    end
+  end
+end

activerecord/lib/active_record/encryption/config.rb

@@ -23,10 +23,23 @@ def previous=(previous_schemes_properties)
         end
       end
 
+      def support_sha1_for_non_deterministic_encryption=(value)
+        if value && has_primary_key?
+          sha1_key_generator = ActiveRecord::Encryption::KeyGenerator.new(hash_digest_class: OpenSSL::Digest::SHA1)
+          sha1_key_provider = ActiveRecord::Encryption::DerivedSecretKeyProvider.new(primary_key, key_generator: sha1_key_generator)
+          add_previous_scheme key_provider: sha1_key_provider
+        end
+      end
+
       %w(key_derivation_salt primary_key deterministic_key).each do |key|
+        silence_redefinition_of_method "has_#{key}?"
+        define_method("has_#{key}?") do
+          instance_variable_get(:"@#{key}").presence
+        end
+
         silence_redefinition_of_method key
         define_method(key) do
-          instance_variable_get(:"@#{key}").presence or
+          public_send("has_#{key}?") or
             raise Errors::Configuration, "Missing Active Record encryption credential: active_record_encryption.#{key}"
         end
       end

activerecord/lib/active_record/encryption/configurable.rb

@@ -22,11 +22,18 @@ def configure(primary_key: nil, deterministic_key: nil, key_derivation_salt: nil
           config.deterministic_key = deterministic_key
           config.key_derivation_salt = key_derivation_salt
 
+          # Set the default for this property here instead of in +Config#set_defaults+ as this needs
+          # to happen *after* the keys have been set.
+          properties[:support_sha1_for_non_deterministic_encryption] = true if properties[:support_sha1_for_non_deterministic_encryption].nil?
+
+          properties.each do |name, value|
+            ActiveRecord::Encryption.config.send "#{name}=", value if ActiveRecord::Encryption.config.respond_to?("#{name}=")
+          end
+
+          ActiveRecord::Encryption.reset_default_context
+
           properties.each do |name, value|
-            [:context, :config].each do |configurable_object_name|
-              configurable_object = ActiveRecord::Encryption.send(configurable_object_name)
-              configurable_object.send "#{name}=", value if configurable_object.respond_to?("#{name}=")
-            end
+            ActiveRecord::Encryption.context.send "#{name}=", value if ActiveRecord::Encryption.context.respond_to?("#{name}=")
           end
         end
 
@@ -47,21 +54,6 @@ def encrypted_attribute_was_declared(klass, name) # :nodoc:
             block.call(klass, name)
           end
         end
-
-        def install_auto_filtered_parameters_hook(app) # :nodoc:
-          ActiveRecord::Encryption.on_encrypted_attribute_declared do |klass, encrypted_attribute_name|
-            filter = [("#{klass.model_name.element}" if klass.name), encrypted_attribute_name.to_s].compact.join(".")
-            unless excluded_from_filter_parameters?(filter)
-              app.config.filter_parameters << filter unless app.config.filter_parameters.include?(filter)
-              klass.filter_attributes += [encrypted_attribute_name]
-            end
-          end
-        end
-
-        private
-          def excluded_from_filter_parameters?(filter_parameter)
-            ActiveRecord::Encryption.config.excluded_from_filter_parameters.find { |excluded_filter| excluded_filter.to_s == filter_parameter }
-          end
       end
     end
   end

activerecord/lib/active_record/encryption/contexts.rb

@@ -14,7 +14,7 @@ module Contexts
       extend ActiveSupport::Concern
 
       included do
-        mattr_reader :default_context, default: Context.new
+        mattr_accessor :default_context, default: Context.new
         thread_mattr_accessor :custom_contexts
       end
 
@@ -66,6 +66,10 @@ def context
         def current_custom_context
           self.custom_contexts&.last
         end
+
+        def reset_default_context
+          self.default_context = Context.new
+        end
       end
     end
   end

activerecord/lib/active_record/encryption/derived_secret_key_provider.rb

@@ -4,9 +4,15 @@ module ActiveRecord
   module Encryption
     # A KeyProvider that derives keys from passwords.
     class DerivedSecretKeyProvider < KeyProvider
-      def initialize(passwords)
-        super(Array(passwords).collect { |password| Key.derive_from(password) })
+      def initialize(passwords, key_generator: ActiveRecord::Encryption.key_generator)
+        super(Array(passwords).collect { |password| derive_key_from(password, using: key_generator) })
       end
+
+      private
+        def derive_key_from(password, using: key_generator)
+          secret = using.derive_key_from(password)
+          ActiveRecord::Encryption::Key.new(secret)
+        end
     end
   end
 end

activerecord/lib/active_record/encryption/key_generator.rb

@@ -6,6 +6,12 @@ module ActiveRecord
   module Encryption
     # Utility for generating and deriving random keys.
     class KeyGenerator
+      attr_reader :hash_digest_class
+
+      def initialize(hash_digest_class: ActiveRecord::Encryption.config.hash_digest_class)
+        @hash_digest_class = hash_digest_class
+      end
+
       # Returns a random key. The key will have a size in bytes of +:length+ (configured +Cipher+'s length by default)
       def generate_random_key(length: key_length)
         SecureRandom.random_bytes(length)
@@ -30,7 +36,7 @@ def generate_random_hex_key(length: key_length)
       #
       # The generated key will be salted with the value of +ActiveRecord::Encryption.key_derivation_salt+
       def derive_key_from(password, length: key_length)
-        ActiveSupport::KeyGenerator.new(password, hash_digest_class: ActiveRecord::Encryption.config.hash_digest_class)
+        ActiveSupport::KeyGenerator.new(password, hash_digest_class: hash_digest_class)
           .generate_key(key_derivation_salt, length)
       end
 

activerecord/lib/active_record/encryption/scheme.rb

@@ -53,7 +53,7 @@ def merge(other_scheme)
       end
 
       def to_h
-        { key_provider: @key_provider_param, key: @key, deterministic: @deterministic, downcase: @downcase, ignore_case: @ignore_case,
+        { key_provider: @key_provider_param, deterministic: @deterministic, downcase: @downcase, ignore_case: @ignore_case,
           previous_schemes: @previous_schemes_param, **@context_properties }.compact
       end
 

activerecord/lib/active_record/railtie.rb

@@ -357,11 +357,17 @@ class Railtie < Rails::Railtie # :nodoc:
     end
 
     initializer "active_record_encryption.configuration" do |app|
-      ActiveRecord::Encryption.configure \
-         primary_key: app.credentials.dig(:active_record_encryption, :primary_key),
-         deterministic_key: app.credentials.dig(:active_record_encryption, :deterministic_key),
-         key_derivation_salt: app.credentials.dig(:active_record_encryption, :key_derivation_salt),
-         **config.active_record.encryption
+      auto_filtered_parameters = ActiveRecord::Encryption::AutoFilteredParameters.new(app)
+
+      config.after_initialize do |app|
+        ActiveRecord::Encryption.configure \
+          primary_key: app.credentials.dig(:active_record_encryption, :primary_key),
+          deterministic_key: app.credentials.dig(:active_record_encryption, :deterministic_key),
+          key_derivation_salt: app.credentials.dig(:active_record_encryption, :key_derivation_salt),
+          **config.active_record.encryption
+
+        auto_filtered_parameters.enable if ActiveRecord::Encryption.config.add_to_filter_parameters
+      end
 
       ActiveSupport.on_load(:active_record) do
         # Support extended queries for deterministic attributes and validations
@@ -377,11 +383,6 @@ class Railtie < Rails::Railtie # :nodoc:
           ActiveRecord::Fixture.prepend ActiveRecord::Encryption::EncryptedFixtures
         end
       end
-
-      # Filtered params
-      if ActiveRecord::Encryption.config.add_to_filter_parameters
-        ActiveRecord::Encryption.install_auto_filtered_parameters_hook(app)
-      end
     end
 
     initializer "active_record.query_log_tags_config" do |app|

activerecord/test/cases/encryption/configurable_test.rb

@@ -43,23 +43,25 @@ class ActiveRecord::Encryption::ConfigurableTest < ActiveRecord::EncryptionTestC
 
   test "installing autofiltered parameters will add the encrypted attribute as a filter parameter using the dot notation" do
     application = OpenStruct.new(config: OpenStruct.new(filter_parameters: []))
-    ActiveRecord::Encryption.install_auto_filtered_parameters_hook(application)
 
-    NamedPirate = Class.new(Pirate) do
-      self.table_name = "pirates"
+    with_auto_filtered_parameters(application) do
+      NamedPirate = Class.new(Pirate) do
+        self.table_name = "pirates"
+      end
+      NamedPirate.encrypts :catchphrase
     end
-    NamedPirate.encrypts :catchphrase
 
     assert_includes application.config.filter_parameters, "named_pirate.catchphrase"
   end
 
   test "installing autofiltered parameters will work with unnamed classes" do
     application = OpenStruct.new(config: OpenStruct.new(filter_parameters: []))
-    ActiveRecord::Encryption.install_auto_filtered_parameters_hook(application)
 
-    Class.new(Pirate) do
-      self.table_name = "pirates"
-      encrypts :catchphrase
+    with_auto_filtered_parameters(application) do
+      Class.new(Pirate) do
+        self.table_name = "pirates"
+        encrypts :catchphrase
+      end
     end
 
     assert_includes application.config.filter_parameters, "catchphrase"
@@ -69,15 +71,23 @@ class ActiveRecord::Encryption::ConfigurableTest < ActiveRecord::EncryptionTestC
     ActiveRecord::Encryption.config.excluded_from_filter_parameters = [:catchphrase]
 
     application = OpenStruct.new(config: OpenStruct.new(filter_parameters: []))
-    ActiveRecord::Encryption.install_auto_filtered_parameters_hook(application)
 
-    Class.new(Pirate) do
-      self.table_name = "pirates"
-      encrypts :catchphrase
+    with_auto_filtered_parameters(application) do
+      Class.new(Pirate) do
+        self.table_name = "pirates"
+        encrypts :catchphrase
+      end
     end
 
     assert_equal application.config.filter_parameters, []
 
     ActiveRecord::Encryption.config.excluded_from_filter_parameters = []
   end
+
+  private
+    def with_auto_filtered_parameters(application)
+      auto_filtered_parameters = ActiveRecord::Encryption::AutoFilteredParameters.new(application)
+      yield
+      auto_filtered_parameters.enable
+    end
 end

activerecord/test/cases/encryption/encryptable_record_test.rb

@@ -64,7 +64,7 @@ class ActiveRecord::Encryption::EncryptableRecordTest < ActiveRecord::Encryption
   end
 
   test "encrypts multiple attributes with different options at the same time" do
-    post = EncryptedPost.create!\
+    post = EncryptedPost.create! \
       title: title = "The Starfleet is here!",
       body: body = "<p>the Starfleet is here, we are safe now!</p>"
 
@@ -308,7 +308,37 @@ def name
     assert_equal book, Marshal.load(Marshal.dump(book))
   end
 
+  test "supports decrypting data encrypted non deterministically with SHA1 when digest class is SHA256" do
+    ActiveRecord::Encryption.configure \
+      primary_key: "the primary key",
+      deterministic_key: "the deterministic key",
+      key_derivation_salt: "the salt",
+      support_sha1_for_non_deterministic_encryption: true
+
+    key_provider_sha1 = build_derived_key_provider_with OpenSSL::Digest::SHA1
+    key_provider_sha256 = build_derived_key_provider_with OpenSSL::Digest::SHA256
+
+    encrypted_post_class_sha_1 = Class.new(Post) do
+      self.table_name = "posts"
+      encrypts :title, key_provider: key_provider_sha1
+    end
+    encrypted_post_class_sha_1.create! title: "Post 1", body: "The post body", type: nil
+
+    encrypted_post_class_sha_256 = Class.new(Post) do
+      self.table_name = "posts"
+      encrypts :title, key_provider: key_provider_sha256
+    end
+
+    assert_equal "Post 1", encrypted_post_class_sha_256.last.title
+  end
+
   private
+    def build_derived_key_provider_with(hash_digest_class)
+      ActiveRecord::Encryption.with_encryption_context(key_generator: ActiveRecord::Encryption::KeyGenerator.new(hash_digest_class: hash_digest_class)) do
+        ActiveRecord::Encryption::DerivedSecretKeyProvider.new(ActiveRecord::Encryption.config.primary_key)
+      end
+    end
+
     class FailingKeyProvider
       def decryption_key(message) end
 

activerecord/test/cases/encryption/key_generator_test.rb

@@ -50,6 +50,6 @@ def assert_derive_key(secret, digest_class: OpenSSL::Digest::SHA256, length: 20)
                                                         .generate_key(ActiveRecord::Encryption.config.key_derivation_salt, length)
       assert_equal length, expected_derived_key.length
       ActiveRecord::Encryption.config.hash_digest_class = digest_class
-      assert_equal expected_derived_key, @generator.derive_key_from(secret, length: length)
+      assert_equal expected_derived_key, ActiveRecord::Encryption::KeyGenerator.new(hash_digest_class: digest_class).derive_key_from(secret, length: length)
     end
 end

guides/source/active_record_encryption.md

@@ -480,6 +480,11 @@ The default encoding for attributes encrypted deterministically. You can disable
 
 The digest algorithm used to derive keys. `OpenSSL::Digest::SHA1` by default.
 
+#### `config.active_record.encryption.support_sha1_for_non_deterministic_encryption`
+
+Supports decrypting data encrypted non-deterministically with a digest class SHA1. Default is false, which
+means it  will only support the digest algorithm configured in `config.active_record.encryption.hash_digest_class`.
+
 ### Encryption Contexts
 
 An encryption context defines the encryption components that are used in a given moment. There is a default encryption context based on your global configuration, but you can configure a custom context for a given attribute or when running a specific block of code.