Implement AR#inspect using ParameterFilter #34208

yskkin · 2018-10-13T10:04:35Z

Summary

AR instance support filter_parameters since #33756.
Though Regex or Proc is valid as filter_parameters, they are not supported as AR#inspect.
#33756 (comment)

This PR tries to fix this issue.

rails-bot · 2018-10-13T10:04:50Z

r? @rafaelfranca

(@rails-bot has picked a reviewer for you, use r? to override)

kamipo · 2018-10-13T14:17:26Z

activerecord/lib/active_record/core.rb

Now .to_set has been useless since self.class.filter_attributes.include?(attribute_name) is replaced.

We could save the cost of recompiling a ParameterFilter per #inspect call by instantiating it on the class when #filter_attributes= is called. For example:

# Returns columns which shouldn't be exposed while calling +#inspect+. def filter_attributes - if defined?(@filter_attributes) - @filter_attributes - else - superclass.filter_attributes - end + inspect_filter.filters # needs a new ParameterFilter.attr_reader :filters end # Specifies columns which shouldn't be exposed while calling +#inspect+. def filter_attributes=(attributes_names) - @filter_attributes = attributes_names.map(&:to_s).to_set + self.inspect_filter = ActiveSupport::ParameterFilter.new(attributes_names) + end + + # Or provide your own… + attr_writer :inspect_filter #:nodoc: + + # Returns columns which shouldn't be exposed while calling +#inspect+. + def inspect_filter #:nodoc: + if defined?(@inspect_filter) + @inspect_filter + else + superclass.inspect_filter + end end # Returns a string like 'Post(id:integer, title:string, body:text)'

I think inspect or pretty_print is not called in production for typical application.
So I will go like this to defer allocation.

def inspection_filter @inspection_filter ||= ActiveSupport::ParameterFilter.new(self.class.filter_attributes, expose: [nil]) end

kamipo · 2018-10-13T18:20:51Z

activerecord/lib/active_record/attribute_methods.rb

I think that anyone probably have no strong opinion to filter a value except nil.
So how about always filtering values whether or not considered as sensitive value?

It's @matthewd's suggestion and also have a test case.
#33756 (comment)

rails/activerecord/test/cases/filter_attributes_test.rb

Line 58 in 99c87ad

test "filter_attributes should not filter nil value" do

Can we change this?

kamipo · 2018-10-13T18:25:43Z

activerecord/lib/active_record/attribute_methods.rb

The attributes_for_inspect is internal API and is always passed self.class.filter_attributes, so we could just use ActiveSupport::ParameterFilter.new(self.class.filter_attributes) here instead of passing as a method argument.

kamipo · 2018-10-13T18:42:11Z

activerecord/lib/active_record/core.rb

.compact is no longer needed.

jeremy

This is a great change. It exposes new demands on ParameterFilter that we should probably satisfy. I left some review comments as well as suggestions for refactoring and API naming for you to consider.

In summary:

we can store one ParameterFilter instance rather than the list of filtered attributes
ParameterFilter should allow us to filter a single name, value pair rather than a full params hash
we needn't introduce new attributes_for_inspect public API, but we can extract a private API for attribute_names_for_inspect that's used by #inspect and #pretty_print
we need API to format an attribute value for #inspect, but attribute_for_inspect takes a name argument only

jeremy · 2018-10-13T18:06:24Z

activerecord/lib/active_record/attribute_methods.rb

Rather than expose public API now, let's start with a private method in Core that supports #inspect and #pretty_print. Public API may have different expectations; for example, #attribute_for_inspect doesn't filter.

Furthermore, if we move this to a private #attributes_for_inspect then we could rely on self.class.inspect_filter (see comment below) to provide the ParameterFilter instance, and we needn't take an attribute list argument.

jeremy · 2018-10-13T18:23:20Z

activerecord/lib/active_record/core.rb

We could save the cost of recompiling a ParameterFilter per #inspect call by instantiating it on the class when #filter_attributes= is called. For example:

# Returns columns which shouldn't be exposed while calling +#inspect+. def filter_attributes - if defined?(@filter_attributes) - @filter_attributes - else - superclass.filter_attributes - end + inspect_filter.filters # needs a new ParameterFilter.attr_reader :filters end # Specifies columns which shouldn't be exposed while calling +#inspect+. def filter_attributes=(attributes_names) - @filter_attributes = attributes_names.map(&:to_s).to_set + self.inspect_filter = ActiveSupport::ParameterFilter.new(attributes_names) + end + + # Or provide your own… + attr_writer :inspect_filter #:nodoc: + + # Returns columns which shouldn't be exposed while calling +#inspect+. + def inspect_filter #:nodoc: + if defined?(@inspect_filter) + @inspect_filter + else + superclass.inspect_filter + end end # Returns a string like 'Post(id:integer, title:string, body:text)'

jeremy · 2018-10-13T18:33:57Z

activerecord/lib/active_record/attribute_methods.rb

We can switch #inspect to #each_with_object and omit the return value.

jeremy · 2018-10-13T18:55:32Z

activesupport/lib/active_support/parameter_filter.rb

If a ParameterFilter is always meant to allow nil values, let's pass the option to #initialize.

Since our filters language always means filtering parameter keys, a new filter_nil option would suggest that we filter nil keys rather than nil values. Is there other language that suggests that we bypass the filters for certain values? Some possibilities:

# Too clunky? "Match keys to these filters, excepting nil values" ParameterFilter.new(filters, except_nil_values: true) # Too generic? "Match keys to these filters, with these exceptions" ParameterFilter.new(filters, except: ->(key, value) { value.nil? || … }) # Too particular? "Match keys to these filters, but treat these values as exceptions" ParameterFilter.new(filters, except_values: [ nil, "" ]) # Too unclear? "Match keys to these filters, but expose these values anyway" ParameterFilter.new(filters, expose: [ nil, "" ])

jeremy · 2018-10-13T19:15:44Z

activerecord/lib/active_record/attribute_methods.rb

This usage suggests that a ParameterFilter#filter_param(key, value) API would be desirable so we don't have to preprocess, filter, then postprocess an attributes hash. Instead we could enumerate attributes directly:

def attributes_for_inspect attributes.each_with_object({}) do |(attr_name, value), filtered| filtered[attr_name] = attribute_value_for_inspect(self.class.inspect_filter.filter_param(attr_name, value))) end end # change implementation to delegate to new value-formatter method… def attribute_for_inspect(attr_name) attribute_value_for_inspect(read_attribute(attr_name)) end # Extract implementation from current attribute_for_inspect def attribute_value_for_inspect(value) … end

I also considered that path, but gave up since proc filters with arity of 3 need an entire hash.

rails/activesupport/lib/active_support/parameter_filter.rb

Line 95 in 99c87ad

blocks.each { |b| b.arity == 2 ? b.call(key, value) : b.call(key, value, original_params) }

What should filter_param(key, value) API do if proc filter which takes three args is given?
Passing nil as third block parameter may be imcompatible since filters do not expect to get nil

filter_param is new API so backward compatibility may not be severe than I thought.
I will add it.

jeremy · 2018-10-13T19:25:40Z

activerecord/lib/active_record/core.rb

Only #inspect wants truncated strings and formatted Date values, so perhaps it should be calling #attribute_value_for_inspect here, rather than expecting #attributes_for_inspect to provide them:

attributes_for_inspect.map do |name, value| "#{name}: #{attribute_value_for_inspect(value)}" end.join(", ")

jeremy · 2018-10-13T19:31:03Z

activerecord/lib/active_record/core.rb

Both #inspect and #pretty_print need a list of attribute names to display. They don't share handling of the filtered values, however. That suggests extracting #attribute_names_for_inspect rather than an #attributes_for_inspect.

Then we may use the new single-parameter ParameterFilter API below instead of using a conditional:

pp.pp self.class.inspect_filter.filter_param(attr_name, value)

yskkin · 2018-10-14T06:00:03Z

activerecord/lib/active_record/core.rb

I think column_names here was left over from 8ab9daf so changed to attribute_names.

albertoalmagro · 2018-10-14T17:41:10Z

activerecord/lib/active_record/attribute_methods.rb

I'm not sure about the name of this method... I think something like format_for_inspect(value) reveals better the method's intention. Having said this... I also dislike the public API's method name attribute_for_inspect, but that isn't in the scope of this PR.

albertoalmagro · 2018-10-14T17:47:30Z

activesupport/lib/active_support/parameter_filter.rb

Typo:

Supported type of filter is +String+, +Regexp+, and +Proc+. ->
Supported types of filters are +String+, +Regexp+, and +Proc+.

albertoalmagro · 2018-10-14T17:47:56Z

activesupport/lib/active_support/parameter_filter.rb

Typo: type -> types

jeremy

Looking good for merge. Great work. Please squash commits and add a changelog entry 👍

kamipo · 2018-10-15T16:45:31Z

activesupport/lib/active_support/parameter_filter.rb

The :expose option is only implemented a string filter, not implemented yet regex and proc filters, how this option should be implemented is out of scope in this PR though.

The reason that this option is needed is to make filter_attributes behave differently than http params filter.

My preference is consistent behavior and less code, so the extra code for the purpose makes less sense to me.

ah I've misreading the doc, the :expose option behave filtering keys by filters first, then excluding values of the matched keys if values exactly matches in the expose collection.

At least we could implement more less code for the purpose even without this option.

diff --git a/activerecord/lib/active_record/core.rb b/activerecord/lib/active_record/core.rb index b83facf4b7..47fc382d70 100644 --- a/activerecord/lib/active_record/core.rb +++ b/activerecord/lib/active_record/core.rb @@ -540,7 +540,8 @@ def pretty_print(pp) pp.text attr_name pp.text ":" pp.breakable - value = inspection_filter.filter_param(attr_name, read_attribute(attr_name)) + value = read_attribute(attr_name) + value = inspection_filter.filter_param(attr_name, value) unless value.nil? if value == ActiveSupport::ParameterFilter::FILTERED pp.text value else @@ -598,7 +599,7 @@ def custom_inspect_method_defined? end def inspection_filter - @inspection_filter ||= ActiveSupport::ParameterFilter.new(self.class.filter_attributes, expose: [nil]) + @inspection_filter ||= ActiveSupport::ParameterFilter.new(self.class.filter_attributes) end end end

activerecord/lib/active_record/core.rb

kamipo · 2018-10-15T18:04:37Z

activerecord/lib/active_record/core.rb

If read_attribute(attr_name) returns "[FILTERED]", this condition would be true too even if the attribute is not filtered.

yskkin · 2018-10-16T02:54:44Z

I forgot to address method name of attribute_value_for_inspect.
I'll change it in next push/squash.

yskkin · 2018-10-18T04:48:25Z

I sqashed all the changes so far.
Do you have any other comment?
Thanks in advance ❤️

matthewd · 2018-10-18T06:08:24Z

I don't immediately have a better suggestion, and it doesn't need to block us merging this, but I'm a little uncomfortable with the fact it's called ParameterFilter now that it's being used to filter things that.. aren't parameters (except in the broadest possible "input value for code" sense of the word).

Does anyone else have feelings on that?

kamipo · 2018-10-18T10:14:43Z

activerecord/lib/active_record/core.rb

Is this alias necessary? mask.inspect isn't called in our test cases.

And also, I'm not fond to reference ActiveSupport::ParameterFilter::FILTERED repeatedly.
How about defining pretty_print only?

diff --git a/activerecord/lib/active_record/core.rb b/activerecord/lib/active_record/core.rb index 34bde3fbe9..9780258302 100644 --- a/activerecord/lib/active_record/core.rb +++ b/activerecord/lib/active_record/core.rb @@ -596,16 +596,10 @@ def custom_inspect_method_defined? def inspection_filter @inspection_filter ||= begin - mask = Class.new do - def to_s - ActiveSupport::ParameterFilter::FILTERED - end - alias inspect to_s - - def pretty_print(pp) - pp.text ActiveSupport::ParameterFilter::FILTERED - end - end.new.freeze + mask = DelegateClass(::String).new(ActiveSupport::ParameterFilter::FILTERED) + def mask.pretty_print(pp) + pp.text __getobj__ + end ActiveSupport::ParameterFilter.new(self.class.filter_attributes, mask: mask) end end

I added alias for completeness but that is not necessary.
I'll update as you suggested.

AR instance support `filter_parameters` since rails#33756. Though Regex or Proc is valid as `filter_parameters`, they are not supported as AR#inspect. I also add :mask option and #filter_params to `ActiveSupport::ParameterFilter#new` to implement this.

kamipo

👍

yskkin · 2018-10-26T02:57:26Z

As for the naming of ParameterFilter, I don't have strong opinion on it, but open to change it.

I came up with candidate like

SecurityFilter
HashFilter
ValueMask

That said, I feel changing class name is not the scope of this PR and inclined to just merge this and open new PR about discussing naming.

kamipo · 2018-10-26T10:05:02Z

Yeah, I think that changing the class name is not the scope of this PR too, it open to change though.

I think what to do naming the class depends on what filter is passed.

rails/activesupport/lib/active_support/parameter_filter.rb

Lines 43 to 46 in 32b03b4

    
           # Mask value of +params+ if key matches one of filters. 
        
           def filter(params) 
        
             compiled_filter.call(params) 
        
           end

Anyway, I'm going to merge this PR.

rails-bot bot added activerecord activesupport labels Oct 13, 2018

rails-bot assigned rafaelfranca Oct 13, 2018

rails-bot bot added the railties label Oct 13, 2018

kamipo reviewed Oct 13, 2018

View reviewed changes

activerecord/lib/active_record/core.rb Outdated

Copy link

Member

kamipo Oct 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.compact is no longer needed.

jeremy reacted with thumbs up emoji

jeremy reviewed Oct 13, 2018

View reviewed changes

yskkin commented Oct 14, 2018

View reviewed changes

albertoalmagro reviewed Oct 14, 2018

View reviewed changes

jeremy approved these changes Oct 15, 2018

View reviewed changes

kamipo reviewed Oct 15, 2018

View reviewed changes

activerecord/lib/active_record/core.rb Outdated Show resolved Hide resolved

kamipo reviewed Oct 15, 2018

View reviewed changes

yskkin force-pushed the inspect_with_parameter_filter branch from e1b020a to 924cc4e Compare October 17, 2018 05:26

kamipo reviewed Oct 18, 2018

View reviewed changes

yskkin force-pushed the inspect_with_parameter_filter branch from 924cc4e to 32b03b4 Compare October 19, 2018 05:18

kamipo approved these changes Oct 19, 2018

View reviewed changes

kamipo merged commit 6149080 into rails:master Oct 26, 2018

yskkin deleted the inspect_with_parameter_filter branch November 1, 2018 12:05

eugeneius mentioned this pull request Sep 7, 2019

Allow inspected records to be marshaled #37153

Merged

Implement AR#inspect using ParameterFilter #34208

Implement AR#inspect using ParameterFilter #34208

Uh oh!

Conversation

yskkin commented Oct 13, 2018

Summary

Uh oh!

rails-bot commented Oct 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kamipo Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy Oct 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yskkin commented Oct 16, 2018

jeremy Oct 13, 2018 •

edited

Loading

kamipo Oct 13, 2018 •

edited

Loading

jeremy left a comment •

edited

Loading

jeremy Oct 13, 2018 •

edited

Loading

jeremy Oct 13, 2018 •

edited

Loading

jeremy Oct 13, 2018 •

edited

Loading

jeremy Oct 13, 2018 •

edited

Loading