Strong parameters: allow hashes with unknown keys to be permitted #9454

Closed
spohlenz opened this Issue Feb 27, 2013 · 55 comments

Projects

None yet
@spohlenz
Contributor

From what I can tell, strong parameters currently has no ability to permit a hash with unknown keys. Where this would be particularly useful is for the newly supported Hstore and JSON data types -- a good example of this use case can be found here: http://schneems.com/post/19298469372/you-got-nosql-in-my-postgres-using-hstore-in-rails

I would have expected that passing an empty hash as an permitted value would allow a hash to be passed through. e.g.

params = ActionController::Parameters.new(product: { name: 'Test', data: { weight: '12kg' } })
params.require(:product).permit(:name, data: {})

however this does not pass the data hash through (though it is not documented that it should work).

Assigning the data parameter separately is an option but it complicates my code unnecessarily -- I would prefer to be able to stick with mass-assignment for all attributes.

Happy to work on a patch for this if this proposal is reasonable. I've only just started looking into strong parameters though so there may be drawbacks I haven't considered.

@spohlenz
Contributor

Another possibility which I think I like even better would be to allow permit! to take an optional parameter, to allow this:

params.require(:product).permit(:name).permit!(:data)
@sideshowcoder
Contributor

I am really interested in a way to solve this as well, so +1. I think your 2. approach looks cleaner to me as well, passing {} to permit all sub keys looks weird to me.

@senny
Member
senny commented Feb 27, 2013

strong parameters is not designed to handle every possible situation. We wanted to keep the API on point to handle the most common situations. As assigning a hash with unknown keys more or less defeates the purpose of strong parameters (restricting allowed keys), It's not supported because it could make your code vulnerable.

As you mentioned already you can simply fall back on a normal assignment if you know that you want to permit unknown keys.

/cc @fxn @rafaelfranca

@fxn
Member
fxn commented Feb 27, 2013

Confirm, the API is designed to whitelist every single key. If your use case does not fit well then you need to resort to Ruby. That flexibility is also by design, in the end it is just Ruby.

@fxn fxn closed this Feb 27, 2013
@spohlenz
Contributor

strong parameters is not designed to handle every possible situation. We wanted to keep the API on point to handle the most common situations.

Understand, but this doesn't exactly seem like an uncommon use case (particularly with the introduction of Hstore and JSON types), and with attr_accessible/attr_protected being removed from Rails core, options are severely reduced.

I would argue that this actually makes the API more consistent, since permit! would accept a list of keys, closely mirroring permit.

As assigning a hash with unknown keys more or less defeates the purpose of strong parameters (restricting allowed keys), It's not supported because it could make your code vulnerable.

Except you'd only be permitting an unknown hash on a single key within the params. It's certainly more secure than doing a permit! on the whole params hash (which is supported behaviour).

As you mentioned already you can simply fall back on a normal assignment if you know that you want to permit unknown keys.

My main problem with this is that I go from having to test one message (Product.create) to having to test three messages (Product.new, @product.data= and @product.save). The separate assignment for my hash also has to be duplicated across both my create and update actions.

Here's a concrete example of how my proposal would improve things. Before:

class ProductsController < ApplicationController
  def create
    @product = Product.new(product_params)
    @product.data = params[:product][:data]
    @product.save!
  end

  def update
    @product = Product.find(params[:id])
    @product.data = params[:product][:data]
    @product.update!(product_params)
  end

private
  def product_params
    params.require(:product).permit(:name)
  end
end

After:

class ProductsController < ApplicationController
  def create
    @product = Product.create!(product_params)
  end

  def update
    @product = Product.find(params[:id])
    @product.update!(product_params)
  end

private
  def product_params
    params.require(:product).permit(:name).permit!(:data)
  end
end

I do think this needs more discussion. It's all well and good to say just do it manually, but it feels wrong to have to work around a feature (and make my code more complicated), particularly when the alternatives have been removed.

@fxn
Member
fxn commented Feb 27, 2013

To avoid duplication set the trusted parameter in the helper (untested):

def product_params
  params.require(:product).permit(:name).tap do |whitelisted|
    whitelisted[:data] = params[:product][:data]
  end
end
@spohlenz
Contributor

Thanks @fxn. That is a reasonable solution I had not thought of. :)

@fxn
Member
fxn commented Feb 27, 2013

@spohlenz awesome :), this feedback was useful we are going to document a few edge cases like this or similar.

@senny senny added a commit to senny/rails that referenced this issue Mar 3, 2013
@senny senny integrate the strong params README into the AC guide.
The current ActionController guide does not mention strong parameters
at all. I integrated the README into the guide to explain the API.

I also included a section to illustrate that the API does not solve
all possible whitelisting scenarios.

The origin was #9454.
fb63753
@hakanensari
Contributor

@senny It may be helpful to have the Strong Parameters README mirror the last paragraph on using with Hstore and JSON data types. Latter ends up being a more obvious reference point thanks to Google.

@senny
Member
senny commented Jul 16, 2013

@hakanensari I'm not sure if duplicating the guides in the README is a good thing. They will get out of sync quickly. I'd like to keep the additional examples in the guides because they are the reference for Rails.

Maybe we could link from the README to the relevant section in the guides? (http://guides.rubyonrails.org/action_controller_overview.html#more-examples)

@fxn what do you think?

@fxn
Member
fxn commented Jul 16, 2013

@senny definitely linking.

@lanej
lanej commented Jan 6, 2014

The solution mentioned by @fxn only works if you have decided to log instead of raise when un-permitted parameters are present. There are plenty of situations (metadata, configuration, etc.) where whitelisting a sub-hash is completely valid.

I think that @spohlenz solution might be the most similar and I will have to implement something like it.

@fxn
Member
fxn commented Jan 6, 2014

I like this approach #12609.

@lanej
lanej commented Jan 6, 2014

that's a sufficient workaround but hardly a good solution. what about params.require(:product).permit(:name, :data => Hash) and allowing Hash as a permitted scalar value IF it is explicitly defined.

@fxn
Member
fxn commented Jan 6, 2014

Yes, I believe this use case deserves API.

Something like data: :permit! would be nice, I'd need to think about a way to express that that feels like a natural addition to the existing API.

@lanej
lanej commented Jan 6, 2014

fyi, the approach in #12609 is not feasible for hashes of a depth greater than 2.

@jhubert
Contributor
jhubert commented Jan 14, 2014

👍 This has also come up with us as an issue for supporting HStore metadata values on our models.

Also, and perhaps I'm doing something wrong, but the solution from @fxn seems to actually set nil values on the params if they aren't already set:

params = ActionController::Parameters.new({ test: true })
params.permit(:test).tap do |whitelisted|
  whitelisted[:more] = params[:more]
end
# => {"test"=>true, "more"=>nil}
@lanej
lanej commented Jan 14, 2014

@jhubert that definitely makes sense. whitelisted is a hash and you are setting whitelisted[:more] to params[:more] which is nil.

@jhubert
Contributor
jhubert commented Jan 14, 2014

@lanej Thank. Yeah, that makes sense. It was unexpected behavior for me and my tests failed because the parameter was being set to nil instead of being unprovided. Figured I would mention it in case other people just copy / paste it expecting it to just permit the variable if it was provided and not change the params themselves.

@richardkmichael

@jhubert FWIW, ActionController::Parameters#fetch() will raise ActionController::ParameterMissing, instead of simply returning nil.

[1] pry(main)> p = ActionController::Parameters.new({test: true})
=> {"test"=>true}
[2] pry(main)> p[:foo]
=> nil
[3] pry(main)> p.fetch :foo
ActionController::ParameterMissing: param is missing or the value is empty: foo # .... 
[4] pry(main)> p.fetch :foo, nil
=> nil
@nhattan
nhattan commented Jun 20, 2014

thanks, this works for me:

def product_params
  params.require(:product).permit(:name).tap do |while_listed|
    while_listed[:data] = params[:product][:data]
  end
end
@chinshr
chinshr commented Jan 6, 2015

@fxn agree, that's an acceptable solution. @nhattan, I ended up adding testing for nil value.

def product_params
  params.require(:product).permit(:name).tap do |whitelisted|
    whitelisted[:data] = params[:product][:data] if params[:product][:data]
  end
end
@NullVoxPopuli

i think this would be a neat feature to add.
I have data that looks like this currently:
=> {"47"=>{"M"=>{"quantity"=>"2"}, "XL"=>{"quantity"=>"4"}, "XXXL"=>{"quantity"=>"1"}}}
where it maps the ID of an object to some properties for that particular object...

but maybe there is a better way to do that? idk. ('usually is)

@cohesivejones86

My company doesn't use the key value of the nested we expect the nested hash to have and id property if its persisted for example
{ root_model: {
nested_model: {
GUID: { id: 1, value: 'something persisted' },
ANOTHER GUID: { value: 'a new record as it has no id'},
GUID: { id: 3, _destroy: true,
value: 'a record to delete but maybe we should
have another controller for this'
}
}
}
maybe we need to have wildcards ie %id% or %GUID% or just stop using nested attributes

@nhattan
nhattan commented Apr 15, 2015

@chinshr You're right! thanks for pointing this case.

@joelpresence

This is an issue for us as well - we have nested hashes where near the top we have unknown keys (e.g. product ids) but underneath those unknown keys we have hashes with known keys. Can we somehow incorporate this into the strong params API? As it stands it doesn't work well for serialized Hash with arbitrary keys at any point in the nesting ...

Thanks!

@chrisbloom7

Adding this from http://stackoverflow.com/a/24752108/83743 since it's what finally worked for me:

If you use :raise instead of :log for config.action_controller.action_on_unpermitted_parameters in your environment then remember to remove properties from params before calling permit. Then the method will be

def product_params
  properties = params[:product].delete(:properties)
  params.require(:product).permit(:title, :description).tap do |whitelisted|
    whitelisted[:properties] = properties
  end
end
@gregblass

Really stuck here with this issue. See my stackoverflow question here:

http://stackoverflow.com/questions/31945048/rails-4-strong-params-with-multiple-objects-and-integer-keys/31947212#31947212

In summary, I'm submitting 2-4 records at once.

This is what I've reproduced in the console:

params = ActionController::Parameters.new( { knockouts: {1 => {volume: 3.0, temperature: 2.0}, 2 => {volume: 4.1, temperature: 2.5}}} )

permitted = params.require( :knockouts ).permit( :id => [ :volume, :temperature ])
Unpermitted parameters: 1, 2
=> {} 

I just want to whitelist the first value to be any integer...the ID of the record coming through. Is that possible?

@ArunSakthivel

I think, This will help,

params.require( :knockouts ).permit(params[:knockouts].keys.map {|c| {:"#{c}" => [:volume, :temperature]}})
@nhattan
nhattan commented Sep 13, 2015

@ArunSakthivel I think the result he wants is:

=> {1 => {volume: 3.0, temperature: 2.0}, 2 => {volume: 4.1, temperature: 2.5}}

but your code still returns {}

@uberllama
Contributor

FYI, this is code I'm using in production to get around this issue.

# Strong params workaround to allow arbitrary json column field values on params
#
# @example
#   post_params
#     permit_json_params(params[:post], :custom_field_values) do
#       params.require(:post).permit(:title, :body)
#     end
#   end
#
# @param hash [Hash]   params hash
# @param key  [Symbol] key to allow arbitrary values for
#
def permit_json_params(hash, key)
  json_values = hash.delete(key)
  permitted_params = yield
  permitted_params[key] = json_values if json_values
  permitted_params
end
@ecuageo
ecuageo commented Oct 5, 2015

I know this won't work for everyone's use case. If you have some validation in your model through store_accessor you would have a list of keys to permit through stored_attributes.
klass = params.require(:type).constantize
params.permit(:type, content: klass.stored_attributes[:settings])

@sdepold sdepold referenced this issue in rails/strong_parameters Oct 13, 2015
Open

Add possibility to trust a parameter branch #231

@PandaWhisperer

None of the workarounds suggested in this thread seemed to work for me (Rails 4.2.4), so I devised the following workaround:

def product_params
  properties_keys = params[:product][:properties].keys
  params.require(:product).permit(:title, :description, properties: properties_keys)
end

Hope that helps someone.

@jhubert
Contributor
jhubert commented Jan 5, 2016

@PandaWhisperer 👍 Thanks

@lanej
lanej commented Jan 5, 2016

@PandaWhisperer @jhubert problem with that solution is that it requires you to enforce the presence of params[:product] AND params[:product][:properties] before you can then allow / disallow.

@PandaWhisperer

@lanej you are correct. Might need to throw an if in there to safeguard against that.

@sashman
sashman commented Jan 11, 2016

@uberllama I have modified your answer a little and it worked nicely for any number of dynamic keys. Example params: post: { something: {...}, something_else: {...} }

  def post_params
    permit_key_params(params[:post]) do
      params.require(:post)
    end
  end

  def permit_key_params(hash)
    permitted_params = yield
    hash.keys.each do |key|
      values = hash.delete(key)
      permitted_params[key] = values if values
    end
    permitted_params
  end
@iagopiimenta

This monkey patch can help: https://gist.github.com/iagopiimenta/43a66712c021a9d3f540#file-monkey_patch_rails_parameters-rb-L21

It can be used like this:

params = ActionController::Parameters.new(
  form: {
    scalar: 'foobar',
    many: [
      {
        field1: 'foo'
      },
      {
        fiedl2: 'bar'
      }
    ],
    single: {
      filed3: 'baz'
    }
  }
)

params.require(:form).permit(:scalar, many: Hash, single: Hash)
@aliibrahim

I modified @PandaWhisperer solution to safeguard against non-existing key:

def product_params
  properties_keys = params[:product].try(:fetch, :properties, {}).keys
  params.require(:product).permit(:title, :description, properties: properties_keys)
end
@koenpunt
Contributor
koenpunt commented Mar 21, 2016 edited

For nested params I now use the following:

def item_params
  params.require(:item).permit(values: permit_recursive_params(params[:item][:values]))
end

def permit_recursive_params(params)
  params.map do |key, value|
    if value.is_a?(Array)
      { key => [ permit_recursive_params(value.first) ] }
    elsif value.is_a?(Hash) || value.is_a?(ActionController::Parameters)
      { key => permit_recursive_params(value) }
    else
      key
    end
  end
end

Edit:
Updated version can be found in this gist

@MSCAU
MSCAU commented Jul 13, 2016

@koenpunt - very nice, thanks. I need something like that as I am trying to store a tree (of variable depth) in a JSONB column. However, your permit_recursive_params output breaks for me where the value is an array of hashes.

This tree:

budget: {
    tree: {
        name: "Australia",
        value: 39904,
        id: 6,
        description: "",
        type: {
            name: "Other",
            icon: "<i class="fa fa-pagelines"></i>"
        },
        children: [
            {
            name: "Boxes",
            value: 451609,
            id: 7,
            description: "Used to store things.",
            type: {
                name: "Storage",
                icon: "<i class="fa fa-archive"></i>"
            },
            children: [
etc...

becomes this in PostgreSQL 9.4:

{"id": "6", "name": "Australia", "type": {"icon": "<i class=\"fa fa-pagelines\"></i>", "name": "Other"}, "value": "39904", "children": {"0": {}, "1": {}}, "description": ""}

FWIW, here's the strong parameter definition I am using:

params.require(:budget).permit({tree: permit_recursive_params(params[:budget][:tree])})

Can you advise what do do to get those children to render correctly?

@koenpunt
Contributor

With the data you provide, I believe I get correct results: https://gist.github.com/koenpunt/ac279e05cfeb0954ca763344fc0240b4

@bdmac
Contributor
bdmac commented Jul 13, 2016 edited

@koenpunt I'm hitting an error as well with permit_recursive_params calling map on a String object eventually. Here is a small example:

params = ActionController::Parameters.new(
  budget: {
    tree: {
      name: "Australia",
      value: 39904,
      id: 6,
      description: "",
      type: {
        name: "Other",
        icon: "<i class=\"fa fa-pagelines\"></i>"
      },
      children: [
        { required: ["phone"] },
        { required: ["email"] }
      ]
    }
  }
)

Running this through permit_recursive_params gives NoMethodError: undefined method 'map' for "phone":String

@bdmac
Contributor
bdmac commented Jul 13, 2016

A working version looks like this:

def permit_recursive_params(params)
  (params.try(:to_unsafe_h) || params).map do |key, value|
    if value.is_a?(Array)
      value = value.first
      if value.is_a?(Array) || value.is_a?(Hash)
        { key => [ permit_recursive_params(value) ] }
      else
        { key => [] }
      end
    elsif value.is_a?(Hash) || value.is_a?(ActionController::Parameters)
      { key => permit_recursive_params(value) }
    else
      key
    end
  end
end
@koenpunt
Contributor

@bdmac good catch, I've updated my gist to check if the first value responds to map, and if so, pass it along.

@koenpunt
Contributor

And now that to_unsafe_h is used, we no longer have to check for is_a?(ActionController::Parameters)

@MSCAU
MSCAU commented Jul 14, 2016 edited

Thanks, @koenpunt. From the server logs, it looks as though the JSON I submit to the server (ie. as I originally presented it):

children: [{},{}, etc... ]

is getting converted by Rails to

"children": { "0": {}, "1": {}, etc... }

I just need to figure out why...

@MSCAU
MSCAU commented Jul 14, 2016 edited

Hmmm, seems the issue is in the JS before the AJAX call, sorry. jQuery needs contentType to be set to "application/json". See http://stackoverflow.com/questions/6410810/rails-not-decoding-json-from-jquery-correctly-array-becoming-a-hash-with-intege

@Nowaker
Nowaker commented Aug 26, 2016 edited

Hey @fxn and @senny, it's the year of 2016 now. Rails got background jobs and even websockets, things that were originally neglected/ignored. Since 2013, the use of free-form JSON and Hstore values has rapidly grown too, and maybe it's high time to not ignore them either. The values submitted to them are free-form by design, therefore not subject to any whitelisting/filtration. But they don't live alone. They live inside models whose fields are whitelisted and filtered against mass assignment.

Consider Stripe gateway. They allow users to store free-form metadata on customer object, and that's just a hash. How to achieve that in Rails if the JSON request looks like this: POST /customers, {description: ..., source: ..., metadata: {... free form data here...}}. Answer: fight the framework - and many people in this thread did it their own way. Currently, if there's any JSON field without a predefined "schema" inside the params, Strong Parameters has to be worked around for each field. In 2016, letting a non-checked JSON in is a common situation.

There's already @sdepold's pull request to allow this here: rails/strong_parameters#231. I hope you can reevaluate this feature and give it a green light. If there's any issue/feedback about that PR, I can take it over from the original submitter and finish it up if he's unreachable. Please let me know.

@fxn
Member
fxn commented Oct 20, 2016 edited

@Nowaker agree, we've been conservative, but time says this use-case deserves a dedicated API.

The :* proposal seems good to me. The patch would need some work, but the basic idea of a wildcard symbol to say "accept whatever here" might work.

The patch is no big deal, I might write one based on that PR (and give credit of course).

@fxn
Member
fxn commented Nov 11, 2016 edited

Implemented here.

@stiig
stiig commented Nov 12, 2016 edited

it would be nice if it also allow to set structure incoming hash, like a

params.permit(preferences: [{:scheme, font: {:name, :size}}])

for next cases in params:

params = ActionController::Parameters.new(
    username: "fxn",
    preferences: [
        {
            scheme: "Marazul",
            font: {
                name: "Source Code Pro",
                size: 12
            }
        },
        {
            scheme: "new scheme",
            font:
                {
                    name: "Another Font",
                    size: 14
                }
        }
    ])
@temirov
temirov commented Dec 22, 2016

I must be missing something, but Rails 5.0.1 still gives me grief:

irb(main):011:0> ActionController::Parameters.new(
irb(main):012:1*   items: [
irb(main):013:2*     { data: { a: 1}},
irb(main):014:2*     { data: { b: 2}}
irb(main):015:2>   ]
irb(main):016:1> ).permit(items: [ data: {}]).to_h
=> {"items"=>[{"data"=>{}}, {"data"=>{}}]}
@fxn
Member
fxn commented Dec 22, 2016

Patch versions don't have new features, this is going to come with 5.1.

@ignaciocapuccio
ignaciocapuccio commented Jan 5, 2017 edited

I modified @PandaWhisperer solution to safeguard against non-existing key:

def product_params
properties_keys = params[:product].try(:fetch, :properties, {}).keys
params.require(:product).permit(:title, :description, properties: properties_keys)
end

@aliibrahim thanks, that worked for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment