New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do NOT return BigDecimal as string in JSON - breaks DynamoDB support #25017

Closed
joelpresence opened this Issue May 14, 2016 · 12 comments

Comments

Projects
None yet
8 participants
@joelpresence

joelpresence commented May 14, 2016

Steps to reproduce

First, thanks for all of your hard work Rails!

Amazon AWS DynamoDB stores ALL numbers as BigDecimal. This is beyond my control and not something I can change. I have a client app sending in arbitrary JSON data like the following:

{
  "user_id": "some_id",
  "avatar_config": {
    "has_moustache": true,
    "some_string_attribute": "attribute_foo",
    "some_array_attribute": [32.3, 14],
    "some_hash_attribute": {
      "key1": "value1",
      "key2": "value2"
    },
    "some_numeric_attribute": 55.67
  }
}

Some of the field values are strings, some are floats and some are integers. I am not doing anything with giant numbers or arbitrary precision floats. I'm just sending in regular numbers.

However, Rails now sends back all BigDecimal as JSON strings (see) and you have removed the encode_big_decimal_as_string boolean option which I could have used to fix this (see).

This now means that the client sends in actual JSON numbers like 32.3 but gets back JSON strings like "32.3". This is a strong violation of the Principle of Least Surprise.

I don't have control over the client and it is sending arbitrary JSON, so I can't get the client to force these numbers returned as strings back into being numbers clientside. And because I have to return arbitrary JSON, I cannot just use Rabl or some other serialization templating tool to force the BigDecimal back to regular numbers serverside. So this is making it difficult for me to back Rails with DynamoDB as my persistent data store and also send and receive data to my client via a JSON API.

The client is _NOT_ sending in BigDecimal and knows nothing about BigDecimal, the client is sending regular numbers and expects regular numbers back, so the comment "the other other end knows by contract that the data is supposed to be a BigDecimal" (see) is totally inapplicable in this case.

I suppose I could write some code that descends through the returned dictionary, before converting it to JSON, and convert all BigDecimal to Ruby numbers (e.g. Float), but this seems possibly error-prone and a PITA.

Please bring back encode_big_decimal_as_string, otherwise you are making it difficult for people to back a Rails apps with DynamoDB. And this is thousands of people, not just me.

If you can provide some guidance, I'd be happy to submit a PR to restore encode_big_decimal_as_string.

(Guidelines for creating a bug report are available
here
)

Expected behavior

I should have a choice as to whether BigDecimal are returned as a string or regular old number in JSON.

Actual behavior

All BigDecimal are returned as string no matter what. This totally breaks DynamoDB support because DynamoDB stores all numbers as BigDecimal.

System configuration

Rails version:
4.2.5.2
Ruby version:
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca May 14, 2016

Member

Thank you for the report. If you need that behavior use the gem recommended in the deprecation message.

Member

rafaelfranca commented May 14, 2016

Thank you for the report. If you need that behavior use the gem recommended in the deprecation message.

@joelpresence

This comment has been minimized.

Show comment
Hide comment
@joelpresence

joelpresence May 14, 2016

@rafaelfranca thanks for responding. :-)

Look, I don't mean to sound like a whiner and I'm not arguing for the sake of arguing. But I feel disappointed by this response to be honest. The decision to remove encode_big_decimal_as_string breaks literally thousands or tens of thousands of active production rails apps that use the very popular Amazon DynamoDB. Removing this option was a step backward in Rails functionality, not an improvement. I realize that you may have had other reasons for removing it, but I don't feel like enough consideration was given to all of the thousands of Rails apps that it will impact.

I'd like to respectfully ask that we re-open this issue. I don't feel that it has been solved or that it should be marked as closed.

By "the gem recommended in the deprecation message" I assume you mean activesupport-json_encoder? I see that that gem has not had any commits since Oct 22, 2015 which makes me nervous. Is that gem even maintained any more? Can I count on that gem getting security updates?

Also, what is the performance hit when resorting back to activesupport-json_encoder vs the current Rails 4.2+ default? I see that activesupport-json_encoder is pure Ruby whereas I assume the newer default is native, so won't I see a performance hit?

You're also asking me to change the JSON encoding gem for an active production app used by hundreds of thousands of active users - I don't want to do that lightly. Of course I have full automated test suites, but still, I'm reluctant to change json encoding just to get around this.

And I _still_ argue that this is a violation of the principle of least surprise: if a user PUTs 3.14 as a number to my API, they expect to get 3.14 as a number back when they GET. However, because DynamoDB stores all numbers as BigDecimal and because Rails now encodes BigDecimal as a string, the user now GETs back "3.14" a string when they PUT 3.14 a number. That is totally unexpected and buggy. These users are not using large-magnitude or arbitrary-precision floating point numbers.

Is there another way to work around this other than using activesupport-json_encoder? For example, could I personally override BigDecimal.as_json and where in my Rails app setup should I do that to ensure that it takes effect and holds / stays sticky?

Thanks again for all of your help and enjoy your weekend. I really appreciate your hard work on Rails.

Best Wishes,

Joel

joelpresence commented May 14, 2016

@rafaelfranca thanks for responding. :-)

Look, I don't mean to sound like a whiner and I'm not arguing for the sake of arguing. But I feel disappointed by this response to be honest. The decision to remove encode_big_decimal_as_string breaks literally thousands or tens of thousands of active production rails apps that use the very popular Amazon DynamoDB. Removing this option was a step backward in Rails functionality, not an improvement. I realize that you may have had other reasons for removing it, but I don't feel like enough consideration was given to all of the thousands of Rails apps that it will impact.

I'd like to respectfully ask that we re-open this issue. I don't feel that it has been solved or that it should be marked as closed.

By "the gem recommended in the deprecation message" I assume you mean activesupport-json_encoder? I see that that gem has not had any commits since Oct 22, 2015 which makes me nervous. Is that gem even maintained any more? Can I count on that gem getting security updates?

Also, what is the performance hit when resorting back to activesupport-json_encoder vs the current Rails 4.2+ default? I see that activesupport-json_encoder is pure Ruby whereas I assume the newer default is native, so won't I see a performance hit?

You're also asking me to change the JSON encoding gem for an active production app used by hundreds of thousands of active users - I don't want to do that lightly. Of course I have full automated test suites, but still, I'm reluctant to change json encoding just to get around this.

And I _still_ argue that this is a violation of the principle of least surprise: if a user PUTs 3.14 as a number to my API, they expect to get 3.14 as a number back when they GET. However, because DynamoDB stores all numbers as BigDecimal and because Rails now encodes BigDecimal as a string, the user now GETs back "3.14" a string when they PUT 3.14 a number. That is totally unexpected and buggy. These users are not using large-magnitude or arbitrary-precision floating point numbers.

Is there another way to work around this other than using activesupport-json_encoder? For example, could I personally override BigDecimal.as_json and where in my Rails app setup should I do that to ensure that it takes effect and holds / stays sticky?

Thanks again for all of your help and enjoy your weekend. I really appreciate your hard work on Rails.

Best Wishes,

Joel

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca May 14, 2016

Member

The gem was extracted from the Rails framework. It is still maintainer and will be maintainer as any other gem that was extracted. We don't have plans to integrate it again. You can find more about this decision here #12183. I was not involved on it so I'll defer to @chancancode the decision to reopen this issue or not.

Member

rafaelfranca commented May 14, 2016

The gem was extracted from the Rails framework. It is still maintainer and will be maintainer as any other gem that was extracted. We don't have plans to integrate it again. You can find more about this decision here #12183. I was not involved on it so I'll defer to @chancancode the decision to reopen this issue or not.

@kamen-hursev

This comment has been minimized.

Show comment
Hide comment
@kamen-hursev

kamen-hursev Aug 5, 2016

The suggested activesupport-json_encoder looks like is not maintained any more, or at least doesn't support Rails 5. Any advice how this behaviour can be brought back?

kamen-hursev commented Aug 5, 2016

The suggested activesupport-json_encoder looks like is not maintained any more, or at least doesn't support Rails 5. Any advice how this behaviour can be brought back?

@pixeltrix

This comment has been minimized.

Show comment
Hide comment
@pixeltrix

pixeltrix Aug 5, 2016

Member

@joelpresence and @kamen-hursev I'm wondering why it's not just a matter of overriding the BigDecimal#as_json method in your applications?

Member

pixeltrix commented Aug 5, 2016

@joelpresence and @kamen-hursev I'm wondering why it's not just a matter of overriding the BigDecimal#as_json method in your applications?

@kamen-hursev

This comment has been minimized.

Show comment
Hide comment
@kamen-hursev

kamen-hursev Aug 8, 2016

Yes, this is what I eventually did. Just wanted to see if you have some other plans about this matter.

kamen-hursev commented Aug 8, 2016

Yes, this is what I eventually did. Just wanted to see if you have some other plans about this matter.

@pixeltrix

This comment has been minimized.

Show comment
Hide comment
@pixeltrix

pixeltrix Aug 8, 2016

Member

@kamen-hursev since the workaround is simple, then adding a layer of configuration for what is a very application specific requirement seems offer little benefit to the wider Rails userbase. There are all sorts of scenarios where the basic JSON datatypes don't cover what's needed - dates and times being the obvious one. What if there was an API that needed to encode a time as a UNIX timestamp - should we add that as an option?

The main point of these configuration options is to provide compatibility during a transition - a recent example is the change of to_time preserving the timezone in Ruby 2.4. Since we support Rails 5 on 2.2.2 or later we need to cater for it, but once we're only supporting 2.4 then we'll remove that option.

Member

pixeltrix commented Aug 8, 2016

@kamen-hursev since the workaround is simple, then adding a layer of configuration for what is a very application specific requirement seems offer little benefit to the wider Rails userbase. There are all sorts of scenarios where the basic JSON datatypes don't cover what's needed - dates and times being the obvious one. What if there was an API that needed to encode a time as a UNIX timestamp - should we add that as an option?

The main point of these configuration options is to provide compatibility during a transition - a recent example is the change of to_time preserving the timezone in Ruby 2.4. Since we support Rails 5 on 2.2.2 or later we need to cater for it, but once we're only supporting 2.4 then we'll remove that option.

@kamen-hursev

This comment has been minimized.

Show comment
Hide comment
@kamen-hursev

kamen-hursev Aug 8, 2016

@pixeltrix fair point, you can't support all different use cases with config options. Just I got the impression that activesupport-json_encoder is "still maintainer and will be maintainer" as @rafaelfranca said and wanted to confirm if something is planned for Rails 5.

Thanks for the info!

kamen-hursev commented Aug 8, 2016

@pixeltrix fair point, you can't support all different use cases with config options. Just I got the impression that activesupport-json_encoder is "still maintainer and will be maintainer" as @rafaelfranca said and wanted to confirm if something is planned for Rails 5.

Thanks for the info!

@chancancode

This comment has been minimized.

Show comment
Hide comment
@chancancode

chancancode Aug 8, 2016

Member

@joelpresence sorry for not getting back to you earlier, here is an explanation with more details.

...literally thousands or tens of thousands of active production rails apps ... use the very popular Amazon DynamoDB...

[citation needed] 😐

I personally find that number very difficult to imagine, and I don't think that kind of exaggeration is helping your case here. For the future, let's stick to facts.

However, even if we do go with that, I still believe this to be a better default. I'll try to explain.

Presumably, the number is represented in BigDecimal for a reason – usually that the number came from a high-precision source and/or is used in context where the precision matters (e.g. calculating money).

In this case, that decision was made for you by the database driver. Presumably, that is for a good and important reason (otherwise, you should just file that as a bug against your DB driver, since paying the performance penalty of BigDecimals for no reason seems a bit silly). Perhaps your database stores them internally as a high-precision numeric data type and offers certain guarantees about that.

If your DB offers that kind of feature, then presumably at least some of those "thousands of apps" would try to use that feature and depend on those guarantees. While the JSON spec does not put a limit on the precision of the number data type, in practice all clients would parse that into a float/double data type, so suddenly the 0.1234567890123456789012345678901234567890123456789 Bitcoins you sent across the wire became 0.12345678901234568.

Perhaps it is not a big deal in isolation, but if you process thousands or tens of thousands of transactions per day, you could end up losing a lot of money. I don't get paid doing this, and I definitely couldn't afford people coming after me for their losses. (Dear lawyers: this statement does not imply warranty – see our LICENSE for details.)

Hopefully you can now see why this is the better default for BigDecimals.

However, since you are asking this, I assume you are not sending Bitcoins across the wire. Maybe you are just transmitting a count of the number of kitten gifs you have in your collection, or perhaps a timestamp. I understand that this protection mechanism could be annoying there.

No problem. If you assert that none of these matters at all, anywhere in your application, and you are 100% sure that none of your gems, engines and so on depend on it, you can simply drop the precision yourself, by down-casting them into a Float before encoding:

require 'active_support'
require 'active_support/core_ext/object/json'

class BigDecimal
  def as_json(*)
    to_f
  end
end

In fact, if you are sure that you are only transmitting kitten gif counts, and since those counts must be integers, you can even do this:

require 'active_support'
require 'active_support/core_ext/object/json'

class BigDecimal
  def as_json(*)
    to_i
  end
end

This snippet converts all BigDecimals into integers before encoding them as json, i.e. BigDecimal.new("1.23456789").to_json => "1".

If that sounded like a terrible idea, that's because it is a terrible idea! You see, changing global defaults like that would affect everything in your application. Even though your code – incidentally – only uses BigDecimals to represent integers/low-precision floats today (or rather, you don't care that they are high-precision), it could suddenly change tomorrow when you add a new feature, add a new gem, etc, and you might have forgotten about this change you made. Or it could be a new developer that got hired on to the project.

The point is, your claim that it "doesn't matter" to your application is probably overly strong. My personally recommendation is that you do this on a case-by-case scenario, either on the server or on the client.

On the server, you can use the serializer pattern (something like Active Model Serializers or jsonapi-resources) to transform your data before encoding them. If you go with those, you can easily customize the type for each field based on your domain knowledge, and/or use the usual OO/Ruby/meta-programming techniques to make things more concise. Alternatively, you can also used view-based libraries like jbuilder, which you can write custom helpers for the same purpose.

On the client, you would need to use whatever transformation hooks your framework provides, which is out-of-scope for the discussion here.

Two extra data-points:

  1. DynamoDB itself also uses strings as the wire-format for its numeric type: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html
  2. The standard library JSON gem (which Rails uses as the encoding backend) also uses strings to encode BigDecimals out of the box

Arguably, it might be more "unexpected and buggy" to not encode them as strings.


@kamen-hursev the gem is scheduled to be EOL'ed with Rails 5. However, I believe that it should be fairly easy to support it for another minor release or so. If, after reading this response, you still think you have a good reason to go with that route, I can investigate extending the support for that gem.

Member

chancancode commented Aug 8, 2016

@joelpresence sorry for not getting back to you earlier, here is an explanation with more details.

...literally thousands or tens of thousands of active production rails apps ... use the very popular Amazon DynamoDB...

[citation needed] 😐

I personally find that number very difficult to imagine, and I don't think that kind of exaggeration is helping your case here. For the future, let's stick to facts.

However, even if we do go with that, I still believe this to be a better default. I'll try to explain.

Presumably, the number is represented in BigDecimal for a reason – usually that the number came from a high-precision source and/or is used in context where the precision matters (e.g. calculating money).

In this case, that decision was made for you by the database driver. Presumably, that is for a good and important reason (otherwise, you should just file that as a bug against your DB driver, since paying the performance penalty of BigDecimals for no reason seems a bit silly). Perhaps your database stores them internally as a high-precision numeric data type and offers certain guarantees about that.

If your DB offers that kind of feature, then presumably at least some of those "thousands of apps" would try to use that feature and depend on those guarantees. While the JSON spec does not put a limit on the precision of the number data type, in practice all clients would parse that into a float/double data type, so suddenly the 0.1234567890123456789012345678901234567890123456789 Bitcoins you sent across the wire became 0.12345678901234568.

Perhaps it is not a big deal in isolation, but if you process thousands or tens of thousands of transactions per day, you could end up losing a lot of money. I don't get paid doing this, and I definitely couldn't afford people coming after me for their losses. (Dear lawyers: this statement does not imply warranty – see our LICENSE for details.)

Hopefully you can now see why this is the better default for BigDecimals.

However, since you are asking this, I assume you are not sending Bitcoins across the wire. Maybe you are just transmitting a count of the number of kitten gifs you have in your collection, or perhaps a timestamp. I understand that this protection mechanism could be annoying there.

No problem. If you assert that none of these matters at all, anywhere in your application, and you are 100% sure that none of your gems, engines and so on depend on it, you can simply drop the precision yourself, by down-casting them into a Float before encoding:

require 'active_support'
require 'active_support/core_ext/object/json'

class BigDecimal
  def as_json(*)
    to_f
  end
end

In fact, if you are sure that you are only transmitting kitten gif counts, and since those counts must be integers, you can even do this:

require 'active_support'
require 'active_support/core_ext/object/json'

class BigDecimal
  def as_json(*)
    to_i
  end
end

This snippet converts all BigDecimals into integers before encoding them as json, i.e. BigDecimal.new("1.23456789").to_json => "1".

If that sounded like a terrible idea, that's because it is a terrible idea! You see, changing global defaults like that would affect everything in your application. Even though your code – incidentally – only uses BigDecimals to represent integers/low-precision floats today (or rather, you don't care that they are high-precision), it could suddenly change tomorrow when you add a new feature, add a new gem, etc, and you might have forgotten about this change you made. Or it could be a new developer that got hired on to the project.

The point is, your claim that it "doesn't matter" to your application is probably overly strong. My personally recommendation is that you do this on a case-by-case scenario, either on the server or on the client.

On the server, you can use the serializer pattern (something like Active Model Serializers or jsonapi-resources) to transform your data before encoding them. If you go with those, you can easily customize the type for each field based on your domain knowledge, and/or use the usual OO/Ruby/meta-programming techniques to make things more concise. Alternatively, you can also used view-based libraries like jbuilder, which you can write custom helpers for the same purpose.

On the client, you would need to use whatever transformation hooks your framework provides, which is out-of-scope for the discussion here.

Two extra data-points:

  1. DynamoDB itself also uses strings as the wire-format for its numeric type: http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeValue.html
  2. The standard library JSON gem (which Rails uses as the encoding backend) also uses strings to encode BigDecimals out of the box

Arguably, it might be more "unexpected and buggy" to not encode them as strings.


@kamen-hursev the gem is scheduled to be EOL'ed with Rails 5. However, I believe that it should be fairly easy to support it for another minor release or so. If, after reading this response, you still think you have a good reason to go with that route, I can investigate extending the support for that gem.

@kamen-hursev

This comment has been minimized.

Show comment
Hide comment
@kamen-hursev

kamen-hursev Aug 8, 2016

@chancancode, no need to extend support for that gem. I understand your motivation and it is fine for me to take the responsibility and override (for now) BigDecimal#as_json.

kamen-hursev commented Aug 8, 2016

@chancancode, no need to extend support for that gem. I understand your motivation and it is fine for me to take the responsibility and override (for now) BigDecimal#as_json.

@pynixwang

This comment has been minimized.

Show comment
Hide comment
@pynixwang

pynixwang Aug 22, 2017

today, I do some bitcoin system and I need a big decimal scale, I think rails may cast my decimal to float when param deserialization, then I debug in controller, but rails gives me a string repr.

pynixwang commented Aug 22, 2017

today, I do some bitcoin system and I need a big decimal scale, I think rails may cast my decimal to float when param deserialization, then I debug in controller, but rails gives me a string repr.

@frankoid

This comment has been minimized.

Show comment
Hide comment
@frankoid

frankoid Mar 26, 2018

I'm developing JSON APIs using Scala + Play Framework and found this issue whilst Googling to find out about how other frameworks handle raw (i.e. not string) numbers in JSON.

@chancancode wrote:

in practice all clients would parse that into a float/double data type

This is not actually true of Play Framework's JSON support - when reading JSON into a BigDecimal field Play does not parse via a float/double, it parses the JSON directly into a BigDecimal so there is no loss of precision.

Just thought it might be useful for you to know that there are some clients out there that can parse the non-string version losslessly. I'm not making any suggestion about what Rails should do just passing on this info :)

frankoid commented Mar 26, 2018

I'm developing JSON APIs using Scala + Play Framework and found this issue whilst Googling to find out about how other frameworks handle raw (i.e. not string) numbers in JSON.

@chancancode wrote:

in practice all clients would parse that into a float/double data type

This is not actually true of Play Framework's JSON support - when reading JSON into a BigDecimal field Play does not parse via a float/double, it parses the JSON directly into a BigDecimal so there is no loss of precision.

Just thought it might be useful for you to know that there are some clients out there that can parse the non-string version losslessly. I'm not making any suggestion about what Rails should do just passing on this info :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment