Assume payload is a table column of JSON type. A Ruby nil value of payload can be stored in the database in one of two ways
When loading data from the database both NULL and 'null' are deserialized correctly into Ruby nil. This part is fine.
When data are persisted into the DB nil is translated to 'null', not SQL NULL. This leads to several problems:
Although x.update_attributes(payload: nil) translates to SQL UPDATE products SET payload = 'null', a query Product.where(payload: nil) translates to SELECT * from products WHERE payload IS NULL. Such a SQL query returns nothing because 'null' IS NOT NULL. This is demonstrated by the failed test I added in this PR.
UPDATE products SET payload = 'null'
SELECT * from products WHERE payload IS NULL
'null' IS NOT NULL
One way to fix this bug is to translate where(payload: nil) to WHERE payload IS NULL OR payload = 'null'. We'd have to check if payload type is JSON first and such a SQL statement is slower than the simpler payload IS NOT NULL.
WHERE payload IS NULL OR payload = 'null'
payload IS NOT NULL
Support for JSON types in Postgresql in Rails 4.x translates nil to SQL NULL, not JSON 'null'. This change of behavior was introduced when AR added JSON support for MySQL (#21110). In our code base we have a few complicated SQL statements which rely on the old behavior. They are broken when we upgrade to Rails 5.0. Queries for nil, not nil are pretty common, I suspect we are not the only ones affected.
I think the preferred fix is to map nil to SQL NULL, exactly as how it used to work in Rails 4.x.
Thanks for the pull request, and welcome! The Rails team is excited to review your changes, and you should hear from @senny (or someone else) soon.
If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.
Please see the contribution instructions for more information.
We just ran into this bug in a newly upgraded Rails 5 application. Previously nil was coerced to SQL NULL, but now it's coerced to a JSON string of "null", causing issues when attempting to search for records that have no data in that column. Using this patch successfully changed all "null" strings to SQL NULL.
Would be great to get this fix included in 5.0.1!
This looks good. @sgrif may have some insights to share.
@tdtran can you squash your commits and add an entry to the changelog?
@senny changelog entry added, commits squashed, branch rebased against latest rails[master]
@tdtran can you rebase and I'll merge?
Serialize JSON attribute value nil as SQL NULL, not JSON 'null'
Test: JSON attribute value nil can be used in where(attr: nil)
Add changelog entry
Should this be backported to 5-0-stable and/or added to the 5.0.1 milestone?
@connorshea yes, I'll backport to 5-0-stable