Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

AR supporting new JSON data type on PostgreSQL >= 9.2 #7527

Merged
merged 1 commit into from

5 participants

@guedes

Hello all!

The next PostgreSQL version (9.2) will supports a native JSON type. Once the 9.2 version will be release soon I think that would be nice if AR supports it too.

Before started this, I searched for someone that could be working on this already, and tweeted @tenderlove asking if he remembers about someone doing this job, since seems that nobody is working on this I'm sending this pull request and I'd like to know your opinions about this feature and about my implementation. I marked two "FIXMEs" that could be a DRY candidate, IMO. Maybe should AR:Store be changed too?

Thanks.

@robin850 robin850 commented on the diff
activerecord/test/cases/adapters/postgresql/json_test.rb
((12 lines not shown))
+ def setup
+ @connection = ActiveRecord::Base.connection
+ begin
+ @connection.transaction do
+ @connection.create_table('json_data_type') do |t|
+ t.json 'payload', :default => {}
+ end
+ end
+ rescue ActiveRecord::StatementInvalid
+ return skip "do not test on PG without json"
+ end
+ @column = JsonDataType.columns.find { |c| c.name == 'payload' }
+ end
+
+ def teardown
+ @connection.execute 'drop table if exists json_data_type'
@robin850 Collaborator
robin850 added a note

Do you mean this instead ?

@connection.execute 'drop table' if exists json_data_type
@guedes
guedes added a note

Hi!

No, I'm not. json_data_type is the name I'm using for the test table. See line 16 above.

Thanks for your review.

@robin850 Collaborator
robin850 added a note

Sorry, I thought. ^^ BTW, thanks for this pull request, awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@robin850 robin850 commented on the diff
activerecord/test/schema/postgresql_specific_schema.rb
@@ -82,6 +82,15 @@
_SQL
end
+ if 't' == select_value("select 'json'=ANY(select typname from pg_type)")
@robin850 Collaborator
robin850 added a note

Do you mean typename ? ^^

@guedes
guedes added a note

No, that statement is correct it must be typname because this is the column name from pg_type that gives me the name of type.

Thanks for your review!

@robin850 Collaborator
robin850 added a note

Okay, sorry once again. ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@robin850
Collaborator

I'm :+1: for this feature ! :-)

...tive_record/connection_adapters/postgresql_adapter.rb
((5 lines not shown))
+ if Hash === object
+ JSON.generate(object)
+ else
+ object
+ end
+ end
+
+ def string_to_json(string)
+ if string.nil?
+ nil
+ elsif String === string
+ JSON.parse(string)
+ else
+ string
+ end
+ end

Shouldn't it use ActiveSupport::JSON?

@guedes
guedes added a note

Yes, I'm changing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...tive_record/connection_adapters/postgresql_adapter.rb
@@ -80,6 +80,24 @@ def string_to_hstore(string)
end
end
+ def json_to_string(object)
+ if Hash === object
+ JSON.generate(object)
+ else
+ object
+ end
+ end
+
+ def string_to_json(string)
+ if string.nil?
+ nil

I don't think you need to check for nil? here, since it's not a String, it'll return whatever string is, even if it's nil.

@guedes
guedes added a note

Agreed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...tive_record/connection_adapters/postgresql_adapter.rb
@@ -600,8 +631,14 @@ def type_cast(value, column)
return super unless 'bytea' == column.sql_type
{ :value => value, :format => 1 }
when Hash
- return super unless 'hstore' == column.sql_type
- PostgreSQLColumn.hstore_to_string(value)
+ # FIXME: refactor this?
+ if 'hstore' == column.sql_type
+ PostgreSQLColumn.hstore_to_string(value)
+ elsif 'json' == column.sql_type
+ PostgreSQLColumn.json_to_string(value)
+ else
+ return super
+ end

I think a case statement, pretty much as the other in line 586 that you changed, would come in handy here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carlosantoniodasilva

Looks good, I've made a few comments. Thanks!

/cc @rafaelfranca @tenderlove

@carlosantoniodasilva

And apparently after 9e0a14f you'll have to rebase and move some code around :)

@rafaelfranca
Owner

Oopss! I didn't see this pull request. Sorry

@guedes

@carlosantoniodasilva Thank you for your suggestions, I rebased from master and did the changes squashing my commits. I ran the tests against postgres 9.1 and 9.2RC1 and everything passed.

/cc @rafaelfranca

activerecord/test/cases/adapters/postgresql/json_test.rb
((4 lines not shown))
+require 'active_record/base'
+require 'active_record/connection_adapters/postgresql_adapter'
+
+class PostgresqlJSONTest < ActiveRecord::TestCase
+ class JsonDataType < ActiveRecord::Base
+ self.table_name = 'json_data_type'
+ end
+
+ def setup
+ @connection = ActiveRecord::Base.connection
+ begin
+ @connection.transaction do
+ @connection.create_table('json_data_type') do |t|
+ t.json 'payload', :default => {}
+ end
+ end

Do you need the transaction block to create only one table?

@guedes
guedes added a note

No, I don't. I'll remove it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...tive_record/connection_adapters/postgresql/quoting.rb
@@ -66,8 +67,12 @@ def type_cast(value, column)
return super unless 'bytea' == column.sql_type
{ :value => value, :format => 1 }
when Hash
- return super unless 'hstore' == column.sql_type
- PostgreSQLColumn.hstore_to_string(value)
+ case column.sql_type
+ when 'hstore' then PostgreSQLColumn.hstore_to_string(value)
+ when 'json' then PostgreSQLColumn.json_to_string(value)
+ else
+ super

I know this is nitpicking, but would you mind using else super like the other one above? :)

@guedes
guedes added a note

Sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@carlosantoniodasilva

@guedes everything looks great! I just added another question, and I have to ask you to add a changelog entry to Active Record with the new json type for PostgreSQL.

Just ping us after that, and we'll merge. Thanks!

@guedes

Thanks for suggestions!

@guedes

@carlosantoniodasilva and @rafaelfranca : I fixed the code following your suggestions. Thank you for you time on this!

@rafaelfranca
Owner

Great! Should you squash the commits?

@guedes

@rafaelfranca I squashed the commits and changed the commit message too. Tests still passing.

Thanks!

@steveklabnik
Collaborator

This is somehow out of date. CHANGELOGs! /me shakes his fist.

@rafaelfranca
Owner

Yes. Please rebase it.

@guedes guedes ActiveRecord support to PostgreSQL 9.2 JSON type
This implements the support to encode/decode JSON
data to/from database and creating columns of type
JSON using a native type [1] supported by PostgreSQL
from version 9.2.

[1] http://www.postgresql.org/docs/9.2/static/datatype-json.html
3b516b5
@guedes

@rafaelfranca Sorry, I hope this is OK now. Thanks @steveklabnik to point me that.

@rafaelfranca rafaelfranca merged commit a690935 into from
@guedes

Thanks for accepting this!

@jgradim jgradim referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
@vjpr vjpr referenced this pull request in tgriesser/knex
Merged

JSON datatype support for Postgres #20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 6, 2012
  1. @guedes

    ActiveRecord support to PostgreSQL 9.2 JSON type

    guedes authored
    This implements the support to encode/decode JSON
    data to/from database and creating columns of type
    JSON using a native type [1] supported by PostgreSQL
    from version 9.2.
    
    [1] http://www.postgresql.org/docs/9.2/static/datatype-json.html
This page is out of date. Refresh to see the latest.
View
5 activerecord/CHANGELOG.md
@@ -14,6 +14,11 @@
*Ian Lesperance*
+* Allow JSON columns to be created in PostgreSQL and properly encoded/decoded
+ to/from database.
+
+ *Dickson S. Guedes*
+
* Fix time column type casting for invalid time string values to correctly return nil.
*Adam Meehan*
View
1  activerecord/lib/active_record/connection_adapters/column.rb
@@ -124,6 +124,7 @@ def type_cast_code(var_name)
when :boolean then "#{klass}.value_to_boolean(#{var_name})"
when :hstore then "#{klass}.string_to_hstore(#{var_name})"
when :inet, :cidr then "#{klass}.string_to_cidr(#{var_name})"
+ when :json then "#{klass}.string_to_json(#{var_name})"
else var_name
end
end
View
16 activerecord/lib/active_record/connection_adapters/postgresql/cast.rb
@@ -37,6 +37,22 @@ def string_to_hstore(string)
end
end
+ def json_to_string(object)
+ if Hash === object
+ ActiveSupport::JSON.encode(object)
+ else
+ object
+ end
+ end
+
+ def string_to_json(string)
+ if String === string
+ ActiveSupport::JSON.decode(string)
+ else
+ string
+ end
+ end
+
def string_to_cidr(string)
if string.nil?
nil
View
9 activerecord/lib/active_record/connection_adapters/postgresql/oid.rb
@@ -145,6 +145,14 @@ def type_cast(value)
end
end
+ class Json < Type
+ def type_cast(value)
+ return if value.nil?
+
+ ConnectionAdapters::PostgreSQLColumn.string_to_json value
+ end
+ end
+
class TypeMap
def initialize
@mapping = {}
@@ -244,6 +252,7 @@ def self.registered_type?(name)
register_type 'polygon', OID::Identity.new
register_type 'circle', OID::Identity.new
register_type 'hstore', OID::Hstore.new
+ register_type 'json', OID::Json.new
register_type 'cidr', OID::Cidr.new
alias_type 'inet', 'cidr'
View
8 activerecord/lib/active_record/connection_adapters/postgresql/quoting.rb
@@ -22,6 +22,7 @@ def quote(value, column = nil) #:nodoc:
when Hash
case column.sql_type
when 'hstore' then super(PostgreSQLColumn.hstore_to_string(value), column)
+ when 'json' then super(PostgreSQLColumn.json_to_string(value), column)
else super
end
when IPAddr
@@ -66,8 +67,11 @@ def type_cast(value, column)
return super unless 'bytea' == column.sql_type
{ :value => value, :format => 1 }
when Hash
- return super unless 'hstore' == column.sql_type
- PostgreSQLColumn.hstore_to_string(value)
+ case column.sql_type
+ when 'hstore' then PostgreSQLColumn.hstore_to_string(value)
+ when 'json' then PostgreSQLColumn.json_to_string(value)
+ else super
+ end
when IPAddr
return super unless ['inet','cidr'].includes? column.sql_type
PostgreSQLColumn.cidr_to_string(value)
View
13 activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
@@ -106,6 +106,9 @@ def self.extract_value_from_default(default)
# Hstore
when /\A'(.*)'::hstore\z/
$1
+ # JSON
+ when /\A'(.*)'::json\z/
+ $1
# Object identifier types
when /\A-?\d+\z/
$1
@@ -201,6 +204,9 @@ def simplified_type(field_type)
# UUID type
when 'uuid'
:uuid
+ # JSON type
+ when 'json'
+ :json
# Small and big integer types
when /^(?:small|big)int$/
:integer
@@ -267,6 +273,10 @@ def macaddr(name, options = {})
def uuid(name, options = {})
column(name, 'uuid', options)
end
+
+ def json(name, options = {})
+ column(name, 'json', options)
+ end
end
ADAPTER_NAME = 'PostgreSQL'
@@ -290,7 +300,8 @@ def uuid(name, options = {})
inet: { name: "inet" },
cidr: { name: "cidr" },
macaddr: { name: "macaddr" },
- uuid: { name: "uuid" }
+ uuid: { name: "uuid" },
+ json: { name: "json" }
}
include Quoting
View
69 activerecord/test/cases/adapters/postgresql/json_test.rb
@@ -0,0 +1,69 @@
+# encoding: utf-8
+
+require "cases/helper"
+require 'active_record/base'
+require 'active_record/connection_adapters/postgresql_adapter'
+
+class PostgresqlJSONTest < ActiveRecord::TestCase
+ class JsonDataType < ActiveRecord::Base
+ self.table_name = 'json_data_type'
+ end
+
+ def setup
+ @connection = ActiveRecord::Base.connection
+ begin
+ @connection.create_table('json_data_type') do |t|
+ t.json 'payload', :default => {}
+ end
+ rescue ActiveRecord::StatementInvalid
+ return skip "do not test on PG without json"
+ end
+ @column = JsonDataType.columns.find { |c| c.name == 'payload' }
+ end
+
+ def teardown
+ @connection.execute 'drop table if exists json_data_type'
@robin850 Collaborator
robin850 added a note

Do you mean this instead ?

@connection.execute 'drop table' if exists json_data_type
@guedes
guedes added a note

Hi!

No, I'm not. json_data_type is the name I'm using for the test table. See line 16 above.

Thanks for your review.

@robin850 Collaborator
robin850 added a note

Sorry, I thought. ^^ BTW, thanks for this pull request, awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ end
+
+ def test_column
+ assert_equal :json, @column.type
+ end
+
+ def test_type_cast_json
+ assert @column
+
+ data = "{\"a_key\":\"a_value\"}"
+ hash = @column.class.string_to_json data
+ assert_equal({'a_key' => 'a_value'}, hash)
+ assert_equal({'a_key' => 'a_value'}, @column.type_cast(data))
+
+ assert_equal({}, @column.type_cast("{}"))
+ assert_equal({'key'=>nil}, @column.type_cast('{"key": null}'))
+ assert_equal({'c'=>'}','"a"'=>'b "a b'}, @column.type_cast(%q({"c":"}", "\"a\"":"b \"a b"})))
+ end
+
+ def test_rewrite
+ @connection.execute "insert into json_data_type (payload) VALUES ('{\"k\":\"v\"}')"
+ x = JsonDataType.first
+ x.payload = { '"a\'' => 'b' }
+ assert x.save!
+ end
+
+ def test_select
+ @connection.execute "insert into json_data_type (payload) VALUES ('{\"k\":\"v\"}')"
+ x = JsonDataType.first
+ assert_equal({'k' => 'v'}, x.payload)
+ end
+
+ def test_select_multikey
+ @connection.execute %q|insert into json_data_type (payload) VALUES ('{"k1":"v1", "k2":"v2", "k3":[1,2,3]}')|
+ x = JsonDataType.first
+ assert_equal({'k1' => 'v1', 'k2' => 'v2', 'k3' => [1,2,3]}, x.payload)
+ end
+
+ def test_null_json
+ @connection.execute %q|insert into json_data_type (payload) VALUES(null)|
+ x = JsonDataType.first
+ assert_equal(nil, x.payload)
+ end
+end
View
7 activerecord/test/cases/schema_dumper_test.rb
@@ -236,6 +236,13 @@ def test_schema_dump_includes_xml_shorthand_definition
end
end
+ def test_schema_dump_includes_json_shorthand_definition
+ output = standard_dump
+ if %r{create_table "postgresql_json_data_type"} =~ output
+ assert_match %r|t.json "json_data", :default => {}|, output
+ end
+ end
+
def test_schema_dump_includes_inet_shorthand_definition
output = standard_dump
if %r{create_table "postgresql_network_address"} =~ output
View
11 activerecord/test/schema/postgresql_specific_schema.rb
@@ -1,7 +1,7 @@
ActiveRecord::Schema.define do
%w(postgresql_tsvectors postgresql_hstores postgresql_arrays postgresql_moneys postgresql_numbers postgresql_times postgresql_network_addresses postgresql_bit_strings postgresql_uuids
- postgresql_oids postgresql_xml_data_type defaults geometrics postgresql_timestamp_with_zones postgresql_partitioned_table postgresql_partitioned_table_parent).each do |table_name|
+ postgresql_oids postgresql_xml_data_type defaults geometrics postgresql_timestamp_with_zones postgresql_partitioned_table postgresql_partitioned_table_parent postgresql_json_data_type).each do |table_name|
execute "DROP TABLE IF EXISTS #{quote_table_name table_name}"
end
@@ -82,6 +82,15 @@
_SQL
end
+ if 't' == select_value("select 'json'=ANY(select typname from pg_type)")
@robin850 Collaborator
robin850 added a note

Do you mean typename ? ^^

@guedes
guedes added a note

No, that statement is correct it must be typname because this is the column name from pg_type that gives me the name of type.

Thanks for your review!

@robin850 Collaborator
robin850 added a note

Okay, sorry once again. ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ execute <<_SQL
+ CREATE TABLE postgresql_json_data_type (
+ id SERIAL PRIMARY KEY,
+ json_data json default '{}'::json
+ );
+_SQL
+ end
+
execute <<_SQL
CREATE TABLE postgresql_moneys (
id SERIAL PRIMARY KEY,
Something went wrong with that request. Please try again.