Skip to content
This repository

AR supporting new JSON data type on PostgreSQL >= 9.2 #7527

Merged
merged 1 commit into from over 1 year ago

5 participants

Dickson S. Guedes Robin Dupret Carlos Antonio da Silva Rafael Mendonça França Steve Klabnik
Dickson S. Guedes

Hello all!

The next PostgreSQL version (9.2) will supports a native JSON type. Once the 9.2 version will be release soon I think that would be nice if AR supports it too.

Before started this, I searched for someone that could be working on this already, and tweeted @tenderlove asking if he remembers about someone doing this job, since seems that nobody is working on this I'm sending this pull request and I'd like to know your opinions about this feature and about my implementation. I marked two "FIXMEs" that could be a DRY candidate, IMO. Maybe should AR:Store be changed too?

Thanks.

Robin Dupret robin850 commented on the diff September 05, 2012
activerecord/test/cases/adapters/postgresql/json_test.rb
((12 lines not shown))
  12
+  def setup
  13
+    @connection = ActiveRecord::Base.connection
  14
+    begin
  15
+      @connection.transaction do
  16
+        @connection.create_table('json_data_type') do |t|
  17
+          t.json 'payload', :default => {}
  18
+        end
  19
+      end
  20
+    rescue ActiveRecord::StatementInvalid
  21
+      return skip "do not test on PG without json"
  22
+    end
  23
+    @column = JsonDataType.columns.find { |c| c.name == 'payload' }
  24
+  end
  25
+
  26
+  def teardown
  27
+    @connection.execute 'drop table if exists json_data_type'
3
Robin Dupret Collaborator

Do you mean this instead ?

@connection.execute 'drop table' if exists json_data_type
Dickson S. Guedes
guedes added a note September 05, 2012

Hi!

No, I'm not. json_data_type is the name I'm using for the test table. See line 16 above.

Thanks for your review.

Robin Dupret Collaborator

Sorry, I thought. ^^ BTW, thanks for this pull request, awesome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Robin Dupret robin850 commented on the diff September 05, 2012
activerecord/test/schema/postgresql_specific_schema.rb
... ...
@@ -82,6 +82,15 @@
82 82
 _SQL
83 83
   end
84 84
 
  85
+  if 't' == select_value("select 'json'=ANY(select typname from pg_type)")
3
Robin Dupret Collaborator

Do you mean typename ? ^^

Dickson S. Guedes
guedes added a note September 05, 2012

No, that statement is correct it must be typname because this is the column name from pg_type that gives me the name of type.

Thanks for your review!

Robin Dupret Collaborator

Okay, sorry once again. ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Robin Dupret
Collaborator

I'm :+1: for this feature ! :-)

activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
((5 lines not shown))
  84
+          if Hash === object
  85
+            JSON.generate(object)
  86
+          else
  87
+            object
  88
+          end
  89
+        end
  90
+
  91
+        def string_to_json(string)
  92
+          if string.nil?
  93
+            nil
  94
+          elsif String === string
  95
+            JSON.parse(string)
  96
+          else
  97
+            string
  98
+          end
  99
+        end
2

Shouldn't it use ActiveSupport::JSON?

Dickson S. Guedes
guedes added a note September 05, 2012

Yes, I'm changing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
... ...
@@ -80,6 +80,24 @@ def string_to_hstore(string)
80 80
           end
81 81
         end
82 82
 
  83
+        def json_to_string(object)
  84
+          if Hash === object
  85
+            JSON.generate(object)
  86
+          else
  87
+            object
  88
+          end
  89
+        end
  90
+
  91
+        def string_to_json(string)
  92
+          if string.nil?
  93
+            nil
2

I don't think you need to check for nil? here, since it's not a String, it'll return whatever string is, even if it's nil.

Dickson S. Guedes
guedes added a note September 05, 2012

Agreed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
... ...
@@ -600,8 +631,14 @@ def type_cast(value, column)
600 631
           return super unless 'bytea' == column.sql_type
601 632
           { :value => value, :format => 1 }
602 633
         when Hash
603  
-          return super unless 'hstore' == column.sql_type
604  
-          PostgreSQLColumn.hstore_to_string(value)
  634
+          # FIXME: refactor this?
  635
+          if 'hstore' == column.sql_type
  636
+            PostgreSQLColumn.hstore_to_string(value)
  637
+          elsif 'json' == column.sql_type
  638
+            PostgreSQLColumn.json_to_string(value)
  639
+          else
  640
+            return super
  641
+          end
1

I think a case statement, pretty much as the other in line 586 that you changed, would come in handy here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Carlos Antonio da Silva

Looks good, I've made a few comments. Thanks!

/cc @rafaelfranca @tenderlove

Carlos Antonio da Silva

And apparently after 9e0a14f you'll have to rebase and move some code around :)

Rafael Mendonça França

Oopss! I didn't see this pull request. Sorry

Dickson S. Guedes

@carlosantoniodasilva Thank you for your suggestions, I rebased from master and did the changes squashing my commits. I ran the tests against postgres 9.1 and 9.2RC1 and everything passed.

/cc @rafaelfranca

activerecord/test/cases/adapters/postgresql/json_test.rb
((4 lines not shown))
  4
+require 'active_record/base'
  5
+require 'active_record/connection_adapters/postgresql_adapter'
  6
+
  7
+class PostgresqlJSONTest < ActiveRecord::TestCase
  8
+  class JsonDataType < ActiveRecord::Base
  9
+    self.table_name = 'json_data_type'
  10
+  end
  11
+
  12
+  def setup
  13
+    @connection = ActiveRecord::Base.connection
  14
+    begin
  15
+      @connection.transaction do
  16
+        @connection.create_table('json_data_type') do |t|
  17
+          t.json 'payload', :default => {}
  18
+        end
  19
+      end
2

Do you need the transaction block to create only one table?

Dickson S. Guedes
guedes added a note September 05, 2012

No, I don't. I'll remove it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
activerecord/lib/active_record/connection_adapters/postgresql/quoting.rb
... ...
@@ -66,8 +67,12 @@ def type_cast(value, column)
66 67
             return super unless 'bytea' == column.sql_type
67 68
             { :value => value, :format => 1 }
68 69
           when Hash
69  
-            return super unless 'hstore' == column.sql_type
70  
-            PostgreSQLColumn.hstore_to_string(value)
  70
+            case column.sql_type
  71
+            when 'hstore' then PostgreSQLColumn.hstore_to_string(value)
  72
+            when 'json' then PostgreSQLColumn.json_to_string(value)
  73
+            else
  74
+                super
2

I know this is nitpicking, but would you mind using else super like the other one above? :)

Dickson S. Guedes
guedes added a note September 05, 2012

Sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Carlos Antonio da Silva

@guedes everything looks great! I just added another question, and I have to ask you to add a changelog entry to Active Record with the new json type for PostgreSQL.

Just ping us after that, and we'll merge. Thanks!

Dickson S. Guedes

Thanks for suggestions!

Dickson S. Guedes

@carlosantoniodasilva and @rafaelfranca : I fixed the code following your suggestions. Thank you for you time on this!

Rafael Mendonça França

Great! Should you squash the commits?

Dickson S. Guedes

@rafaelfranca I squashed the commits and changed the commit message too. Tests still passing.

Thanks!

Steve Klabnik
Collaborator

This is somehow out of date. CHANGELOGs! /me shakes his fist.

Rafael Mendonça França

Yes. Please rebase it.

Dickson S. Guedes ActiveRecord support to PostgreSQL 9.2 JSON type
This implements the support to encode/decode JSON
data to/from database and creating columns of type
JSON using a native type [1] supported by PostgreSQL
from version 9.2.

[1] http://www.postgresql.org/docs/9.2/static/datatype-json.html
3b516b5
Dickson S. Guedes

@rafaelfranca Sorry, I hope this is OK now. Thanks @steveklabnik to point me that.

Rafael Mendonça França rafaelfranca merged commit a690935 into from September 05, 2012
Rafael Mendonça França rafaelfranca closed this September 05, 2012
Dickson S. Guedes

Thanks for accepting this!

João Gradim jgradim referenced this pull request from a commit September 18, 2012
Commit has since been removed from the repository and is no longer available.
Vaughan Rouesnel vjpr referenced this pull request in tgriesser/knex June 17, 2013
Merged

JSON datatype support for Postgres #20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Sep 05, 2012
Dickson S. Guedes ActiveRecord support to PostgreSQL 9.2 JSON type
This implements the support to encode/decode JSON
data to/from database and creating columns of type
JSON using a native type [1] supported by PostgreSQL
from version 9.2.

[1] http://www.postgresql.org/docs/9.2/static/datatype-json.html
3b516b5
This page is out of date. Refresh to see the latest.
5  activerecord/CHANGELOG.md
Source Rendered
@@ -14,6 +14,11 @@
14 14
 
15 15
     *Ian Lesperance*
16 16
 
  17
+*   Allow JSON columns to be created in PostgreSQL and properly encoded/decoded
  18
+    to/from database.
  19
+
  20
+    *Dickson S. Guedes*
  21
+
17 22
 *   Fix time column type casting for invalid time string values to correctly return nil.
18 23
 
19 24
     *Adam Meehan*
1  activerecord/lib/active_record/connection_adapters/column.rb
@@ -124,6 +124,7 @@ def type_cast_code(var_name)
124 124
         when :boolean              then "#{klass}.value_to_boolean(#{var_name})"
125 125
         when :hstore               then "#{klass}.string_to_hstore(#{var_name})"
126 126
         when :inet, :cidr          then "#{klass}.string_to_cidr(#{var_name})"
  127
+        when :json                 then "#{klass}.string_to_json(#{var_name})"
127 128
         else var_name
128 129
         end
129 130
       end
16  activerecord/lib/active_record/connection_adapters/postgresql/cast.rb
@@ -37,6 +37,22 @@ def string_to_hstore(string)
37 37
           end
38 38
         end
39 39
 
  40
+        def json_to_string(object)
  41
+          if Hash === object
  42
+            ActiveSupport::JSON.encode(object)
  43
+          else
  44
+            object
  45
+          end
  46
+        end
  47
+
  48
+        def string_to_json(string)
  49
+          if String === string
  50
+            ActiveSupport::JSON.decode(string)
  51
+          else
  52
+            string
  53
+          end
  54
+        end
  55
+
40 56
         def string_to_cidr(string)
41 57
           if string.nil?
42 58
             nil
9  activerecord/lib/active_record/connection_adapters/postgresql/oid.rb
@@ -145,6 +145,14 @@ def type_cast(value)
145 145
           end
146 146
         end
147 147
 
  148
+        class Json < Type
  149
+          def type_cast(value)
  150
+            return if value.nil?
  151
+
  152
+            ConnectionAdapters::PostgreSQLColumn.string_to_json value
  153
+          end
  154
+        end
  155
+
148 156
         class TypeMap
149 157
           def initialize
150 158
             @mapping = {}
@@ -244,6 +252,7 @@ def self.registered_type?(name)
244 252
         register_type 'polygon', OID::Identity.new
245 253
         register_type 'circle', OID::Identity.new
246 254
         register_type 'hstore', OID::Hstore.new
  255
+        register_type 'json', OID::Json.new
247 256
 
248 257
         register_type 'cidr', OID::Cidr.new
249 258
         alias_type 'inet', 'cidr'
8  activerecord/lib/active_record/connection_adapters/postgresql/quoting.rb
@@ -22,6 +22,7 @@ def quote(value, column = nil) #:nodoc:
22 22
           when Hash
23 23
             case column.sql_type
24 24
             when 'hstore' then super(PostgreSQLColumn.hstore_to_string(value), column)
  25
+            when 'json' then super(PostgreSQLColumn.json_to_string(value), column)
25 26
             else super
26 27
             end
27 28
           when IPAddr
@@ -66,8 +67,11 @@ def type_cast(value, column)
66 67
             return super unless 'bytea' == column.sql_type
67 68
             { :value => value, :format => 1 }
68 69
           when Hash
69  
-            return super unless 'hstore' == column.sql_type
70  
-            PostgreSQLColumn.hstore_to_string(value)
  70
+            case column.sql_type
  71
+            when 'hstore' then PostgreSQLColumn.hstore_to_string(value)
  72
+            when 'json' then PostgreSQLColumn.json_to_string(value)
  73
+            else super
  74
+            end
71 75
           when IPAddr
72 76
             return super unless ['inet','cidr'].includes? column.sql_type
73 77
             PostgreSQLColumn.cidr_to_string(value)
13  activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
@@ -106,6 +106,9 @@ def self.extract_value_from_default(default)
106 106
           # Hstore
107 107
           when /\A'(.*)'::hstore\z/
108 108
             $1
  109
+          # JSON
  110
+          when /\A'(.*)'::json\z/
  111
+            $1
109 112
           # Object identifier types
110 113
           when /\A-?\d+\z/
111 114
             $1
@@ -201,6 +204,9 @@ def simplified_type(field_type)
201 204
           # UUID type
202 205
           when 'uuid'
203 206
             :uuid
  207
+        # JSON type
  208
+        when 'json'
  209
+          :json
204 210
           # Small and big integer types
205 211
           when /^(?:small|big)int$/
206 212
             :integer
@@ -267,6 +273,10 @@ def macaddr(name, options = {})
267 273
         def uuid(name, options = {})
268 274
           column(name, 'uuid', options)
269 275
         end
  276
+
  277
+        def json(name, options = {})
  278
+          column(name, 'json', options)
  279
+        end
270 280
       end
271 281
 
272 282
       ADAPTER_NAME = 'PostgreSQL'
@@ -290,7 +300,8 @@ def uuid(name, options = {})
290 300
         inet:        { name: "inet" },
291 301
         cidr:        { name: "cidr" },
292 302
         macaddr:     { name: "macaddr" },
293  
-        uuid:        { name: "uuid" }
  303
+        uuid:        { name: "uuid" },
  304
+        json:        { name: "json" }
294 305
       }
295 306
 
296 307
       include Quoting
69  activerecord/test/cases/adapters/postgresql/json_test.rb
... ...
@@ -0,0 +1,69 @@
  1
+# encoding: utf-8
  2
+
  3
+require "cases/helper"
  4
+require 'active_record/base'
  5
+require 'active_record/connection_adapters/postgresql_adapter'
  6
+
  7
+class PostgresqlJSONTest < ActiveRecord::TestCase
  8
+  class JsonDataType < ActiveRecord::Base
  9
+    self.table_name = 'json_data_type'
  10
+  end
  11
+
  12
+  def setup
  13
+    @connection = ActiveRecord::Base.connection
  14
+    begin
  15
+      @connection.create_table('json_data_type') do |t|
  16
+        t.json 'payload', :default => {}
  17
+      end
  18
+    rescue ActiveRecord::StatementInvalid
  19
+      return skip "do not test on PG without json"
  20
+    end
  21
+    @column = JsonDataType.columns.find { |c| c.name == 'payload' }
  22
+  end
  23
+
  24
+  def teardown
  25
+    @connection.execute 'drop table if exists json_data_type'
  26
+  end
  27
+
  28
+  def test_column
  29
+    assert_equal :json, @column.type
  30
+  end
  31
+
  32
+  def test_type_cast_json
  33
+    assert @column
  34
+
  35
+    data = "{\"a_key\":\"a_value\"}"
  36
+    hash = @column.class.string_to_json data
  37
+    assert_equal({'a_key' => 'a_value'}, hash)
  38
+    assert_equal({'a_key' => 'a_value'}, @column.type_cast(data))
  39
+
  40
+    assert_equal({}, @column.type_cast("{}"))
  41
+    assert_equal({'key'=>nil}, @column.type_cast('{"key": null}'))
  42
+    assert_equal({'c'=>'}','"a"'=>'b "a b'}, @column.type_cast(%q({"c":"}", "\"a\"":"b \"a b"})))
  43
+  end
  44
+
  45
+  def test_rewrite
  46
+    @connection.execute "insert into json_data_type (payload) VALUES ('{\"k\":\"v\"}')"
  47
+    x = JsonDataType.first
  48
+    x.payload = { '"a\'' => 'b' }
  49
+    assert x.save!
  50
+  end
  51
+
  52
+  def test_select
  53
+    @connection.execute "insert into json_data_type (payload) VALUES ('{\"k\":\"v\"}')"
  54
+    x = JsonDataType.first
  55
+    assert_equal({'k' => 'v'}, x.payload)
  56
+  end
  57
+
  58
+  def test_select_multikey
  59
+    @connection.execute %q|insert into json_data_type (payload) VALUES ('{"k1":"v1", "k2":"v2", "k3":[1,2,3]}')|
  60
+    x = JsonDataType.first
  61
+    assert_equal({'k1' => 'v1', 'k2' => 'v2', 'k3' => [1,2,3]}, x.payload)
  62
+  end
  63
+
  64
+  def test_null_json
  65
+    @connection.execute %q|insert into json_data_type (payload) VALUES(null)|
  66
+    x = JsonDataType.first
  67
+    assert_equal(nil, x.payload)
  68
+  end
  69
+end
7  activerecord/test/cases/schema_dumper_test.rb
@@ -236,6 +236,13 @@ def test_schema_dump_includes_xml_shorthand_definition
236 236
       end
237 237
     end
238 238
 
  239
+    def test_schema_dump_includes_json_shorthand_definition
  240
+      output = standard_dump
  241
+      if %r{create_table "postgresql_json_data_type"} =~ output
  242
+        assert_match %r|t.json "json_data", :default => {}|, output
  243
+      end
  244
+    end
  245
+
239 246
     def test_schema_dump_includes_inet_shorthand_definition
240 247
       output = standard_dump
241 248
       if %r{create_table "postgresql_network_address"} =~ output
11  activerecord/test/schema/postgresql_specific_schema.rb
... ...
@@ -1,7 +1,7 @@
1 1
 ActiveRecord::Schema.define do
2 2
 
3 3
   %w(postgresql_tsvectors postgresql_hstores postgresql_arrays postgresql_moneys postgresql_numbers postgresql_times postgresql_network_addresses postgresql_bit_strings postgresql_uuids
4  
-      postgresql_oids postgresql_xml_data_type defaults geometrics postgresql_timestamp_with_zones postgresql_partitioned_table postgresql_partitioned_table_parent).each do |table_name|
  4
+      postgresql_oids postgresql_xml_data_type defaults geometrics postgresql_timestamp_with_zones postgresql_partitioned_table postgresql_partitioned_table_parent postgresql_json_data_type).each do |table_name|
5 5
     execute "DROP TABLE IF EXISTS #{quote_table_name table_name}"
6 6
   end
7 7
 
@@ -82,6 +82,15 @@
82 82
 _SQL
83 83
   end
84 84
 
  85
+  if 't' == select_value("select 'json'=ANY(select typname from pg_type)")
  86
+  execute <<_SQL
  87
+  CREATE TABLE postgresql_json_data_type (
  88
+    id SERIAL PRIMARY KEY,
  89
+    json_data json default '{}'::json
  90
+  );
  91
+_SQL
  92
+  end
  93
+
85 94
   execute <<_SQL
86 95
   CREATE TABLE postgresql_moneys (
87 96
     id SERIAL PRIMARY KEY,
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.