JSON functions

[groue]

This pull request adds support for SQLite JSON Functions And Operators.

Added the full set of SQLite json functions

Database.json, Database.jsonExtract, Database.jsonArray, ... match the SQL functions json, json_extract, json_array, etc.

The only notable omission is the support for json_each and json_tree. These are not supported, as GRDB has not yet learned to handle table-valued functions.

The JSONColumn type and the SQLJSONExpressible protocol grant convenience methods such as access to JSON subcomponents

// Fetch the country of player 1, assuming the
// address column contains a JSON payload:
//
// SELECT address ->> 'country' FROM player WHERE id = 1
let address = JSONColumn("address")
let country = Player
    .filter(id: 42)
    .select(address["country"], as: String.self)
    .fetchOne(db)

Use JSON expressions when defining the database schema

// NEW: .jsonText column type
// It's actually identical to .text, but better describes the intent
try db.create(table: "player") { t in
    t.autoIncrementedPrimaryKey("id")
    t.column("name", .text).notNull()
    t.column("address", .jsonText).notNull()
}

// NEW: indexes on JSON expressions
try db.create(
    index: "player_on_country",
    on: "player",
    expressions: [
        JSONColumn("address")["country"],
        JSONColumn("address")["city"],
    ])

// NEW: generated columns based on JSON expressions
try db.create(table: "team") { t in
    t.autoIncrementedPrimaryKey("id")
    t.column("color", .text).notNull()
    t.column("schedule", .jsonText).notNull()
    t.column("nextGameDate", .datetime)
        .generatedAs(JSONColumn("schedule")["nextGameDate"])
        .indexed()
}

Codable record that want to manipulate JSON data as Foundation Data can save it as database strings

SQLite can only manipulate JSON when it is stored as text at the database level, and that's why it is necessary to convert Data JSON columns into text in the database:

// Use `Data` for flexible JSON schema
struct Player: Codable {
    var id: Int64
    var name: String
    var address: Data // JSON UTF8 data
}

extension Player: FetchableRecord, PersistableRecord {
    // Store JSON data as text, so that SQLite JSON
    // functions and operators work correctly:
    static let databaseDataEncodingStrategy = DatabaseDataEncodingStrategy.text
}

[groue]

@myyra, this PR is based on the foundations you have grounded in #1423. Above all, thank you!

I would be happy if you could review the user-facing, public, API of this PR, in order to check if it fits your needs.

Files of interest, documentation-wise, are:

• JSON.md
• JSONColumn.swift
• SQLJSONExpressible.swift
• SQLJSONFunctions.swift

New tests can look like some actual use cases, even if they sometimes are a little bit low-level:

• JSONExpressionsTests.swift
• JSONColumnTests.swift

In our previous conversations, we had a few problems to solve:

How to expose the SQLite JSON functions?

My solution is to expose all of them as static Database methods, with names directly inspired from their SQL origin (json_array -> Database.jsonArray):

// JSON('[1, 2, 3]')
Database.json("[1, 2, 3]")

This way:

• We do not pollute the global space of top-level functions
• We make sure people understand that those are JSON database functions.
• People who are looking for, say, json_patch immediately find Database.jsonPatch.

Which subset of JSON features should ship?

All (but json_each and json_tree) :-)

This also made it possible to write the JSON documentation tour, JSON.md, and patch necessary holes (such as the new DatabaseDataEncodingStrategy.text, which is not a pure JSON feature).

How to expose json_extract, -> and ->>?

As all SQLite JSON functions, json_extract is available as Database.jsonExtract.

json_extract, ->, and ->> are available on values of type SQLJSONExpressible, such as JSONColum("info"), or the equivalent Colum("info").asJSONValue.

I have favored ->> above all others with a subscript, because it is the only way to provide a plain JSON key instead of a $-prefixed path:

• The ->> SQL operator:

  let info = JSONColumn("info")
  
  // SELECT info ->> 'firstName' FROM player
  // → 'Arthur'
  let firstName = try Player
      .select(info["firstName"], as: String.self)
      .fetchOne(db)
  
  // SELECT info ->> 'address' FROM player
  // → '{"street":"Rue de Belleville","city":"Paris"}'
  let address = try Player
      .select(info["address"], as: String.self)
      .fetchOne(db)

• The -> SQL operator:

  let info = JSONColumn("info")
  
  // SELECT info -> 'firstName' FROM player
  // → '"Arthur"'
  let name = try Player
      .select(info.jsonRepresentation(forKey: "firstName"), as: String.self)
      .fetchOne(db)
  
  // SELECT info -> 'address' FROM player
  // → '{"street":"Rue de Belleville","city":"Paris"}'
  let name = try Player
      .select(info.jsonRepresentation(forKey: "address"), as: String.self)
      .fetchOne(db)

  The name of jsonRepresentation for -> might be discussed. It always returns a valid JSON string. In particular, extracted JSON string values are returned wrapped inside ".

• The JSON_EXTRACT SQL function:

  let info = JSONColumn("info")
  
  // SELECT JSON_EXTRACT(info, '$.firstName') FROM player
  // → 'Arthur'
  let firstName = try Player
      .select(info.jsonExtract(atPath: "$.firstName"), as: String.self)
      .fetchOne(db)
  
  // SELECT JSON_EXTRACT(info, '$.address') FROM player
  // → '{"street":"Rue de Belleville","city":"Paris"}'
  let address = try Player
      .select(info.jsonExtract(atPath: "$.address"), as: String.self)
      .fetchOne(db)

What is JSONColumn?

JSONColumn(columnName) is just sugar. All column-building techniques are supported:

struct Player: Codable {
    var address: Address

    enum Columns1 {
        static let address = JSONColumn(CodingKeys.address)
    }

    enum Columns2 {
        static let address = Column(CodingKeys.address)
    }

    enum Columns3: String, ColumnExpression {
        case address
    }
}

// 100% identical behavior
Player.Columns1.address
Player.Columns2.address.asJSONValue
Player.Columns3.address.asJSONValue
JSONColumn("address")
Column("address").asJSONValue

In this PR, JSONColumn and values returned by asJSONValue are Swift values that:

1. Provide access to JSON subcomponents:

   JSONColumn("address")["street"] // Compiles
   Column("address")["street"]     // Does not compile

2. Have no special behavior in SQL...

   // SELECT address, address, address FROM player
   let json = try Player
       .select(
           Column("address"),
           Column("address").asJSONValue,
           JSONColumn("address"))
       .fetchOne(db)

3. ... except in JSON building contexts.

   In the example below, we can see how the JSONColumn is interpreted as JSON, while the Column with the same name is interpreted as a plain string:

   let elements: [any SQLExpressible] = [
       JSONColumn("address"),
       Column("address"),
   ]
   
   let array = Database.jsonArray(elements)
   
   // SELECT JSON_ARRAY(JSON(address), address) FROM player
   // → '[{"country":"FR"},"{\"country\":\"FR\"}"]'
   //     <--- object ---> <------ string ------>
   let json = try Player
       .select(array, as: String.self)
       .fetchOne(db)

[myyra]

This looks awesome and solves so many more use cases than I had planned! I have a busier than usual week, so it'll likely take until the weekend before I can test this in my app and review it properly. Feel free to merge this earlier if you feel like it. Meanwhile, I had a quick read-through and added some comments below.

How to expose the SQLite JSON functions?

My solution is to expose all of them as static Database methods, with names directly inspired from their SQL origin (json_array -> Database.jsonArray):

// JSON('[1, 2, 3]')
Database.json("[1, 2, 3]")

This way:

• We do not pollute the global space of top-level functions
• We make sure people understand that those are JSON database functions.
• People who are looking for, say, json_patch immediately find Database.jsonPatch.

What a neat solution you ended up with. I like how Database.jsonPatch reads, fits really well into my mental model at least.

Which subset of JSON features should ship?

All (but json_each and json_tree) :-)

This also made it possible to write the JSON documentation tour, JSON.md, and patch necessary holes (such as the new DatabaseDataEncodingStrategy.text, which is not a pure JSON feature).

🎉

I had started writing a doc about handling JSON with SQLite and GRDB, but it's a bit more practical and end-to-end, maybe I can expand the docs once I finish it.

How to expose json_extract, -> and ->>?

As all SQLite JSON functions, json_extract is available as Database.jsonExtract.

json_extract, ->, and ->> are available on values of type SQLJSONExpressible, such as JSONColum("info"), or the equivalent Colum("info").asJSONValue.

I have favored ->> above all others with a subscript, because it is the only way to provide a plain JSON key instead of a $-prefixed path:

After playing around with your earlier experimental branch, I realized that this is probably the best choice (vs. json_extract for example) 👍

• The ->> SQL operator:

  let info = JSONColumn("info")
  
  // SELECT info ->> 'firstName' FROM player
  // → 'Arthur'
  let firstName = try Player
      .select(info["firstName"], as: String.self)
      .fetchOne(db)
  
  // SELECT info ->> 'address' FROM player
  // → '{"street":"Rue de Belleville","city":"Paris"}'
  let address = try Player
      .select(info["address"], as: String.self)
      .fetchOne(db)

This turned out nicely. The subscript syntax has really grown on me, it nicely indicates the sub-queryish nature of accessing JSON.

• The -> SQL operator:

  let info = JSONColumn("info")
  
  // SELECT info -> 'firstName' FROM player
  // → '"Arthur"'
  let name = try Player
      .select(info.jsonRepresentation(forKey: "firstName"), as: String.self)
      .fetchOne(db)
  
  // SELECT info -> 'address' FROM player
  // → '{"street":"Rue de Belleville","city":"Paris"}'
  let name = try Player
      .select(info.jsonRepresentation(forKey: "address"), as: String.self)
      .fetchOne(db)

  The name of jsonRepresentation for -> might be discussed. It always return a valid JSON string. In particular, extracted JSON string values are returned wrapped inside ".

I think jsonRepresentation is good, but maybe forKey could be forPath or atPath to indicate that it supports accessing nested keys (considering forKey in Apple's APIs tends to refer to dictionary-style access)?

What is JSONColumn?

JSONColumn(columnName) is just sugar. All column-building techniques are supported:

struct Player: Codable {
    var address: Address

    enum Columns1 {
        static let address = JSONColumn(CodingKeys.address)
    }

    enum Columns2 {
        static let address = Column(CodingKeys.address)
    }

    enum Columns3: String, ColumnExpression {
        case address
    }
}

// 100% identical behavior
Player.Columns1.address
Player.Columns2.address.asJSONValue
Player.Columns3.address.asJSONValue

In this PR, JSONColumn and values returned by asJSONValue are Swift values that:

1. Provide access to JSON subcomponents:

   JSONColumn("address")["street"] // Compiles
   Column("address")["street"]     // Does not compile

2. Have no special behavior in SQL...

   // SELECT address, address, address FROM player
   let json = try Player
       .select(
           Column("address"),
           Column("address").asJSONValue,
           JSONColumn("address"))
       .fetchOne(db)

3. ... except in JSON building contexts.
   In the example below, we can see how the JSONColumn is interpreted as JSON, while the Column with the same name is interpreted as a plain string:

   let elements: [any SQLExpressible] = [
       JSONColumn("address"),
       Column("address"),
   ]
   
   let array = Database.jsonArray(elements)
   
   // SELECT JSON_ARRAY(JSON(address), address) FROM player
   // → '[{"country":"FR"},"{\"country\":\"FR\"}"]'
   //     <--- object ---> <------ string ------>
   let json = try Player
       .select(array, as: String.self)
       .fetchOne(db)

Having a separate JSONColumn entity was the part I was most unsure about in your earlier experimentation, but now it makes a ton of sense.

Overall, I really like the improvements on ensuring that values are proper JSON, and how that's indicatated in the APIs. I think I was missing the larger context earlier.

[groue]

This looks awesome and solves so many more use cases than I had planned! I have a busier than usual week, so it'll likely take until the weekend before I can test this in my app and review it properly. Feel free to merge this earlier if you feel like it. Meanwhile, I had a quick read-through and added some comments below.

Glad it seems to fit 👍

I had started writing a doc about handling JSON with SQLite and GRDB, but it's a bit more practical and end-to-end, maybe I can expand the docs once I finish it.

I'll be happy to hear your suggestions on the GRDB doc!

How to expose json_extract, -> and ->>?
As all SQLite JSON functions, json_extract is available as Database.jsonExtract.
json_extract, ->, and ->> are available on values of type SQLJSONExpressible, such as JSONColum("info"), or the equivalent Colum("info").asJSONValue.
I have favored ->> above all others with a subscript, because it is the only way to provide a plain JSON key instead of a $-prefixed path:

After playing around with your earlier experimental branch, I realized that this is probably the best choice (vs. json_extract for example) 👍

I'm relieved. Your opinion on this is very important to me, because you'll be an actual user of those features :-)

• The -> SQL operator:
  [...]
  The name of jsonRepresentation for -> might be discussed. It always return a valid JSON string. In particular, extracted JSON string values are returned wrapped inside ".

I think jsonRepresentation is good, but maybe forKey could be forPath or atPath to indicate that it supports accessing nested keys (considering forKey in Apple's APIs tends to refer to dictionary-style access)?

You're right. atPath is just better, even if -> accepts plain keys not prefixed with $.

What is JSONColumn?

Having a separate JSONColumn entity was the part I was most unsure about in your earlier experimentation, but now it makes a ton of sense.

I'm relieved, again :-)

OK, I'll rename jsonRepresentation, maybe rename asJSONValue to asJSON, massage the doc a little bit more, and eventually merge. Thank you so much @myyra for your initiative and help!

[groue]

The PR was just shipped in v6.19.0 :-)