Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON data type #44

Open
sgrif opened this issue Dec 3, 2015 · 21 comments

Comments

Projects
None yet
@sgrif
Copy link
Member

commented Dec 3, 2015

So let's talk borderline absurd features. I'm trying to figure out how we can support this type, while still being useful. I feel like this is harder to do than in dynamic languages where we're just an arbitrary map or array. However, I think we can do something moderately useful with serde integration (I am unsure if this should be in core, or a separate trait)

However, I am fairly convinced that we can do impl<T> ToSql<Json> for T where T: serde::ser::Serialize and impl<T> FromSql<Json> for T where T: serde::de::Deserialize (note: As written, those two almost certainly require specialization. We can work around this with boilerplate macros to implement for individual types in the short term).

The following operators exist for the JSON data types:

  • ->: Query DSL should just implement Index for Json expressions. Output should be Expression<SqlType=Json>. We might need to do some hackery since Index requires returning a reference. A Copy constraint will likely be involved as well. Might need to be a method if we really can't work with Index, but boy it would be amazing to write filter(json_column["foo"].eq("bar"))
  • ->>: Unsure if we should support. If so, will need to be a method. Unclear on the right name.
  • #>: Unsupported
  • #>>: Unsupported
  • @>: Implemented as method contains. Argument should be AsExpression<Json>, return type is Bool`
  • <@: Unsupported
  • ?: Implemented as method has_key. Argument should be AsExpression<VarChar>, return type is Bool
  • ?| and ?&: Same as above. Don't have an opinion on the method names.

We should also support as many functions as possible from http://www.postgresql.org/docs/9.4/static/functions-json.html. As I write this, I'm fairly convinced this should be a separate crate. If anyone wants to tackle this, let me know and I'll add a new repo for it. I do want this to be supported under the general Diesel umbrella though, even if it's not in the core crate.

@sgrif

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

And just so it's clear, if we do this right, we should be able to have anything which implements serde::ser::Serialize implement AsExpression<Json> via ToSql<Json>

@archer884

This comment has been minimized.

Copy link

commented May 10, 2016

So, I was just trying to use Jsonb in a table and I had kind of assumed--until I ran into this error--that diesel would give the json to me as a string and I could deserialize it myself. No dice. Instead, I get a compilation error. Rather than wait for the absurd feature (querying jsonb with diesel, I guess?), what about just letting diesel give me the data instead of exploding? :)

@sgrif

This comment has been minimized.

Copy link
Member Author

commented May 10, 2016

Yes, you can fairly easily add support for any types that you want within
your application. I'm on my phone but will give examples later

Rather than wait for the absurd feature

Unsure what that is supposed to mean

On Tue, May 10, 2016, 5:02 PM archer884 notifications@github.com wrote:

So, I was just trying to use Jsonb in a table and I had kind of
assumed--until I ran into this error--that diesel would give the json to me
as a string and I could deserialize it myself. No dice. Instead, I get a
compilation error. Rather than wait for the absurd feature (querying jsonb
with diesel, I guess?), what about just letting diesel give me the data
instead of exploding? :)


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#44 (comment)

@archer884

This comment has been minimized.

Copy link

commented May 10, 2016

That part was quoting you from above:

So let's talk borderline absurd features.

:)

@sgrif

This comment has been minimized.

Copy link
Member Author

commented May 10, 2016

Oh. I didn't realize this was a comment on an existing issue. XD

@sgrif

This comment has been minimized.

Copy link
Member Author

commented May 10, 2016

So basically what you'd need to do is:

struct JsonB;

impl HasSqlType<JsonB> for Pg {
    // ...
}

impl FromSql<JsonB, Pg> for String {
    // ...
}

expression_impls!(JsonB -> String);

Unsure what the binary rep of jsonb is so you might need to do some digging there

@jimmycuadra

This comment has been minimized.

Copy link
Contributor

commented Jun 20, 2016

@archer884 Did you end up implementing this as Sean described? I have a table with a jsonb column and was hoping someone had already done the heavy lifting of creating a Diesel type for it that I could copy/paste. :}

@lancecarlson

This comment has been minimized.

Copy link
Contributor

commented Nov 21, 2016

I need this too. This and Tsvector get hung up. How difficult is this to implement? It looks like I may have to drop down to rust-postgres...

@theduke

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

Any movement on this?

It's really crucial for me.

Most important would be just supporting fields with a "Value" type from serde_json.

@norcalli

This comment has been minimized.

Copy link

commented Dec 23, 2016

I would like to use diesel for a project at my job, and we use jsonb extensively. I would like to take on the project of implementing this, but I would need some help getting started.

@emk

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2016

I have a rather pressing need for this on a work project that's experimenting heavily with diesel, and I can almost certainly talk my boss into letting me work on this on company time extremely soon. (We really like diesel so far! But we want to use it for more things.)

Probably the nicest and richest JSON representation in Rust right now is serde_json::Value. It's supported by lots of other libraries, and you can serialize and deserialize it in either direction:

raw JSON String or Vec<u8>serde_json::Value ↔ Rust data structure implementing Serialize or Deserialize

It's super handy. The rust-postgres crate has some code showing how to set this up.

Of course, this could be put under a serde_json feature, like the Uuid and chrono types are.

Should I just go ahead and bang out a PR showing what I mean?

@killercup

This comment has been minimized.

Copy link
Member

commented Dec 31, 2016

Wow, I hadn't realized how many people want to have this! :) @emk, it would be amazing if you got company time to work on this! Maybe you, @archer884 and @norcalli can all work together on this!

Should I just go ahead and bang out a PR showing what I mean?

Maybe. Or, you could try and do this as a plugin crate. (If you think adding this to diesel is easier and more efficient, you can of course still open a PR!)

Diesel has a bunch of facilities to make implementing custom types easier, and as @sgrif mentions in the issue description, it may be possible to add this in a separate crate. This may be a worthwhile effort as it could also show that it is possible to write plugins for diesel and how to do it. Currently, there is just diesel_full_text_search that I know of (and it doesn't even have a Readme… I just field diesel-rs/diesel_full_text_search#1).

(Initially I was worried how quickly you'll run into the limits of the current orphan rules, but I think you'll be fine if you define a custom Jsonb type and impl all traits on this, or as extension traits.)

@emk

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2016

@killercup Thank you for the encouragement!

I've opened a "work in progress" PR at #561, just so we can visualize what I'm thinking about. I agree that it would be an interesting possibility to do this as a separate plugin, but I'd prefer to try that after we see it working inside diesel itself!

Also, for the first version, I'm just targeting serialization and deserialization, and not the specialized JSON query operators. If we can begin by getting JSON into and out of the database, that will already help with a lot of use cases.

Anyway, anybody who wants to help out is encouraged to check out #561, and submit comments, bug reports, and further PRs. :-) I'll continue the implementation discussion there.

@emk

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2016

I now have a preliminary implementation of the json and jsonb data types and conversion to and from serde_json::Value at #561! I haven't attempted to implement any of the JSON-specific query operators.

See #561 for instructions on how to try this out, and please let me know if it works!

@sgrif sgrif added this to the 0.10 milestone Jan 3, 2017

@sgrif

This comment has been minimized.

Copy link
Member Author

commented Jan 3, 2017

Hey folks -- today's my first day back from holiday vacation. I'm getting caught up on issues now. I've gone ahead and added this feature to the 0.10 timeline, as there's clearly a lot of demand for it.

https://github.com/diesel-rs/diesel_full_text_search was mostly just a proof of concept to demonstrate some of the basics of how to add support for additional extensions outside of Diesel. It "conveniently" tackled some types which didn't require ToSql/FromSql implementations. I'd like to see a plugin crate done to ensure we have appropriate APIs in place, but I'm fine with adding this to Diesel proper for now and exploring that space at a later date.

I'll leave implementation specific comments on #561.

@lholden

This comment has been minimized.

Copy link

commented Jan 4, 2017

@sgrif have you seen the rust-postgres-derive crate? "Syntax extensions to automatically derive FromSql and ToSql implementations for Postgres enum, domain, and composite types." https://github.com/sfackler/rust-postgres-derive

Something like it might make sense for Diesel.

@sgrif sgrif removed this from the 0.11 milestone Feb 15, 2017

@dbrgn

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2017

PR #561 was merged, but this is still open. (For the record, Jsonb support can be enabled with the serde_json feature.)

It looks like the basic support is here, but without the operators, right?

@killercup

This comment has been minimized.

Copy link
Member

commented Jun 27, 2017

@dbrgn exactly. We should probably make this a meta issue with a check list of stuff we want to have.

@tyre

This comment has been minimized.

Copy link

commented Aug 23, 2018

What's the work involved in supporting any serde_json::Serializable vs. just serde_json::Value? Happy to help out if I can.

The use case is an API client that returns a struct. The struct is serializable, but isn't a serde_json::Value, so to store the raw value I need to re-serialize, then re-de-serialize into a generic serde_json::Value.

@jonnybrooks

This comment has been minimized.

Copy link

commented Nov 30, 2018

Was wondering if any progress has been made implementing json operators e.g. -> and such yet? If not, I'd love to try to implement this as I'm very interested in seeing this functionality in Diesel

@weiznich

This comment has been minimized.

Copy link
Member

commented Nov 30, 2018

@jonnybrooks As far as I know there is no process in those operators made. Feel free to try to implement it. If you hit any problem just ask in out gitter room.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.