-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JSON type representations and conversions to store types #30727
Comments
Design decisions:
|
Related - #10434 (comment) |
I did an investigation into binary support on SQL Server.
SELECT y.bar
FROM OPENJSON('["AAAwOQ==", "AAEJMg=="]') AS x
CROSS APPLY OPENJSON('["' + x.value + '"]') WITH (bar VARBINARY(MAX) '$') AS y
ORDER BY x.[key]; Full investigationSELECT * FROM (VALUES(CAST(12345 AS VARBINARY(MAX)))) AS foo(bar) FOR JSON AUTO; -- [{"bar":"AAAwOQ=="}]
-- OPENJSON does the same, decoding the JSON base64 string to VARBINARY:
SELECT * FROM OPENJSON('["AAAwOQ=="]') WITH (bar VARBINARY(MAX) '$'); -- 0x00003039
-- In fact, trying to decode a non-base64 fails very explicitly with:
-- Cannot convert a string value found in the JSON text to binary value because it is not Base64 encoded.
SELECT * FROM OPENJSON('["foo"]') WITH (bar VARBINARY(MAX) '$');
-- However, OPENJSON with WITH doesn't support preserving the original order.
-- For that we use OPENJSON without WITH, but then we need to do regular relational casting.
-- This *doesn't* work with base64:
SELECT CAST(value AS VARBINARY(MAX)) FROM OPENJSON('["AAAwOQ=="]'); -- 41007700-4100-4100-4f00-51003d003d00
-- The only option seems to both preserve ordering and properly decode a base64 string out of JSON seems to be to use
-- OPENJSON *again* with WITH to perform the conversion:
SELECT y.bar
FROM OPENJSON('["AAAwOQ==", "AAEJMg=="]') AS x
CROSS APPLY OPENJSON('["' + x.value + '"]') WITH (bar VARBINARY(MAX) '$') AS y
ORDER BY x.[key]; |
Update on this... As previously decided, we ended up implementing some conversions when extracting JSON values out in SQLite (Guid to uppercase, add remove T from timestamp). For some types, it was impossible to reliably convert (binary, decimal, DateTimeOffset); we decided to allow storing these types, but to block their querying as that can't be supported. However, JSON value extraction is also necessary for the new Contains implementation; this means that it would no longer be possible to do We considered two options:
Since supporting standard JSON representations has turned out to be increasingly difficult and has added complexity, we've opted for option 2. Additional notes:
|
@roji Take a look at SQLite translations. |
When querying against values coming out of a JSON document, we need to make sure that the JSON values get projected out as the correct store type, otherwise comparison with e.g. a regular column will fail. For SQL Server, most store types work by simply casting the string representation returned from OPENJSON; but with SQLite - which has very few types - there are some issues:
Options:
For option 1, we need type mapping API support which tells us how to convert values coming out of a JSON document into their equivalent relational store representation. Note that this is complicated somewhat in SQL Server, where in most cases we can simply use OPENJSON/WITH which automatically does the conversion for us, but in some cases we can't do that (because OPENJSON/WITH doesn't preserve ordering, and is also incompatible with geometry).
The text was updated successfully, but these errors were encountered: