New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Binary objects truncated to 255 characters when read #202
Comments
|
@etiennebr I think the issue is that nanodbc doesn't provide support for JSON (probably because it's not required in the ANSI SQL standard) and falls back to One solution is to cast the JSON as text before collecting the result from the database. I think it would be a nice feature to be able to register custom types - e.g. automatically cast JSON columns as text and pass to |
|
Is this a problem across different databases? I've seen problems for accessing a geometry column from SQL Server, where using |
|
@jimhester: Does this ring a bell? Is there anything specific in |
|
I think there likely is a parameter that could be changed in the driver options, but I have not yet looked into how Postgres stores JSON data. Will certainly do so next time I am working on odbc, which should be in the next month or two. |
|
I'd like to stress @etiennebr's last sentence, meaning that this issue is not limited to JSON data, but binary data in general. Originally, it came up with reading a geometry column from PostgreSQL/PostGIS. |
|
A workaround is to convert json to jsonb, which is the recommended format in the PostgreSQL documentation.
library(magrittr)
library(DBI)
con <- dbConnect(odbc::odbc(), "PostgreSQL")
query <- "SELECT ('[' || repeat('1,', 150) || '2]')::json;"
dbGetQuery(con, query)$json
#> [1] "[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2]"
dbExecute(con, "DROP TABLE x")
#> [1] 154109836
dbExecute(con, glue::glue("CREATE TABLE x AS {query}"))
#> [1] 1
dbGetQuery(con, "SELECT to_jsonb(json) as json2 FROM x")$json2 %>% nchar()
#> [1] 453Created on 2019-11-19 by the reprex package (v0.3.0) Note the data is still a character vector when transmitted to R, but it does not have the 255 truncation limit. The last query is larger than the first due to added spaces between items in the output. |
|
But there still seems to be issues, and notably even |
|
Setting the |
|
As I don't think we will be adding Postgres specific types anytime soon and the fact there is a decent workaround for this I think I am going to close the issue. |
|
I think @ellisvalentiner's solution works beautifully. I wonder how we could make it easier for users to either discover the source of the problem, or set a more robust default. I have no idea what's involved in making Since there are no warnings when the varchar are truncated, it is hard for the user to understand what's the origin of the error and can require a significant investigation to understand that the characters are truncated during the conversion of a binary object. I believe raising a warning when the max is reached could be a solution, but is more complex than setting a default. |
|
The default is set by the database, not by nanodbc or the odbc package. Also I doubt that |
Binary objects seem to be truncated when converted to characters somewhere when reading from a table (executing the query straight works fine). It is probably an interaction between the db engine and
odbc, since it works forRPostgresconnections.Created on 2018-07-23 by the reprex package (v0.2.0).
Session info
Database
PostgreSQL 10.3 (Debian 10.3-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bit
Related to r-spatial/sf#721 (I simplified the example by using json rather than postgis extension.)
The text was updated successfully, but these errors were encountered: