Add generics and implementations for custom Date and POSIXt escaping #391

OssiLehtinen · 2019-12-16T09:47:06Z

These modifications address the issue described in #384

…a implementations.

OssiLehtinen · 2019-12-16T10:15:29Z

Travis CI was not too happy about my pull request. I'm not familiar with Travis CI so not sure how to proceed.

OssiLehtinen · 2019-12-18T08:48:59Z

I added export of sql_escape_string.

This would enable optionally setting, e.g.,

sql_escape_string.AthenaConnection <- function(con, x) {
  if(try(as.Date(x), silent=T)  == x) {
    paste0('date ', DBI::dbQuoteString(con, x))
  } else {
    DBI::dbQuoteString(con, x)
  }
}

meaning a call, such as tbl(con, "calendar") %>% filter(date_col == "2019-01-01"), would work, as the string gets identified as an isodate and the 'date' prefix gets added.
This is obviously very much subjective, but in pretty much all of our ad-hoc analysis use cases we need to do filtering of data by date, so any streamlining of this would go a long way.

However, I'm uncertain whether automatic detection of isodates should be the default behaviour. This change would enable doing this if wanted.

OssiLehtinen · 2019-12-20T12:32:15Z

A bit of a mess, but I think something like this would actually be needed to have date vectors work out and have only iso-dates cast to dates. Automatically casting something like "1" to a date is not a good thing.

sql_escape_string.AthenaConnection <- function(con, x) {
  all_dates <- all(try(as.Date(x, tryFormats = "%Y-%m-%d"), silent=T) == x)
  if(all_dates & !is.na(all_dates)) {
    paste0('date ', DBI::dbQuoteString(con, x))
  } else {
    DBI::dbQuoteString(con, x)
  }
}

In any case, I don't think this should be the default behaviour of dbplyr, but exporting the generic would allow a user to shoot him/herself in the foot, if that is what he/she wishes to do.

OssiLehtinen · 2020-01-03T12:00:52Z

One more note:

if one would define sql_escape_string.AthenaConnection as described above, the escape.Date definition would not be necessary, as currently the dates get cast to strings first and the sql_escape_string function is called on those, which would then detect them as dates.

However, still I don't think the sql_escape_string should be defined like that as default, so having the Athena-specific escape.Date (and escape.POSIXt) would be a good thing imho.

p.s. If I'm not mixing up things (still learning about s3-classes etc), exporting sql_escape_string is actually not necessary to be able to define the Athena specific version, right?

hadley

Looks good thanks — don't worry about the travis failures; I'll fix those shortly. Can you please add a bullet to the top of NEWS.md? It should briefly describe the change and end with (@yourname, #issuenumber).

Yeah, we definitely don't want to do that magic date wrapping by default. I'm not sure how can we make that more elegant in the future. Maybe we could add a ymd() helper than dbplyr knew to execute locally?

sql_escape_string() is complicated because it's exported by dplyr. At some point I have to centralise all the database methods in this package, but it's a bit fiddly.

hadley · 2020-04-15T13:45:21Z

R/escape.R

@@ -69,14 +69,13 @@ escape.factor <- function(x, parens = NA, collapse = ", ", con = NULL) {

 #' @export
 escape.Date <- function(x, parens = NA, collapse = ", ", con = NULL) {
-  x <- as.character(x)
-  escape.character(x, parens = parens, collapse = collapse, con = con)
+  x = as.character(x)


I think you should delete this line; sql_escape_date() should be in charge of converting from a date to a character

Moved to coverting bit to sql_escape_date.

hadley · 2020-04-15T13:45:21Z

R/escape.R

@@ -229,13 +228,37 @@ sql_escape_logical <- function(con, x) {
  UseMethod("sql_escape_logical")
 }

+#' @keywords internal
+#' @export
+sql_escape_Date <- function(con, x) {


I'd prefer to call these sql_escape_date() and sql_escape_datetime().

Renamed the functions.

hadley · 2020-04-15T18:58:31Z

(Also since it's a been a while since you submitted this PR, please let me know if you're not interested, and I'll finish it)

krlmlr · 2020-04-16T02:26:50Z

R/escape.R

 # DBIConnection methods --------------------------------------------------------

 #' @export
 sql_escape_string.DBIConnection <- function(con, x) {
  dbQuoteString(con, x)
 }

+#' @export
+sql_escape_Date.DBIConnection <- function(con, x) {
+  dbQuoteString(con, x)


DBI has this, DBI backends can override.

Suggested change

dbQuoteString(con, x)

dbQuoteLiteral(con, x)

I'm sorry, could you explain a bit what would be the benefit of this change? dbQuoteLiteral seems to format a date as something like '2020-04-15 UTC' which does not work at least in Athena (don't have other databases up right now to test against). So if I would use dbQuoteLiteral, I would still need to first cast the date as character.

Surely I'm just missing the point here, so please help me out :)

krlmlr · 2020-04-16T02:26:50Z

R/escape.R

+  x <- strftime(x, "%Y-%m-%dT%H:%M:%OSZ", tz = "UTC")
+  dbQuoteString(con, x)


Should this be %OS6? DBI needs to be fixed here -- I think passing the time zone might be correct here.

Suggested change

x <- strftime(x, "%Y-%m-%dT%H:%M:%OSZ", tz = "UTC")

dbQuoteString(con, x)

dbQuoteLiteral(con, x)

This specific formatting is what is in place in dbplyr at the moment for POSIXt, so I wouldn't want to touch that.

…Date and sql_escape_datetime and move the string casting to sql_escape_date() from escape.Date()

…cs and methods

OssiLehtinen · 2020-04-16T08:22:49Z

I made changes to the code according to the recommendations. I hope I got it right. Could you @hadley take a look at the news bullet, as I'm a bit uncertain about the terminology? Thanks!

hadley · 2020-04-16T13:20:33Z

Thanks @OssiLehtinen — I finished it by updating your branch with the latest changes on master, tweaking the news bullet, and making a couple of small documentation and test fixes. Thanks for all your work on this!

OssiLehtinen · 2020-04-16T13:24:44Z

Thanks @OssiLehtinen — I finished it by updating your branch with the latest changes on master, tweaking the news bullet, and making a couple of small documentation and test fixes. Thanks for all your work on this!

I'm very happy to be able to contribute even a little bit to this great stack of tools!

Ossi Lehtinen added 2 commits Dec 16, 2019

Enable backend specific escaping of dates and datetimes and add Athen…

1996630

…a implementations.

Add Presto specific date and timestamp escaping

7a5b54d

Export sql_escape_string (and reorder S3Methods alphabetically)

d4119e8

hadley reviewed Apr 15, 2020

View changes

krlmlr reviewed Apr 16, 2020

View changes

Ossi Lehtinen added 2 commits Apr 16, 2020

Rename functions sql_escape_Date and sql_escape_POSIXt to sql_escape_…

d6720bb

…Date and sql_escape_datetime and move the string casting to sql_escape_date() from escape.Date()

Add a new bullet about sql_escape_date and sql_escape_datetime generi…

88fbc32

…cs and methods

hadley added 5 commits Apr 16, 2020

merge origin/master

7f0be33

Re-document

31d3807

Tweak NEWS bullet

a419a56

Use sql_escape_string to faciliate test

37803c2

Document new generics

3b8460c

hadley merged commit 6909b1a into tidyverse:master Apr 16, 2020
12 checks passed

OssiLehtinen deleted the athena_dates branch Apr 16, 2020

OssiLehtinen mentioned this pull request Apr 21, 2020

having datetime-strings in input parameters will produce errors DyfanJones/RAthena#75

Open

This was referenced Sep 22, 2020

Non standard handling of iso-date strings in PrestoDB/AWS Athena #384

Closed

Custom escape methods for date-times DyfanJones/RAthena#121

Closed

Custom escape methods for date-times prestodb/RPresto#123

Closed

Add generics and implementations for custom Date and POSIXt escaping #391

Add generics and implementations for custom Date and POSIXt escaping #391

OssiLehtinen commented Dec 16, 2019

OssiLehtinen commented Dec 16, 2019

OssiLehtinen commented Dec 18, 2019 •

edited

OssiLehtinen commented Dec 20, 2019

OssiLehtinen commented Jan 3, 2020

hadley left a comment

hadley Apr 15, 2020

OssiLehtinen Apr 16, 2020

hadley Apr 15, 2020

OssiLehtinen Apr 16, 2020

hadley commented Apr 15, 2020

krlmlr Apr 16, 2020

OssiLehtinen Apr 16, 2020

krlmlr Apr 16, 2020

OssiLehtinen Apr 16, 2020

OssiLehtinen commented Apr 16, 2020

hadley commented Apr 16, 2020

OssiLehtinen commented Apr 16, 2020

		x <- strftime(x, "%Y-%m-%dT%H:%M:%OSZ", tz = "UTC")
		dbQuoteString(con, x)

	x <- strftime(x, "%Y-%m-%dT%H:%M:%OSZ", tz = "UTC")
	dbQuoteString(con, x)
	dbQuoteLiteral(con, x)

Add generics and implementations for custom Date and POSIXt escaping #391

Add generics and implementations for custom Date and POSIXt escaping #391

Conversation

OssiLehtinen commented Dec 16, 2019

OssiLehtinen commented Dec 16, 2019

OssiLehtinen commented Dec 18, 2019 • edited

OssiLehtinen commented Dec 20, 2019

OssiLehtinen commented Jan 3, 2020

hadley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hadley commented Apr 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

OssiLehtinen commented Apr 16, 2020

hadley commented Apr 16, 2020

OssiLehtinen commented Apr 16, 2020

OssiLehtinen commented Dec 18, 2019 •

edited