Add support for quantile()

[halldc]

Would it be possible to add support for the quantile() function?

Quite a few databases support PERCENTILE_CONT and PERCENTILE_DISC, which I think could make this possible (e.g. Google BigQuery, PostgreSQL, and Redshift).

I'd be willing to help of course, but would need some pointers on where to start.

[Prometheus77]

I feel like there might be a way to do this with win_over(). I'm assuming you want to be able to write an expression such as:

SELECT P90 = PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY Score) OVER (PARTITION BY [Month])

Here's the documentation for win_over:
https://dbplyr.tidyverse.org/reference/win_over.html

The closest I can get to this is:

dbplyr::win_over(sql("percentile_cont(0.9)"), "Month", order = "score")

Which yields:

<SQL> percentile_cont(0.9) OVER (PARTITION BY "Month" ORDER BY "score")

I'd have to spend more time with it to see if I can get it to do what I want.

[hadley]

@halldc If you wanted to help, the place to start would be by making a list of databases (starting with SQLite, MySQL, and Postgres, and then continuing as long as you like) with a link to the documentation for their quantile function.

[edavidaja]

I'll work on this.

@batpigandme working on for tidy dev day

[edavidaja]

engine	syntax
msft sql server	PERCENTILE_CONT ( numeric_literal ) WITHIN GROUP ( ORDER BY order_by_expression [ ASC | DESC ] ) OVER ( [ <partition_by_clause> ] )
teradata	approx_percentile(x, percentage) ;
hive	percentile(BIGINT col, array(p1 [, p2]...))
impala	work in progress
postgres	percentile_cont(fraction) WITHIN GROUP (ORDER BY sort_expression)
redshift	PERCENTILE_CONT ( percentile ) WITHIN GROUP (ORDER BY expr) OVER ( [ PARTITION BY expr_list ] )
salesforce	percentile_cont(p) within group (order by expr [asc | desc])
monetdb	QUANTILE '(' scalar_expr ',' scalar_expr ')'
mysql	not supported
mariadb	SELECT name, PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY star_rating) OVER (PARTITION BY name) AS pc FROM book_rating;
sqlite	not supported
bigquery	PERCENTILE_CONT (value_expression, percentile [{RESPECT | IGNORE} NULLS])

[hadley]

@edavidaja can you please reduce the text (by editing your comment) so every thing fits on a single page? It's more important to me to see an overview in one place than to capture all the details (which I can get to by following the links)

[krlmlr]

One way to look at the SQL translation without having access to a real database:

library(tidyverse)
library(dbplyr)
#> 
#> Attaching package: 'dbplyr'
#> The following objects are masked from 'package:dplyr':
#> 
#>     ident, sql

tbl_lazy(iris, src = simulate_dbi("PostgreSQLConnection")) %>%
  transmute(q = quantile(Petal.Length, .25)) %>%
  show_query()
#> <SQL> SELECT QUANTILE(`Petal.Length`, 0.25) AS `q`
#> FROM `df`

^{Created on 2019-01-19 by the reprex package (v0.2.1)}

[hadley]

@edavidaja that just hides the details so I have to do even more clicking. I basically want a one screen cheatsheet that lets me see the variation at a glance.

[halldc]

Thanks for your work on this @edavidaja ! I'm hoping that in a few weeks' time I should be able to contribute to this task, if that helps.

By the way, I see in your table that you're mixing NTILE with PERCENTILE/QUANTILE functions, but these are used in different contexts.

• The NTILE function divides the data into a specified number of ranked groupings, and associates each row with one of these groups. See Redshift example. This is similar to dplyr::ntile().
• The PERCENTILE/QUANTILE functions yield an aggregate statistic (e.g. the median is the 50th percentile). See Redshift example. This is similar to the R function quantile().

[edavidaja]

Thanks @halldc. I've updated the table again--there are a few engines where some percentile_cont like function doesn't appear to be available at all; I've linked to the relevant documentation in case I'm reading it incorrectly (again).

A few engines (postgres, teradata, hive) can accept and return arrays in a way that matches the behavior of quantile in R; the remainder expect a single value.

[hadley]

@edavidaja that is perfect, thank you!