Optional method of using Ibis backend connectors

[haydenbspence]

This is a suggestion for a feature enhancement.

The Ibis Project is a Python library that provides a lightweight, universal interface for data wrangling with the support of 15+ popular backends. Wes McKinney is a co-founder, better known for his authorship of the Pandas library, and Ibis is actively maintained by 100+ contributors.

If someone has both R and Python installed, is it feasible create a method that calls Ibis within DatabaseConnector? For example through Reticulate in such a way that these additional backends that Ibis supports can be used? Potentially by using a pass-through common database like DuckDB? This could significantly reduce the amount of backends needing active support by DatabaseConnector while still offering support so long as Python is able to be installed alongside R.

[haydenbspence]

Here is some sample code that seems to work initially on smaller pulls. Performance is likely not great due to this being all in-memory.

Some ideas:

• Add temp store for larger than memory operations (duckdb?)

library(reticulate)
library(dplyr)

ibis <- import('ibis')

con <- ibis$postgres$connect(
  host = 'hostname',
  port = 5432,
  user = 'username',
  password = 'password',
  database = 'database'
)

ibis_table <- con$table('table')

pandas_df <- ibis_table$execute()

data <- py_to_r(pandas_df)

result <- data %>%
  filter(column_name == "value") %>%
  summarise(mean_value = mean(column))

glimpse.result()

[ablack3]

Personally I think this would be better for a DatabaseConnector/SqlRender component in python. It's an interesting idea to bring it into R...