135 changes: 135 additions & 0 deletions docs/backends/bigquery.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# BigQuery

[https://cloud.google.com/bigquery](https://cloud.google.com/bigquery)

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-BigQuery tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-BigQuery tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the BigQuery backend:

::: {.panel-tabset}

## `pip`

Install with the `bigquery` extra:

```{.bash}
pip install 'ibis-framework[bigquery]'
```

And connect:

```{.python}
import ibis
con = ibis.bigquery.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for BigQuery:

```{.bash}
conda install -c conda-forge ibis-bigquery
```

And connect:

```{.python}
import ibis
con = ibis.bigquery.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for BigQuery:

```{.bash}
mamba install -c conda-forge ibis-bigquery
```

And connect:

```{.python}
import ibis
con = ibis.bigquery.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.bigquery.connect`

```python
con = ibis.bigquery.connect(
project_id="ibis-bq-project",
dataset_id="testing",
)
```

::: {.callout-note}

`ibis.bigquery.connect` is a thin wrapper around [`ibis.backends.bigquery.Backend.do_connect`](#ibis.backends.bigquery.Backend.do_connect).

:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("bigquery")
```

### `ibis.connect` URL format

In addition to `ibis.bigquery.connect`, you can also connect to BigQuery by
passing a properly formatted BigQuery connection URL to `ibis.connect`

```python
con = ibis.connect(f"bigquery://{project_id}/{dataset_id}")
```

::: {.callout-note}
This assumes you have already authenticated via the `gcloud` CLI.
:::

### Finding your `project_id` and `dataset_id`

Log in to the [Google Cloud Console](https://console.cloud.google.com/bigquery)
to see which `project_id`s and `dataset_id`s are available to use.

![bigquery_ids](./images/bigquery_connect.png)

### BigQuery Authentication

The simplest way to authenticate with the BigQuery backend is to use [Google's `gcloud` CLI tool](https://cloud.google.com/sdk/docs/install).

Once you have `gcloud` installed, you can authenticate to BigQuery (and other Google Cloud services) by running

```sh
gcloud auth login
```

For any authentication problems, or information on other ways of authenticating,
see the [`gcloud` CLI authorization
guide](https://cloud.google.com/sdk/docs/authorizing).

```{python}
#| echo: false
BACKEND = "BigQuery"
```

{{< include ./_templates/api.qmd >}}
85 changes: 0 additions & 85 deletions docs/backends/clickhouse.md

This file was deleted.

132 changes: 132 additions & 0 deletions docs/backends/clickhouse.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# ClickHouse

[https://clickhouse.com](https://clickhouse.com)

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-ClickHouse tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-ClickHouse tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the ClickHouse backend:

::: {.panel-tabset}

## `pip`

Install with the `clickhouse` extra:

```{.bash}
pip install 'ibis-framework[clickhouse]'
```

And connect:

```{.python}
import ibis
con = ibis.clickhouse.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for ClickHouse:

```{.bash}
conda install -c conda-forge ibis-clickhouse
```

And connect:

```{.python}
import ibis
con = ibis.clickhouse.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for ClickHouse:

```{.bash}
mamba install -c conda-forge ibis-clickhouse
```

And connect:

```{.python}
import ibis
con = ibis.clickhouse.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.clickhouse.connect`

```python
con = ibis.clickhouse.connect(
user="username",
password="password",
host="hostname",
)
```

::: {.callout-note}
`ibis.clickhouse.connect` is a thin wrapper around [`ibis.backends.clickhouse.Backend.do_connect`](#ibis.backends.clickhouse.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("clickhouse")
```

### `ibis.connect` URL format

In addition to `ibis.clickhouse.connect`, you can also connect to ClickHouse by
passing a properly formatted ClickHouse connection URL to `ibis.connect`

```python
con = ibis.connect(f"clickhouse://{user}:{password}@{host}:{port}?secure={secure}")
```

## ClickHouse playground

ClickHouse provides a free playground with several datasets that you can connect to using `ibis`:

```{python}
from ibis.interactive import *
con = ibis.clickhouse.connect(
host="play.clickhouse.com",
secure=True,
user="play",
password="clickhouse",
)
con.table("actors")
```

or

```{python}
con = ibis.connect("clickhouse://play:clickhouse@play.clickhouse.com:443?secure=True")
con.table("opensky")
```

```{python}
#| echo: false
BACKEND = "ClickHouse"
```

{{< include ./_templates/api.qmd >}}
9 changes: 0 additions & 9 deletions docs/backends/dask.md

This file was deleted.

74 changes: 74 additions & 0 deletions docs/backends/dask.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Dask

[https://www.dask.org](https://www.dask.org)

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-CSV | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Dask backend:

::: {.panel-tabset}

## `pip`

Install with the `dask` extra:

```{.bash}
pip install 'ibis-framework[dask]'
```

And connect:

```{.python}
import ibis
con = ibis.dask.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Dask:

```{.bash}
conda install -c conda-forge ibis-dask
```

And connect:

```{.python}
import ibis
con = ibis.dask.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Dask:

```{.bash}
mamba install -c conda-forge ibis-dask
```

And connect:

```{.python}
import ibis
con = ibis.dask.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

```{python}
#| echo: false
BACKEND = "Dask"
```

{{< include ./_templates/api.qmd >}}
78 changes: 0 additions & 78 deletions docs/backends/datafusion.md

This file was deleted.

104 changes: 104 additions & 0 deletions docs/backends/datafusion.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# DataFusion

[https://arrow.apache.org/datafusion](https://arrow.apache.org/datafusion)

{{< include /_callouts/experimental_backend.qmd >}}

![](https://img.shields.io/badge/memtables-not supported- grey?style=flat-square) ![](https://img.shields.io/badge/inputs-CSV | Delta Lake | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-CSV | Delta Lake | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Apache DataFusion backend:

::: {.panel-tabset}

## `pip`

Install with the Apache `datafusion` extra:

```{.bash}
pip install 'ibis-framework[datafusion]'
```

And connect:

```{.python}
import ibis
con = ibis.datafusion.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Apache DataFusion:

```{.bash}
conda install -c conda-forge ibis-datafusion
```

And connect:

```{.python}
import ibis
con = ibis.datafusion.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Apache DataFusion:

```{.bash}
mamba install -c conda-forge ibis-datafusion
```

And connect:

```{.python}
import ibis
con = ibis.datafusion.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.datafusion.connect`

```python
con = ibis.datafusion.connect()
```

```python
con = ibis.datafusion.connect(
config={"table1": "path/to/file.parquet", "table2": "path/to/file.csv"}
)
```

::: {.callout-note}
`ibis.datafusion.connect` is a thin wrapper around [`ibis.backends.datafusion.Backend.do_connect`](#ibis.backends.datafusion.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("datafusion")
```

```{python}
#| echo: false
BACKEND = "DataFusion"
```

{{< include ./_templates/api.qmd >}}
70 changes: 0 additions & 70 deletions docs/backends/druid.md

This file was deleted.

111 changes: 111 additions & 0 deletions docs/backends/druid.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Druid

[https://druid.apache.org](https://druid.apache.org)

{{< include /_callouts/experimental_backend.qmd >}}

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-Druid tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-Druid tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Druid backend:

::: {.panel-tabset}

## `pip`

Install with the `druid` extra:

```{.bash}
pip install 'ibis-framework[druid]'
```

And connect:

```{.python}
import ibis
con = ibis.druid.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Druid:

```{.bash}
conda install -c conda-forge ibis-druid
```

And connect:

```{.python}
import ibis
con = ibis.druid.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Druid:

```{.bash}
mamba install -c conda-forge ibis-druid
```

And connect:

```{.python}
import ibis
con = ibis.druid.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.druid.connect`

```python
con = ibis.druid.connect(
host="hostname",
port=8082,
database="druid/v2/sql",
)
```

::: {.callout-note}
`ibis.druid.connect` is a thin wrapper around [`ibis.backends.druid.Backend.do_connect`](#ibis.backends.druid.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("druid")
```

### `ibis.connect` URL format

In addition to `ibis.druid.connect`, you can also connect to Druid by
passing a properly formatted Druid connection URL to `ibis.connect`

```python
con = ibis.connect("druid://localhost:8082/druid/v2/sql")
```

```{python}
#| echo: false
BACKEND = "Druid"
```

{{< include ./_templates/api.qmd >}}
158 changes: 0 additions & 158 deletions docs/backends/duckdb.md

This file was deleted.

155 changes: 155 additions & 0 deletions docs/backends/duckdb.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
description: "Connect to DuckDB with Ibis, the portable dataframe library"
---

# DuckDB

[https://duckdb.org](https://duckdb.org)

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-DuckDB tables | CSV | Delta Lake | JSON | pandas | Parquet | Postgres | PyArrow | SQLite-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-DuckDB tables | CSV | Delta Lake | pandas | Parquet | PyArrow | PyTorch-orange?style=flat-square)

## Install

Install Ibis and dependencies for the DuckDB backend:

::: {.panel-tabset}

## `pip`

Install with the `duckdb` extra:

```{.bash}
pip install 'ibis-framework[duckdb]'
```

And connect:

```{.python}
import ibis
con = ibis.duckdb.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for DuckDB:

```{.bash}
conda install -c conda-forge ibis-duckdb
```

And connect:

```{.python}
import ibis
con = ibis.duckdb.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for DuckDB:

```{.bash}
mamba install -c conda-forge ibis-duckdb
```

And connect:

```{.python}
import ibis
con = ibis.duckdb.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.duckdb.connect`

Connect to an in-memory database:

```python
con = ibis.duckdb.connect()
```

Connect to, or create, a local DuckDB file

```python
con = ibis.duckdb.connect("mydb.duckdb")
```

::: {.callout-note}
`ibis.duckdb.connect` is a thin wrapper around [`ibis.backends.duckdb.Backend.do_connect`](#ibis.backends.duckdb.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("duckdb")
```

### `ibis.connect` URL format

In addition to `ibis.duckdb.connect`, you can also connect to DuckDB by
passing a properly formatted DuckDB connection URL to `ibis.connect`

```{python}
import ibis
con = ibis.connect("duckdb://local.ddb")
```

Without an empty path, `ibis.connect` will connect to an ephemeral, in-memory database.

```{python}
con = ibis.connect("duckdb://")
```

## MotherDuck

The DuckDB backend supports [MotherDuck](https://motherduck.com). If you have an
account, you can connect to MotherDuck by passing in the string `md:` or
`motherduck:`. `ibis` will trigger the authentication prompt in-browser.

```{.python}
import ibis
con = ibis.duckdb.connect("md:")
```

::: {.callout-note}
Authentication to MotherDuck will trigger on the first call that requires retrieving information (in this case `list_tables`)
:::

```{.python}
con.list_tables()
Attempting to automatically open the SSO authorization page in your default browser.
1. Please open this link to login into your account: https://auth.motherduck.com/activate
2. Enter the following code: ZSRQ-GJQS
Token successfully retrieved ✅
You can store it as an environment variable to avoid having to log in again:
$ export motherduck_token='****************'
['penguins']
```

```{python}
#| echo: false
BACKEND = "DuckDB"
```

{{< include ./_templates/api.qmd >}}
500 changes: 228 additions & 272 deletions docs/backends/impala.md → docs/backends/impala.qmd

Large diffs are not rendered by default.

9 changes: 0 additions & 9 deletions docs/backends/index.md

This file was deleted.

18 changes: 0 additions & 18 deletions docs/backends/memtable-template.md

This file was deleted.

68 changes: 0 additions & 68 deletions docs/backends/mssql.md

This file was deleted.

110 changes: 110 additions & 0 deletions docs/backends/mssql.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# MSSQL

[https://www.microsoft.com/sql-server](https://www.microsoft.com/sql-server)

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-MSSQL tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-MSSQL tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the MSSQL backend:

::: {.panel-tabset}

## `pip`

Install with the `mssql` extra:

```{.bash}
pip install 'ibis-framework[mssql]'
```

And connect:

```{.python}
import ibis
con = ibis.mssql.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for MSSQL:

```{.bash}
conda install -c conda-forge ibis-mssql
```

And connect:

```{.python}
import ibis
con = ibis.mssql.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for MSSQL:

```{.bash}
mamba install -c conda-forge ibis-mssql
```

And connect:

```{.python}
import ibis
con = ibis.mssql.connect() # <1>
```

1. Adjust connection parameters as needed.

:::


## Connect

### `ibis.mssql.connect`

```python
con = ibis.mssql.connect(
user="username",
password="password",
host="hostname",
)
```

::: {.callout-note}
`ibis.mssql.connect` is a thin wrapper around [`ibis.backends.mssql.Backend.do_connect`](#ibis.backends.mssql.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("mssql")
```

### `ibis.connect` URL format

In addition to `ibis.mssql.connect`, you can also connect to MSSQL by
passing a properly formatted MSSQL connection URL to `ibis.connect`

```python
con = ibis.connect(f"mssql://{user}:{password}@{host}:{port}")
```

```{python}
#| echo: false
BACKEND = "MSSQL"
```

{{< include ./_templates/api.qmd >}}
69 changes: 0 additions & 69 deletions docs/backends/mysql.md

This file was deleted.

111 changes: 111 additions & 0 deletions docs/backends/mysql.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# MySQL

[https://www.mysql.com](https://www.mysql.com)

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-MySQL tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-MySQL tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the MySQL backend:

::: {.panel-tabset}

## `pip`

Install with the `mysql` extra:

```{.bash}
pip install 'ibis-framework[mysql]'
```

And connect:

```{.python}
import ibis
con = ibis.mysql.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for MySQL:

```{.bash}
conda install -c conda-forge ibis-mysql
```

And connect:

```{.python}
import ibis
con = ibis.mysql.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for MySQL:

```{.bash}
mamba install -c conda-forge ibis-mysql
```

And connect:

```{.python}
import ibis
con = ibis.mysql.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.mysql.connect`

```python
con = ibis.mysql.connect(
user="username",
password="password",
host="hostname",
port=3306,
database="database",
)
```

::: {.callout-note}
`ibis.mysql.connect` is a thin wrapper around [`ibis.backends.mysql.Backend.do_connect`](#ibis.backends.mysql.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("mssql")
```

### `ibis.connect` URL format

In addition to `ibis.mysql.connect`, you can also connect to MySQL by
passing a properly formatted MySQL connection URL to `ibis.connect`

```python
con = ibis.connect(f"mysql://{user}:{password}@{host}:{port}/{database}")
```

```{python}
#| echo: false
BACKEND = "MySQL"
```

{{< include ./_templates/api.qmd >}}
94 changes: 0 additions & 94 deletions docs/backends/oracle.md

This file was deleted.

131 changes: 131 additions & 0 deletions docs/backends/oracle.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Oracle

[https://docs.oracle.com/database/oracle/oracle-database](https://docs.oracle.com/database/oracle/oracle-database)

{{< include /_callouts/experimental_backend.qmd >}}

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-Oracle tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-Oracle tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Oracle backend:

::: {.panel-tabset}

## `pip`

Install with the `oracle` extra:

```{.bash}
pip install 'ibis-framework[oracle]'
```

And connect:

```{.python}
import ibis
con = ibis.oracle.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Oracle:

```{.bash}
conda install -c conda-forge ibis-oracle
```

And connect:

```{.python}
import ibis
con = ibis.oracle.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Oracle:

```{.bash}
mamba install -c conda-forge ibis-oracle
```

And connect:

```{.python}
import ibis
con = ibis.oracle.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.oracle.connect`

```python
con = ibis.oracle.connect(
user="username",
password="password",
host="hostname",
port=1521,
database="database",
)
```

::: {.callout-note}
`ibis.oracle.connect` is a thin wrapper around [`ibis.backends.oracle.Backend.do_connect`](#ibis.backends.oracle.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("oracle")
```

### `ibis.connect` URL format

In addition to `ibis.oracle.connect`, you can also connect to Oracle by
passing a properly formatted Oracle connection URL to `ibis.connect`

```python
con = ibis.connect(f"oracle://{user}:{password}@{host}:{port}/{database}")
```

## Connecting to older Oracle databases

`ibis` uses the `python-oracledb` "thin client" to connect to Oracle databases.
Because early versions of Oracle did not perform case-sensitive checks in
passwords, some DBAs disable case sensitivity to avoid requiring users to update
their passwords. If case-sensitive passwords are disabled, then Ibis will not be
able to connect to the database.

To check if case-sensitivity is enforced you can run

```sql
show parameter sec_case_sensitive_logon;
```

If the returned value is `FALSE` then Ibis will _not_ connect.

For more information, see this [issue](https://github.com/oracle/python-oracledb/issues/26).

```{python}
#| echo: false
BACKEND = "Oracle"
```

{{< include ./_templates/api.qmd >}}
88 changes: 76 additions & 12 deletions docs/backends/pandas.md → docs/backends/pandas.qmd
Original file line number Diff line number Diff line change
@@ -1,15 +1,72 @@
---
backend_name: pandas
backend_url: https://pandas.pydata.org/
backend_module: pandas
intro: Ibis's pandas backend is available in core Ibis.
backend_param_style: a dictionary of paths
do_connect_base: BasePandasBackend
is_core: true
memtable_impl: native
---

{% include 'backends/template.md' %}
# pandas

[https://pandas.pydata.org/](https://pandas.pydata.org/)

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-CSV | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the pandas backend:

::: {.panel-tabset}

## `pip`

Install with the `pandas` extra:

```{.bash}
pip install 'ibis-framework[pandas]'
```

And connect:

```{.python}
import ibis
con = ibis.pandas.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for pandas:

```{.bash}
conda install -c conda-forge ibis-pandas
```

And connect:

```{.python}
import ibis
con = ibis.pandas.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for pandas:

```{.bash}
mamba install -c conda-forge ibis-pandas
```

And connect:

```{.python}
import ibis
con = ibis.pandas.connect() # <1>
```

1. Adjust connection parameters as needed.

:::



## User Defined functions (UDF)

Expand Down Expand Up @@ -137,3 +194,10 @@ def add_two_with_none(x, y=None):
y = 2.0
return x + y
```

```{python}
#| echo: false
BACKEND = "Pandas"
```

{{< include ./_templates/api.qmd >}}
76 changes: 0 additions & 76 deletions docs/backends/polars.md

This file was deleted.

98 changes: 98 additions & 0 deletions docs/backends/polars.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Polars

[https://www.pola.rs](https://www.pola.rs)

{{< include /_callouts/experimental_backend.qmd >}}

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-CSV | Delta Lake | pandas | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-CSV | pandas | Delta Lake | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Polars backend:

::: {.panel-tabset}

## `pip`

Install with the `polars` extra:

```{.bash}
pip install 'ibis-framework[polars]'
```

And connect:

```{.python}
import ibis
con = ibis.polars.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Polars:

```{.bash}
conda install -c conda-forge ibis-polars
```

And connect:

```{.python}
import ibis
con = ibis.polars.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Polars:

```{.bash}
mamba install -c conda-forge ibis-polars
```

And connect:

```{.python}
import ibis
con = ibis.polars.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.polars.connect`

```python
con = ibis.polars.connect()
```

::: {.callout-note}
`ibis.polars.connect` is a thin wrapper around [`ibis.backends.polars.Backend.do_connect`](#ibis.backends.polars.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("polars")
```

```{python}
#| echo: false
BACKEND = "Polars"
```

{{< include ./_templates/api.qmd >}}
69 changes: 0 additions & 69 deletions docs/backends/postgresql.md

This file was deleted.

111 changes: 111 additions & 0 deletions docs/backends/postgresql.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# PostgreSQL

[https://www.postgresql.org](https://www.postgresql.org)

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-Postgres tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-Postgres tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Postgres backend:

::: {.panel-tabset}

## `pip`

Install with the `postgres` extra:

```{.bash}
pip install 'ibis-framework[postgres]'
```

And connect:

```{.python}
import ibis
con = ibis.postgres.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Postgres:

```{.bash}
conda install -c conda-forge ibis-postgres
```

And connect:

```{.python}
import ibis
con = ibis.postgres.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Postgres:

```{.bash}
mamba install -c conda-forge ibis-postgres
```

And connect:

```{.python}
import ibis
con = ibis.postgres.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

### `ibis.postgres.connect`

```python
con = ibis.postgres.connect(
user="username",
password="password",
host="hostname",
port=5432,
database="database",
)
```

::: {.callout-note}
`ibis.postgres.connect` is a thin wrapper around [`ibis.backends.postgres.Backend.do_connect`](#ibis.backends.postgres.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("postgres")
```

### `ibis.connect` URL format

In addition to `ibis.postgres.connect`, you can also connect to Postgres by
passing a properly formatted Postgres connection URL to `ibis.connect`

```python
con = ibis.connect(f"postgres://{user}:{password}@{host}:{port}/{database}")
```

```{python}
#| echo: false
BACKEND = "Postgres"
```

{{< include ./_templates/api.qmd >}}
71 changes: 0 additions & 71 deletions docs/backends/pyspark.md

This file was deleted.

102 changes: 102 additions & 0 deletions docs/backends/pyspark.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# PySpark

[https://spark.apache.org/docs/latest/api/python](https://spark.apache.org/docs/latest/api/python)

![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-PySpark SQL tables | CSV | Delta Lake | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-PySpark SQL tables | Delta Lake | pandas | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the PySpark backend:

::: {.panel-tabset}

## `pip`

Install with the `pyspark` extra:

```{.bash}
pip install 'ibis-framework[pyspark]'
```

And connect:

```{.python}
import ibis
con = ibis.pyspark.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for PySpark:

```{.bash}
conda install -c conda-forge ibis-pyspark
```

And connect:

```{.python}
import ibis
con = ibis.pyspark.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for PySpark:

```{.bash}
mamba install -c conda-forge ibis-pyspark
```

And connect:

```{.python}
import ibis
con = ibis.pyspark.connect() # <1>
```

1. Adjust connection parameters as needed.

:::



## Connect

### `ibis.pyspark.connect`

```python
con = ibis.pyspark.connect(session=session)
```

::: {.callout-note}
`ibis.pyspark.connect` is a thin wrapper around [`ibis.backends.pyspark.Backend.do_connect`](#ibis.backends.pyspark.Backend.do_connect).
:::

::: {.callout-note}
The `pyspark` backend does not create `SparkSession` objects, you must create a `SparkSession` and pass that to `ibis.pyspark.connect`.
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("pyspark")
```

```{python}
#| echo: false
BACKEND = "PySpark"
```

{{< include ./_templates/api.qmd >}}
105 changes: 76 additions & 29 deletions docs/backends/snowflake.md → docs/backends/snowflake.qmd
Original file line number Diff line number Diff line change
@@ -1,38 +1,74 @@
---
backend_name: Snowflake
backend_url: https://snowflake.com/
exports: ["PyArrow", "Parquet", "CSV", "Pandas"]
memtable_impl: native
description: "Install Ibis in Python and connect to Snowflake for working with in a pandas-like dataframe library"
---

# Snowflake

{% include 'backends/badges.md' %}
[https://www.snowflake.com](https://www.snowflake.com)

!!! experimental "Introduced in v4.0"
![](https://img.shields.io/badge/memtables-native-green?style=flat-square) ![](https://img.shields.io/badge/inputs-Snowflake tables | CSV | JSON | Parquet-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-Snowflake tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

The Snowflake backend is experimental and is subject to backwards incompatible changes.
## Install

{% include 'backends/memtable-template.md' %}
Install Ibis and dependencies for the Snowflake backend:

## Install
::: {.panel-tabset}

## `pip`

Install with the `snowflake` extra:

Install `ibis` and dependencies for the Snowflake backend:
```{.bash}
pip install 'ibis-framework[snowflake]'
```

And connect:

=== "pip"
```{.python}
import ibis
con = ibis.snowflake.connect() # <1>
```

```sh
pip install 'ibis-framework[snowflake]'
```
1. Adjust connection parameters as needed.

{% for mgr in ["conda", "mamba"] %}
=== "{{ mgr }}"
## `conda`

Install for Snowflake:

```{.bash}
conda install -c conda-forge ibis-snowflake
```

And connect:

```{.python}
import ibis
con = ibis.snowflake.connect() # <1>
```

```sh
{{ mgr }} install -c conda-forge ibis-snowflake
```
1. Adjust connection parameters as needed.

{% endfor %}
## `mamba`

Install for Snowflake:

```{.bash}
mamba install -c conda-forge ibis-snowflake
```

And connect:

```{.python}
import ibis
con = ibis.snowflake.connect() # <1>
```

1. Adjust connection parameters as needed.

:::

## Connect

Expand All @@ -47,17 +83,19 @@ con = ibis.snowflake.connect(
)
```

<!-- prettier-ignore-start -->
!!! info "`ibis.snowflake.connect` is a thin wrapper around [`ibis.backends.snowflake.Backend.do_connect`][ibis.backends.snowflake.Backend.do_connect]."
<!-- prettier-ignore-end -->
::: {.callout-note}
`ibis.snowflake.connect` is a thin wrapper around [`ibis.backends.snowflake.Backend.do_connect`](#ibis.backends.snowflake.Backend.do_connect).
:::

### Connection Parameters

<!-- prettier-ignore-start -->
::: ibis.backends.snowflake.Backend.do_connect
options:
heading_level: 4
<!-- prettier-ignore-end -->
```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("snowflake")
```

### `ibis.connect` URL format

Expand Down Expand Up @@ -102,7 +140,9 @@ A [Snowflake account
identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier#format-1-preferred-account-name-in-your-organization)
consists of an organization ID and a user ID, separated by a hyphen.

!!! info "This user ID is not the same as the username you log in with."
::: {.callout-note}
This user ID is not the same as the username you log in with.
:::

To find your organization ID and user ID, log in to the Snowflake web app, then
click on the text just to the right of the Snowflake logo (in the
Expand All @@ -122,3 +162,10 @@ available databases and schema in the "Data" sidebar item in the Snowflake web
app.

![Snowflake Database](./images/snowflake_database.png)

```{python}
#| echo: false
BACKEND = "Snowflake"
```

{{< include ./_templates/api.qmd >}}
75 changes: 0 additions & 75 deletions docs/backends/sqlite.md

This file was deleted.

120 changes: 120 additions & 0 deletions docs/backends/sqlite.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# SQLite

[https://www.sqlite.org](https://www.sqlite.org)

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-SQLite tables | CSV | JSON | pandas | Parquet | Postgres | PyArrow-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-SQLite tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the SQLite backend:

::: {.panel-tabset}

## `pip`

Install with the `sqlite` extra:

```{.bash}
pip install 'ibis-framework[sqlite]'
```

And connect:

```{.python}
import ibis
con = ibis.sqlite.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for SQLite:

```{.bash}
conda install -c conda-forge ibis-sqlite
```

And connect:

```{.python}
import ibis
con = ibis.sqlite.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for SQLite:

```{.bash}
mamba install -c conda-forge ibis-sqlite
```

And connect:

```{.python}
import ibis
con = ibis.sqlite.connect() # <1>
```

1. Adjust connection parameters as needed.

:::


## Connect

### `ibis.sqlite.connect`

Use an ephemeral, in-memory database.

```python
con = ibis.sqlite.connect()
```

Connect to, or create, a local SQLite file

```python
con = ibis.sqlite.connect("mydb.sqlite")
```

::: {.callout-note}
`ibis.sqlite.connect` is a thin wrapper around [`ibis.backends.sqlite.Backend.do_connect`](#ibis.backends.sqlite.Backend.do_connect).
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("sqlite")
```

### `ibis.connect` URL format

In addition to `ibis.sqlite.connect`, you can also connect to SQLite by
passing a properly formatted SQLite connection URL to `ibis.connect`:

```python
con = ibis.connect("sqlite:///path/to/local/file")
```

The URL can be `sqlite://` which will connect to an ephemeral in-memory database:

```python
con = ibis.connect("sqlite://")
```

```{python}
#| echo: false
BACKEND = "SQLite"
```

{{< include ./_templates/api.qmd >}}
78 changes: 0 additions & 78 deletions docs/backends/template.md

This file was deleted.

62 changes: 0 additions & 62 deletions docs/backends/trino.md

This file was deleted.

105 changes: 105 additions & 0 deletions docs/backends/trino.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Trino

[https://trino.io](https://trino.io)

{{< include /_callouts/experimental_backend.qmd >}}

![](https://img.shields.io/badge/memtables-fallback-yellow?style=flat-square) ![](https://img.shields.io/badge/inputs-Trino tables-blue?style=flat-square) ![](https://img.shields.io/badge/outputs-Trino tables | CSV | pandas | Parquet | PyArrow-orange?style=flat-square)

## Install

Install Ibis and dependencies for the Trino backend:

::: {.panel-tabset}

## `pip`

Install with the `trino` extra:

```{.bash}
pip install 'ibis-framework[trino]'
```

And connect:

```{.python}
import ibis
con = ibis.trino.connect() # <1>
```

1. Adjust connection parameters as needed.

## `conda`

Install for Trino:

```{.bash}
conda install -c conda-forge ibis-trino
```

And connect:

```{.python}
import ibis
con = ibis.trino.connect() # <1>
```

1. Adjust connection parameters as needed.

## `mamba`

Install for Trino:

```{.bash}
mamba install -c conda-forge ibis-trino
```

And connect:

```{.python}
import ibis
con = ibis.trino.connect() # <1>
```

1. Adjust connection parameters as needed.

:::


## Connect

### `ibis.trino.connect`

```python
con = ibis.trino.connect(
user="user",
password="password",
port=8080,
database="database",
schema="default",
)
```

::: {.callout-note}
`ibis.trino.connect` is a thin wrapper around [`ibis.backends.trino.Backend.do_connect`](#ibis.backends.trino.Backend.do_connect)."
:::

### Connection Parameters

```{python}
#| echo: false
#| output: asis
from _utils import render_do_connect
render_do_connect("trino")
```

```{python}
#| echo: false
BACKEND = "Trino"
```

{{< include ./_templates/api.qmd >}}
Loading