Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add API docs for operations #9233

Merged
merged 21 commits into from
May 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion docs/_quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ website:
collapse-level: 2
contents:
- auto: backends/*.qmd
- support_matrix.qmd
- auto: backends/support
- id: how-to
title: "How-to"
style: "docked"
Expand Down Expand Up @@ -506,6 +506,32 @@ quartodoc:
- name: NumericValue.point
package: ibis.expr.types.numeric

- kind: page
path: operations
package: ibis.expr.operations
summary:
name: Operations
desc: Low level operation classes. Subject to change in non-major releases.
contents:
- analytic
- arrays
- generic
- geospatial
- histograms
- json
- logical
- maps
- numeric
- reductions
- relations
- sortkeys
- strings
- structs
- subqueries
- temporal
- temporal_windows
- udf
- window
- kind: page
summary:
name: Column selectors
Expand Down
130 changes: 130 additions & 0 deletions docs/backends/support/matrix.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: "Operation support matrix"
format: dashboard
hide:
- toc
---

## {height=25%}

::: {.card title="Welcome to the operation support matrix!"}

This is a [Quarto dashboard](https://quarto.org/docs/dashboards/) that shows
the operations each backend supports.

Due to differences in SQL dialects and upstream support for different
operations in different backends, support for the full breadth of the Ibis API
varies.

::: {.callout-tip}
Backends with low coverage are good places to start contributing!

Each backend implements operations differently, but this is usually very
similar to other backends. If you want to start contributing to ibis, it's
a good idea to start by adding missing operations to backends that have low
operation coverage.
:::

:::

### {width=25%}

```{python}
#| content: valuebox
#| title: "Number of backends"
import ibis

dict(
value=len(ibis.util.backend_entry_points()),
color="info",
icon="signpost-split-fill",
)
```

### {width=25%}

```{python}
#| content: valuebox
#| title: "Number of SQL backends"
import importlib
from ibis.backends.sql import SQLBackend

sql_backends = sum(
issubclass(
importlib.import_module(f"ibis.backends.{entry_point.name}").Backend, SQLBackend
)
for entry_point in ibis.util.backend_entry_points()
)
assert sql_backends > 0
dict(value=sql_backends, color="green", icon="database")
```

## {height=70%}

```{python}
#| echo: false
import pandas as pd

import ibis
import ibis.expr.operations as ops


def make_support_matrix():
"""Construct the backend operation support matrix data."""

from ibis.backends.sql.compiler import ALL_OPERATIONS

support_matrix_ignored_operations = (ops.ScalarParameter,)

public_ops = ALL_OPERATIONS.difference(support_matrix_ignored_operations)

assert public_ops

support = {"Operation": [f"{op.__module__}.{op.__name__}" for op in public_ops]}
support.update(
(backend, list(map(getattr(ibis, backend).has_operation, public_ops)))
for backend in sorted(ep.name for ep in ibis.util.backend_entry_points())
)

def make_link(parts):
module, op = parts[-2:]
return f'<a href="./operations.html#ibis.expr.operations.{module}.{op}">{op}</a>'

support_matrix = (
pd.DataFrame(support)
.assign(splits=lambda df: df.Operation.str.findall("[a-zA-Z_][a-zA-Z_0-9]*"))
.assign(
Category=lambda df: df.splits.str[-2],
Operation=lambda df: df.splits.map(make_link),
)
.drop(["splits"], axis=1)
.set_index(["Category", "Operation"])
.sort_index()
)
all_visible_ops_count = len(support_matrix)
assert all_visible_ops_count

coverage = pd.Index(
support_matrix.sum()
.map(lambda n: f"{n} ({round(100 * n / all_visible_ops_count)}%)")
.T
)
support_matrix.columns = pd.MultiIndex.from_tuples(
list(zip(support_matrix.columns, coverage)), names=("Backend", "API coverage")
)
return support_matrix
```

```{python}
from itables import show


matrix = make_support_matrix()

show(
matrix.replace({True: "✔", False: "🚫"}),
ordering=False,
paging=False,
buttons=["copy", "excel", "csv"],
)
```
1 change: 1 addition & 0 deletions docs/backends/support/operations.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{< include ../../reference/operations.qmd >}}
2 changes: 1 addition & 1 deletion docs/how-to/extending/unbound_expression.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Because Ibis separates the transformation logic from the execution engine, you
can easily reuse the written transformation for another backend. Here we use
Polars as an example, but you can do the same for any of Ibis' 20+ supported
backends as long as that particular backend supports the operations
(see [the operation support matrix](../../support_matrix.qmd)).
(see [the operation support matrix](../../backends/support/matrix.qmd)).

```{python}
pl = ibis.polars.connect()
Expand Down
52 changes: 0 additions & 52 deletions docs/support_matrix.py

This file was deleted.

74 changes: 0 additions & 74 deletions docs/support_matrix.qmd

This file was deleted.

2 changes: 1 addition & 1 deletion docs/why.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ Ibis already works well with machine learning libraries like:

{{< include ./_tabsets/install.qmd >}}

See the [backend support matrix](support_matrix.qmd) for details on operations
See the [backend support matrix](./backends/support/matrix.qmd) for details on operations
supported. [Open a feature
request](https://github.com/ibis-project/ibis/issues/new?assignees=&labels=feature&projects=&template=feature-request.yml&title=feat)
if you'd like to see support for an operation in a given backend. If the backend
Expand Down
9 changes: 5 additions & 4 deletions gen_redirects.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"/api/expressions/top_level/": "/reference/expressions/top_level/",
"/api/schemas/": "/reference/schemas/",
"/api/selectors/": "/reference/selectors/",
"/backends/": "/support_matrix",
"/backends/": "/backends/support/matrix",
"/backends/BigQuery/": "/backends/bigquery/",
"/backends/Clickhouse/": "/backends/clickhouse/",
"/backends/Dask/": "/backends/dask/",
Expand All @@ -87,8 +87,8 @@
"/backends/SQLite/": "/backends/sqlite/",
"/backends/Snowflake/": "/backends/snowflake/",
"/backends/Trino/": "/backends/trino/",
"/backends/_support_matrix/": "/support_matrix",
"/backends/support_matrix": "/support_matrix",
"/backends/_support_matrix/": "/backends/support/matrix",
"/backends/support_matrix": "/backends/support/matrix",
"/blog": "/posts",
"/blog/Ibis-version-3.0.0-release/": "/posts/Ibis-version-3.0.0-release/",
"/blog/Ibis-version-3.1.0-release/": "/posts/Ibis-version-3.1.0-release/",
Expand Down Expand Up @@ -118,7 +118,7 @@
"/concept/design/": "/concepts/internals",
"/concept/why_ibis/": "/why",
"/docs/": "/",
"/docs/dev/backends/support_matrix/": "/support_matrix",
"/docs/dev/backends/support_matrix/": "/backends/support/matrix",
"/docs/dev/contribute/01_environment/": "/contribute/01_environment",
"/docs/dev/release_notes/": "/release_notes",
"/getting_started/": "/tutorial/getting_started/",
Expand Down Expand Up @@ -154,6 +154,7 @@
"/reference/expressions/tables/": "/reference/expressions-tables",
"/reference/expressions/timestamps/": "/reference/expression-temporal",
"/reference/expressions/top_level/": "/reference/top_level",
"/support_matrix": "/backends/support/matrix",
"/tutorial/": "/tutorials/getting_started/",
"/tutorial/getting_started/": "/tutorials/getting_started",
"/tutorial/ibis-for-dplyr-users/": "/tutorials/ibis-for-dplyr-users/",
Expand Down
Loading
Loading