Skip to content

Commit

Permalink
docs: add API docs for operations (#9233)
Browse files Browse the repository at this point in the history
Document operations and add links in support matrix.

**DISCLAIMER**: Many of the docstrings here were generated with GitHub
Copilot. Each one was reviewed individually by me for correctness.
  • Loading branch information
cpcloud committed May 28, 2024
1 parent c4324f5 commit 11e0530
Show file tree
Hide file tree
Showing 28 changed files with 686 additions and 285 deletions.
28 changes: 27 additions & 1 deletion docs/_quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ website:
collapse-level: 2
contents:
- auto: backends/*.qmd
- support_matrix.qmd
- auto: backends/support
- id: how-to
title: "How-to"
style: "docked"
Expand Down Expand Up @@ -506,6 +506,32 @@ quartodoc:
- name: NumericValue.point
package: ibis.expr.types.numeric

- kind: page
path: operations
package: ibis.expr.operations
summary:
name: Operations
desc: Low level operation classes. Subject to change in non-major releases.
contents:
- analytic
- arrays
- generic
- geospatial
- histograms
- json
- logical
- maps
- numeric
- reductions
- relations
- sortkeys
- strings
- structs
- subqueries
- temporal
- temporal_windows
- udf
- window
- kind: page
summary:
name: Column selectors
Expand Down
130 changes: 130 additions & 0 deletions docs/backends/support/matrix.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
---
title: "Operation support matrix"
format: dashboard
hide:
- toc
---

## {height=25%}

::: {.card title="Welcome to the operation support matrix!"}

This is a [Quarto dashboard](https://quarto.org/docs/dashboards/) that shows
the operations each backend supports.

Due to differences in SQL dialects and upstream support for different
operations in different backends, support for the full breadth of the Ibis API
varies.

::: {.callout-tip}
Backends with low coverage are good places to start contributing!

Each backend implements operations differently, but this is usually very
similar to other backends. If you want to start contributing to ibis, it's
a good idea to start by adding missing operations to backends that have low
operation coverage.
:::

:::

### {width=25%}

```{python}
#| content: valuebox
#| title: "Number of backends"
import ibis
dict(
value=len(ibis.util.backend_entry_points()),
color="info",
icon="signpost-split-fill",
)
```

### {width=25%}

```{python}
#| content: valuebox
#| title: "Number of SQL backends"
import importlib
from ibis.backends.sql import SQLBackend
sql_backends = sum(
issubclass(
importlib.import_module(f"ibis.backends.{entry_point.name}").Backend, SQLBackend
)
for entry_point in ibis.util.backend_entry_points()
)
assert sql_backends > 0
dict(value=sql_backends, color="green", icon="database")
```

## {height=70%}

```{python}
#| echo: false
import pandas as pd
import ibis
import ibis.expr.operations as ops
def make_support_matrix():
"""Construct the backend operation support matrix data."""
from ibis.backends.sql.compiler import ALL_OPERATIONS
support_matrix_ignored_operations = (ops.ScalarParameter,)
public_ops = ALL_OPERATIONS.difference(support_matrix_ignored_operations)
assert public_ops
support = {"Operation": [f"{op.__module__}.{op.__name__}" for op in public_ops]}
support.update(
(backend, list(map(getattr(ibis, backend).has_operation, public_ops)))
for backend in sorted(ep.name for ep in ibis.util.backend_entry_points())
)
def make_link(parts):
module, op = parts[-2:]
return f'<a href="./operations.html#ibis.expr.operations.{module}.{op}">{op}</a>'
support_matrix = (
pd.DataFrame(support)
.assign(splits=lambda df: df.Operation.str.findall("[a-zA-Z_][a-zA-Z_0-9]*"))
.assign(
Category=lambda df: df.splits.str[-2],
Operation=lambda df: df.splits.map(make_link),
)
.drop(["splits"], axis=1)
.set_index(["Category", "Operation"])
.sort_index()
)
all_visible_ops_count = len(support_matrix)
assert all_visible_ops_count
coverage = pd.Index(
support_matrix.sum()
.map(lambda n: f"{n} ({round(100 * n / all_visible_ops_count)}%)")
.T
)
support_matrix.columns = pd.MultiIndex.from_tuples(
list(zip(support_matrix.columns, coverage)), names=("Backend", "API coverage")
)
return support_matrix
```

```{python}
from itables import show
matrix = make_support_matrix()
show(
matrix.replace({True: "✔", False: "🚫"}),
ordering=False,
paging=False,
buttons=["copy", "excel", "csv"],
)
```
1 change: 1 addition & 0 deletions docs/backends/support/operations.qmd
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{{< include ../../reference/operations.qmd >}}
2 changes: 1 addition & 1 deletion docs/how-to/extending/unbound_expression.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Because Ibis separates the transformation logic from the execution engine, you
can easily reuse the written transformation for another backend. Here we use
Polars as an example, but you can do the same for any of Ibis' 20+ supported
backends as long as that particular backend supports the operations
(see [the operation support matrix](../../support_matrix.qmd)).
(see [the operation support matrix](../../backends/support/matrix.qmd)).

```{python}
pl = ibis.polars.connect()
Expand Down
52 changes: 0 additions & 52 deletions docs/support_matrix.py

This file was deleted.

74 changes: 0 additions & 74 deletions docs/support_matrix.qmd

This file was deleted.

2 changes: 1 addition & 1 deletion docs/why.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ Ibis already works well with machine learning libraries like:

{{< include ./_tabsets/install.qmd >}}

See the [backend support matrix](support_matrix.qmd) for details on operations
See the [backend support matrix](./backends/support/matrix.qmd) for details on operations
supported. [Open a feature
request](https://github.com/ibis-project/ibis/issues/new?assignees=&labels=feature&projects=&template=feature-request.yml&title=feat)
if you'd like to see support for an operation in a given backend. If the backend
Expand Down
9 changes: 5 additions & 4 deletions gen_redirects.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
"/api/expressions/top_level/": "/reference/expressions/top_level/",
"/api/schemas/": "/reference/schemas/",
"/api/selectors/": "/reference/selectors/",
"/backends/": "/support_matrix",
"/backends/": "/backends/support/matrix",
"/backends/BigQuery/": "/backends/bigquery/",
"/backends/Clickhouse/": "/backends/clickhouse/",
"/backends/Dask/": "/backends/dask/",
Expand All @@ -87,8 +87,8 @@
"/backends/SQLite/": "/backends/sqlite/",
"/backends/Snowflake/": "/backends/snowflake/",
"/backends/Trino/": "/backends/trino/",
"/backends/_support_matrix/": "/support_matrix",
"/backends/support_matrix": "/support_matrix",
"/backends/_support_matrix/": "/backends/support/matrix",
"/backends/support_matrix": "/backends/support/matrix",
"/blog": "/posts",
"/blog/Ibis-version-3.0.0-release/": "/posts/Ibis-version-3.0.0-release/",
"/blog/Ibis-version-3.1.0-release/": "/posts/Ibis-version-3.1.0-release/",
Expand Down Expand Up @@ -118,7 +118,7 @@
"/concept/design/": "/concepts/internals",
"/concept/why_ibis/": "/why",
"/docs/": "/",
"/docs/dev/backends/support_matrix/": "/support_matrix",
"/docs/dev/backends/support_matrix/": "/backends/support/matrix",
"/docs/dev/contribute/01_environment/": "/contribute/01_environment",
"/docs/dev/release_notes/": "/release_notes",
"/getting_started/": "/tutorial/getting_started/",
Expand Down Expand Up @@ -154,6 +154,7 @@
"/reference/expressions/tables/": "/reference/expressions-tables",
"/reference/expressions/timestamps/": "/reference/expression-temporal",
"/reference/expressions/top_level/": "/reference/top_level",
"/support_matrix": "/backends/support/matrix",
"/tutorial/": "/tutorials/getting_started/",
"/tutorial/getting_started/": "/tutorials/getting_started",
"/tutorial/ibis-for-dplyr-users/": "/tutorials/ibis-for-dplyr-users/",
Expand Down
Loading

0 comments on commit 11e0530

Please sign in to comment.