Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 63 additions & 17 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,11 @@ pub trait ForeignDataWrapper<E: Into<ErrorReport>> {
fn delete(&mut self, rowid: &Cell) -> Result<(), E>;
fn end_modify(&mut self) -> Result<(), E>;

// Optional methods for aggregate pushdown
fn supported_aggregates(&self) -> Vec<AggregateKind>;
fn supports_group_by(&self) -> bool;
fn begin_aggregate_scan(&mut self, aggregates: &[Aggregate], group_by: &[Column], quals: &[Qual], options: &HashMap<String, String>) -> Result<(), E>;

// Optional methods
fn re_scan(&mut self) -> Result<(), E>;
fn get_rel_size(...) -> Result<(i64, i32), E>;
Expand Down Expand Up @@ -89,23 +94,24 @@ pub struct MyFdw { ... }

### Native FDWs (wrappers/src/fdw/)

| FDW | Feature Flag | Supports Write |
|-----|--------------|----------------|
| BigQuery | `bigquery_fdw` | Yes |
| ClickHouse | `clickhouse_fdw` | Yes |
| Stripe | `stripe_fdw` | Yes |
| S3 Vectors | `s3vectors_fdw` | Yes |
| S3 | `s3_fdw` | No |
| Firebase | `firebase_fdw` | No |
| Airtable | `airtable_fdw` | No |
| Auth0 | `auth0_fdw` | No |
| AWS Cognito | `cognito_fdw` | No |
| DuckDB | `duckdb_fdw` | No |
| Apache Iceberg | `iceberg_fdw` | No |
| Logflare | `logflare_fdw` | No |
| Redis | `redis_fdw` | No |
| SQL Server | `mssql_fdw` | No |
| HelloWorld | `helloworld_fdw` | No (demo) |
| FDW | Feature Flag | Supports Write | Aggregate Pushdown |
|-----|--------------|----------------|--------------------|
| BigQuery | `bigquery_fdw` | Yes | Yes |
| ClickHouse | `clickhouse_fdw` | Yes | Yes |
| MySQL | `mysql_fdw` | Yes | Yes |
| Stripe | `stripe_fdw` | Yes | No |
| S3 Vectors | `s3vectors_fdw` | Yes | No |
| S3 | `s3_fdw` | No | No |
| Firebase | `firebase_fdw` | No | No |
| Airtable | `airtable_fdw` | No | No |
| Auth0 | `auth0_fdw` | No | No |
| AWS Cognito | `cognito_fdw` | No | No |
| DuckDB | `duckdb_fdw` | No | No |
| Apache Iceberg | `iceberg_fdw` | No | No |
| Logflare | `logflare_fdw` | No | No |
| Redis | `redis_fdw` | No | No |
| SQL Server | `mssql_fdw` | No | Yes |
| HelloWorld | `helloworld_fdw` | No (demo) | No |

### Wasm FDWs (wasm-wrappers/fdw/)

Expand Down Expand Up @@ -292,6 +298,46 @@ FDWs receive pushdown hints through `begin_scan`:

Use `Qual::deparse()` to convert to SQL-like strings.

### Aggregate Pushdown

FDWs can push `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` (with optional `GROUP BY`) down to the remote source by implementing three optional trait methods:

```rust
fn supported_aggregates(&self) -> Vec<AggregateKind> {
vec![
AggregateKind::Count,
AggregateKind::CountColumn,
AggregateKind::Sum,
AggregateKind::Avg,
AggregateKind::Min,
AggregateKind::Max,
]
}

fn supports_group_by(&self) -> bool { true }

fn begin_aggregate_scan(
&mut self,
aggregates: &[Aggregate],
group_by: &[Column],
quals: &[Qual],
options: &HashMap<String, String>,
) -> Result<(), E> {
// Build remote SQL, set self.tgt_cols (group-by columns first,
// then aggregate aliases), then initiate streaming.
Ok(())
}
```

Key types from `supabase_wrappers::prelude`:
- `AggregateKind`: enum of `Count | CountColumn | Sum | Avg | Min | Max`
- `Aggregate`: holds `kind`, `column: Option<Column>`, `distinct: bool`, `alias: String`, `type_oid`
- `Aggregate::deparse_with_alias()` renders `FUNC(col) AS alias` (no remote quoting — apply your own if needed)

`iter_scan` is reused unchanged; `tgt_cols` must match the SELECT order exactly so column-name lookup works.

FDWs that support aggregate pushdown: BigQuery, ClickHouse, MySQL.

## PostgreSQL Version Support

Supported via feature flags: `pg13`, `pg14`, `pg15` (default), `pg16`, `pg17`, `pg18`
Expand Down
97 changes: 96 additions & 1 deletion docs/catalog/mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,14 +133,45 @@ create foreign table mysql.my_table (

#### Notes

- Supports `where`, `order by` and `limit` clause pushdown
- Supports `where`, `order by`, `limit` and aggregate clause pushdown
Comment thread
burmecia marked this conversation as resolved.
- Data is streamed row-by-row from MySQL, making it suitable for large result sets
- When using `rowid_column`, it must be specified for data modification operations

## Query Pushdown Support

This FDW supports `where`, `order by` and `limit` clause pushdown.

### Aggregate Pushdown

The FDW pushes common aggregate queries down to MySQL so the aggregation
runs remotely and only the final result rows are transferred to Postgres. This
is much faster than fetching every row and aggregating locally, especially
over large tables.

**Supported aggregates** — `count(*)`, `count(col)`, `count(distinct col)`,
`sum(col)`, `avg(col)`, `min(col)`, `max(col)`.

**Supported shapes** — scalar aggregates, `group by` over plain columns, with
or without a `where` clause. Pushdown also works when the foreign `table`
option is a sub-query.

```sql
-- All of these run as a single aggregate query on MySQL:
select count(*) from mysql.my_table;
select status, sum(amount) from mysql.my_table group by status;
select count(distinct name) from mysql.my_table where id = 1;
```

**Cases that are not pushed down** — the query still returns the correct
result, but the aggregation happens in Postgres after fetching the rows:

- The query has a `having` clause
- The aggregate has a `filter (where …)` clause
- A `distinct` modifier is used on anything other than `count`
- The aggregate's argument is not a plain column (for example `sum(a + 1)`)
- A `group by` item is not a plain column (for example `group by id + 1`)
- The aggregate function is not in the list above (for example `stddev`, `group_concat`)

## Import Foreign Schema

This FDW supports [`import foreign schema`](https://www.postgresql.org/docs/current/sql-importforeignschema.html) to automatically create foreign table definitions by reading the MySQL table structure from `information_schema`.
Expand Down Expand Up @@ -282,6 +313,70 @@ delete from mysql.people
where id = 2;
```

### Aggregate Query Examples

These examples assume an `orders` table in MySQL and a matching foreign
table on Postgres:

```sql
-- Run on MySQL
create table orders (
id bigint primary key auto_increment,
user_id bigint not null,
amount decimal(12,2) not null,
status varchar(50) not null
);

insert into orders (user_id, amount, status) values
(1, 100.00, 'paid'),
(1, 50.00, 'paid'),
(2, 200.00, 'pending'),
(2, 75.00, 'paid'),
(3, 300.00, 'paid');
```

```sql
-- Foreign table on Postgres
create foreign table mysql.orders (
id bigint,
user_id bigint,
amount numeric,
status text
)
server mysql_server
options (
table 'orders'
);
```

Each query below runs a single aggregate query against MySQL and returns
just the result rows:

```sql
-- Total order count
select count(*) from mysql.orders;

-- Total revenue from paid orders
select sum(amount) from mysql.orders where status = 'paid';

-- Per-user order count and revenue
select user_id, count(*) as orders, sum(amount) as revenue
from mysql.orders
group by user_id
order by user_id;

-- Smallest and largest order
select min(amount), max(amount) from mysql.orders;

-- Number of distinct users who placed an order
select count(distinct user_id) from mysql.orders;

-- Average order value per status
select status, avg(amount) as avg_amount
from mysql.orders
group by status;
```

### Import foreign schema example

This example imports all tables from a MySQL database automatically:
Expand Down
1 change: 1 addition & 0 deletions wrappers/src/fdw/mysql_fdw/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ This is a foreign data wrapper for [MySQL](https://www.mysql.com/). It is develo

| Version | Date | Notes |
| ------- | ---------- | ---------------------------------------------- |
| 0.1.1 | 2026-04-28 | Added aggregate pushdown support |
| 0.1.0 | 2026-03-27 | Initial version |
Loading
Loading