Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,23 @@

### Added

- ClickHouse support, as two readers behind the new default-on `clickhouse` and
`chdb` features. `clickhouse://[user[:password]@]host[:port][/database]` (or
`clickhouses://` for TLS) talks to a server's HTTP interface and exchanges data
as Arrow IPC streams, so no driver installation is needed; unrecognised URI
query parameters are forwarded as ClickHouse settings, and the host, user and
password fall back to the `CLICKHOUSE_HOST`/`CLICKHOUSE_USER`/`CLICKHOUSE_PASSWORD`
environment variables. `chdb://[path]` runs the embedded chDB engine
in-process (libchdb is loaded at runtime, so the build has no new native
dependency), and `chdb` is also a new in-memory backend for the caching layer
(`chdb+<primary>://…`, `--cache chdb`), so a ClickHouse setup needs no other
database engine. Stat transforms run on the server through session-scoped
temporary tables; a read-only account such as `play.clickhouse.com` is
detected on connect and its intermediate tables are kept in an embedded chDB
cache automatically. `DateTime`, `Enum`, `UUID`, IP address, `FixedString`,
`Decimal` and wide-integer columns are converted server-side to types the plot
pipeline understands. The Jupyter kernel and Positron connections pane
recognise both schemes. ClickHouse 26.8 or newer is required.
- New caching layer that wraps any `Reader` with an in-memory, writeable cache
backend (currently duckdb or sqlite), making write-constrained databases
usable and avoiding repeated remote reads during interactive iteration.
Expand Down
29 changes: 29 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ duckdb = { version = "~1.10502", features = ["bundled", "vtab-arrow"] }
parquet = { version = "58", default-features = false, features = ["arrow", "snap"] }
bytes = "1"
rusqlite = { version = "0.38", features = ["bundled", "chrono", "load_extension"] }
# ClickHouse HTTP interface (also fetches the AI skill at CLI build time)
ureq = "3"

# ODBC
toml_edit = "0.22"
Expand Down
54 changes: 54 additions & 0 deletions doc/get_started/tooling/cli.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,60 @@ col_a, col_b, col_c
12.5, 29.48, gamma
```

### ClickHouse

ggsql connects to [ClickHouse](https://clickhouse.com) 26.8 or newer over its HTTP interface, so no driver needs to be installed. The connection string is

```
clickhouse://[user[:password]@]host[:port][/database][?param=value&...]
```

The port defaults to 8123. Use `clickhouses://` (or add `?secure=1`) for a TLS connection, where the port defaults to 8443. Any other query parameter is forwarded to the server with every request, so ClickHouse settings and HTTP parameters such as `session_timezone`, `max_threads` or `session_timeout` can be set per connection. If the connection string does not name them, the host, user and password are taken from the `CLICKHOUSE_HOST`, `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` environment variables, so a password need not appear on the command line.

```bash
$ ggsql exec --reader clickhouse://localhost:8123/default \
"SELECT number AS x, number * number AS y FROM numbers(10) VISUALISE x, y DRAW line"
```

Each reader holds one ClickHouse session, and ggsql stores intermediate results in temporary tables inside that session, which is what lets stat transforms such as binning and quantiles run on the server. A read-only account without the `CREATE TEMPORARY TABLE` privilege is detected when connecting; ggsql then keeps the intermediate tables in an embedded [chDB](#chdb) engine and sends only your own query to the server. The public playground works this way:

```bash
$ ggsql exec --reader clickhouses://explorer@play.clickhouse.com:443 \
"SELECT toStartOfMonth(created_at) AS month, count() AS events
FROM github_events WHERE created_at >= '2024-01-01'
GROUP BY month ORDER BY month
VISUALISE month AS x, events AS y DRAW line"
```

The same arrangement can be requested explicitly with `chdb+clickhouse://…` (or `--cache chdb`). It also memoizes the server's answers, so re-running a query while iterating on the plot does not hit the server again until the cache entry expires.

ClickHouse `DateTime` and `DateTime64` values are shown in UTC. `Enum`, `UUID`, `IPv4`/`IPv6`, `FixedString` and 128/256-bit integer columns arrive as strings, `Decimal` columns as floating-point numbers, and arrays, tuples and maps as their text form.

### chDB {#chdb}

[chDB](https://clickhouse.com/chdb) is ClickHouse as an in-process library. ggsql uses it in two roles: as a database of its own, and as the in-memory cache in front of a ClickHouse server. Either way it needs `libchdb` on the machine; install it with

```bash
curl -sL https://lib.chdb.io | bash
```

or point the `GGSQL_CHDB_LIBRARY` environment variable at a `libchdb.so` (or `.dylib`) you downloaded yourself. Nothing else changes: the same `ggsql` binary works with or without the library, and only `chdb://` connections need it.

```
chdb:// in-memory engine (also chdb://memory)
chdb:///path/to/directory state persists under the directory
chdb://…?key=value extra engine arguments, passed as --key=value
```

Everything ClickHouse can read, chDB can read too, so a chDB connection is a quick way to plot local files:

```bash
$ ggsql exec --reader chdb://memory \
"SELECT * FROM 'trips.parquet' VISUALISE fare AS x DRAW histogram"
```

chDB keeps a single connection per process, so all `chdb://` connections in one process share one engine and one path.

## Output format

`ggsql exec` and `ggsql run` render with the writer named by `--writer` (short `-w`), defaulting to `--writer vegalite` (the Vega-Lite JSON above). A build that includes the optional `png` writer can also render straight to a PNG image with `--writer png`, which needs a GPU adapter available where it runs.
Expand Down
4 changes: 2 additions & 2 deletions ggsql-cli/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The binary name is `ggsql` (not `ggsql-cli`) — that's what release artifacts a

Only public `ggsql::*` API is used (`reader`, `writer`, `validate`, `parser`, `VERSION`) — this crate has no awareness of internal modules.

`exec`/`run` build their reader via the library factory `ggsql::reader::connection::reader_from_uri`. They accept an in-memory caching layer (off by default) selected either by the composite connection scheme `<cache>+<primary>://…` (e.g. `duckdb+odbc://…`) or the `--cache <duckdb|sqlite>` flag; the two cannot be combined.
`exec`/`run` build their reader via the library factory `ggsql::reader::connection::reader_from_uri`. They accept an in-memory caching layer (off by default) selected either by the composite connection scheme `<cache>+<primary>://…` (e.g. `duckdb+odbc://…`) or the `--cache <duckdb|sqlite|chdb>` flag; the two cannot be combined.

`exec` and `run` share a `WriterSpec { name, options }`: `--writer` names the writer and repeated `--writer-option key=value` flags (short `-D`, visible alias `--writer-options`, several settings per flag when separated by `;`) become a `ggsql::writer::WriterOptions`, parsed up front in `main` so a malformed pair fails before any SQL runs. The two travel together down `cmd_exec` → `exec_with_reader` → `render_spec`, which dispatches on the name and hands the options to `Writer::from_options`. Adding a setting to a writer therefore needs no CLI change; which keys exist is the writer's business, and an unknown one is its error to report. User-facing keys are documented in [`/doc/get_started/tooling/cli.qmd`](../doc/get_started/tooling/cli.qmd).

Expand All @@ -59,7 +59,7 @@ The macOS codesign step uses [`/entitlements.plist`](../entitlements.plist) at t
## Features

```toml
default = ["duckdb", "sqlite", "vegalite", "ipc", "parquet", "builtin-data", "odbc"]
default = ["duckdb", "sqlite", "clickhouse", "chdb", "vegalite", "parquet", "builtin-data", "odbc"]
```

Each feature passes through to `ggsql/<feature>`. The `vegalite` flag also gates the writer-rendering path in `main.rs` via `#[cfg(feature = "vegalite")]`.
Expand Down
8 changes: 5 additions & 3 deletions ggsql-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -41,18 +41,20 @@ termimad = "0.31"

[build-dependencies]
regex.workspace = true
ureq = "3"
ureq.workspace = true

[features]
default = ["duckdb", "sqlite", "vegalite", "parquet", "builtin-data", "odbc"]
default = ["duckdb", "sqlite", "clickhouse", "chdb", "vegalite", "parquet", "builtin-data", "odbc"]
duckdb = ["ggsql/duckdb"]
parquet = ["ggsql/parquet"]
sqlite = ["ggsql/sqlite"]
odbc = ["ggsql/odbc"]
clickhouse = ["ggsql/clickhouse"]
chdb = ["ggsql/chdb"]
vegalite = ["ggsql/vegalite"]
png = ["ggsql/png"]
builtin-data = ["ggsql/builtin-data"]
all-readers = ["duckdb", "sqlite", "odbc"]
all-readers = ["duckdb", "sqlite", "odbc", "clickhouse", "chdb"]

# cargo-packager configuration for cross-platform installers
[package.metadata.packager]
Expand Down
10 changes: 5 additions & 5 deletions ggsql-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,11 @@ pub enum Commands {
/// The ggsql query to execute
query: String,

/// Data source connection string (duckdb://, sqlite://, odbc://)
/// Data source connection string (duckdb://, sqlite://, odbc://, clickhouse://)
#[arg(short, long, default_value = "duckdb://memory")]
reader: String,

/// In-memory cache backend wrapping the reader (duckdb, sqlite). Off by default.
/// In-memory cache backend wrapping the reader (duckdb, sqlite, chdb). Off by default.
#[arg(long)]
cache: Option<String>,

Expand Down Expand Up @@ -104,11 +104,11 @@ pub enum Commands {
/// Path to .sql file containing ggsql query
file: PathBuf,

/// Data source connection string (duckdb://, sqlite://, odbc://)
/// Data source connection string (duckdb://, sqlite://, odbc://, clickhouse://)
#[arg(short, long, default_value = "duckdb://memory")]
reader: String,

/// In-memory cache backend wrapping the reader (duckdb, sqlite). Off by default.
/// In-memory cache backend wrapping the reader (duckdb, sqlite, chdb). Off by default.
#[arg(long)]
cache: Option<String>,

Expand Down Expand Up @@ -154,7 +154,7 @@ pub enum Commands {
/// The ggsql query to validate
query: String,

/// Data source connection string for column validation (duckdb://, sqlite://, polars://)
/// Data source connection string for column validation (duckdb://, sqlite://, odbc://, clickhouse://)
#[arg(short, long)]
reader: Option<String>,
},
Expand Down
4 changes: 3 additions & 1 deletion ggsql-jupyter/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,12 @@ uuid = { version = "1.0", features = ["v4"] }

[features]
default = ["all-readers"]
all-readers = ["sqlite", "odbc", "duckdb"]
all-readers = ["sqlite", "odbc", "duckdb", "clickhouse", "chdb"]
odbc = ["ggsql/odbc"]
sqlite = ["ggsql/sqlite"]
duckdb = ["ggsql/duckdb"]
clickhouse = ["ggsql/clickhouse"]
chdb = ["ggsql/chdb"]

[dev-dependencies]
tempfile = "3.8"
30 changes: 30 additions & 0 deletions ggsql-jupyter/src/executor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,33 @@ pub fn display_name_for_uri(uri: &str) -> String {
}
return "ODBC".to_string();
}
if uri.starts_with("clickhouse://") || uri.starts_with("clickhouses://") {
return format!("ClickHouse ({})", clickhouse_host(uri));
}
if let Some(rest) = uri.strip_prefix("chdb://") {
let path = rest.split('?').next().unwrap_or(rest);
if path.is_empty() || path == "memory" || path == ":memory:" {
return "chDB (memory)".to_string();
}
return format!("chDB ({path})");
}
uri.to_string()
}

/// The `host[:port]` part of a `clickhouse://` / `clickhouses://` URI, without
/// credentials, database or parameters.
fn clickhouse_host(uri: &str) -> String {
let rest = uri.split_once("://").map(|(_, r)| r).unwrap_or(uri);
let rest = rest.split('?').next().unwrap_or(rest);
let rest = rest.rsplit_once('@').map(|(_, h)| h).unwrap_or(rest);
let host = rest.split('/').next().unwrap_or(rest);
if host.is_empty() {
"localhost".to_string()
} else {
host.to_string()
}
}

/// Detect the database type name from a connection URI (e.g. "DuckDB", "Snowflake").
pub fn type_name_for_uri(uri: &str) -> String {
if uri.starts_with("duckdb://") {
Expand All @@ -63,6 +87,12 @@ pub fn type_name_for_uri(uri: &str) -> String {
if uri.starts_with("sqlite://") {
return "SQLite".to_string();
}
if uri.starts_with("clickhouse://") || uri.starts_with("clickhouses://") {
return "ClickHouse".to_string();
}
if uri.starts_with("chdb://") {
return "chDB".to_string();
}
if let Some(odbc) = uri.strip_prefix("odbc://") {
if let Some(driver) = extract_odbc_value(odbc, "driver") {
let lower = driver.to_lowercase();
Expand Down
101 changes: 101 additions & 0 deletions ggsql-vscode/src/connections.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ export function createConnectionDrivers(
return [
createDuckDBDriver(positronApi),
createSQLiteDriver(positronApi),
createClickHouseDriver(positronApi),
createChdbDriver(positronApi),
createSnowflakeDefaultDriver(positronApi),
createSnowflakePasswordDriver(positronApi),
createSnowflakeSSODriver(positronApi),
Expand Down Expand Up @@ -411,6 +413,105 @@ function createSnowflakePATDriver(
};
}

// ============================================================================
// ClickHouse
// ============================================================================

/**
* ClickHouse connection driver (HTTP interface).
*
* Inputs: host (required), port, database, user, password, TLS. Produces
* `clickhouse://` or `clickhouses://`. The password is omitted from the
* generated code when left blank so it can come from `CLICKHOUSE_PASSWORD`.
*/
function createClickHouseDriver(
positronApi: PositronApi
): positron.ConnectionsDriver {
return {
driverId: 'ggsql-clickhouse',
metadata: {
languageId: 'ggsql',
name: 'ClickHouse',
inputs: [
{ id: 'host', label: 'Host', type: 'string', value: 'localhost' },
{ id: 'port', label: 'Port', type: 'number', value: '8123' },
{ id: 'database', label: 'Database', type: 'string', value: '' },
{ id: 'user', label: 'User', type: 'string', value: 'default' },
{ id: 'password', label: 'Password', type: 'string', value: '' },
{
id: 'secure',
label: 'Use TLS (clickhouses://)',
type: 'option',
options: [
{ identifier: 'no', title: 'No' },
{ identifier: 'yes', title: 'Yes' },
],
value: 'no',
},
],
} as ConnectionsDriverMetadata,
generateCode: (inputs) => {
const get = (id: string) =>
inputs.find((i) => i.id === id)?.value?.trim() || '';
const scheme = get('secure') === 'yes' ? 'clickhouses' : 'clickhouse';
const user = get('user');
const password = get('password');
const auth = user
? encodeURIComponent(user) +
(password ? `:${encodeURIComponent(password)}` : '') +
'@'
: '';
const host = get('host') || 'localhost';
const port = get('port') ? `:${get('port')}` : '';
const database = get('database') ? `/${encodeURIComponent(get('database'))}` : '';
return `-- @connect: ${scheme}://${auth}${host}${port}${database}`;
},
connect: async (code: string) => {
await positronApi.runtime.executeCode('ggsql', code, false);
},
};
}

// ============================================================================
// chDB (embedded ClickHouse)
// ============================================================================

/**
* chDB connection driver.
*
* Inputs: data directory (optional; blank = in-memory engine).
*/
function createChdbDriver(
positronApi: PositronApi
): positron.ConnectionsDriver {
return {
driverId: 'ggsql-chdb',
metadata: {
languageId: 'ggsql',
name: 'chDB',
description: 'Embedded ClickHouse (libchdb)',
inputs: [
{
id: 'path',
label: 'Data directory (blank for in-memory)',
type: 'string',
value: '',
},
],
} as ConnectionsDriverMetadata,
generateCode: (inputs) => {
const path = inputs.find((i) => i.id === 'path')?.value?.trim();
if (!path) {
return '-- @connect: chdb://memory';
}
return `-- @connect: chdb://${path}`;
},
connect: async (code: string) => {
await positronApi.runtime.executeCode('ggsql', code, false);
},
};
}

// ============================================================================
// Generic ODBC
// ============================================================================
Expand Down
Loading