diff --git a/spiceaidocs/docs/data-connectors/https.md b/spiceaidocs/docs/data-connectors/https.md new file mode 100644 index 00000000..1191e2aa --- /dev/null +++ b/spiceaidocs/docs/data-connectors/https.md @@ -0,0 +1,43 @@ +--- +title: 'HTTP(s) Data Connector' +sidebar_label: 'HTTP(s) Data Connector' +description: 'HTTP(s) Data Connector Documentation' +pagination_prev: null +--- + +The HTTP(s) Data Connector enables federated SQL query against a variety of tabular formatted (e.g. Parquet/CSV) files stored at a HTTP endpoint. + +The connector supports HTTP authentication via either `param` and/or `secrets`. + +### Parameters + - `http_port`: Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively. + - `http_username`: Optional. Username to provide connection for HTTP basic authentication. Default: None. + - `http_password`: Optional. Password to provide connection for HTTP basic authentication. Default: None. + - `http_password_key`: Key of the secret that contains the value to use for `http_password`. Default: None. + +### Examples +```yaml +datasets: + - from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet + name: laion_freesound + + - from: http://static_username@localhost:3001/report.csv + name: local_report + params: + http_password: BadPa$5w0rd +``` + +To use a secret for the HTTP password, for example, by env variable +```yaml +datasets: + - from: http://static_username@localhost/report.csv + name: local_report + params: + http_password_key: local_password + http_port: 3001 +``` + +With the associated secret set, for example: +```shell +export SPICE_SECRET_HTTP_LOCAL_PASSWORD="BadPa$5w0rd" +``` diff --git a/spiceaidocs/docs/data-connectors/index.md b/spiceaidocs/docs/data-connectors/index.md index 58c2633e..97080e89 100644 --- a/spiceaidocs/docs/data-connectors/index.md +++ b/spiceaidocs/docs/data-connectors/index.md @@ -11,21 +11,22 @@ Data Connectors provide connections to databases, data warehouses, and data lake Currently supported Data Connectors include: -| Name | Description | Status | Protocol/Format | Refresh Modes | Supports Inserts | -| ------------ | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- | -| `databricks` | Databricks | Alpha | Spark Connect
S3 / Delta Lake | `full` | ❌ | -| `postgres` | PostgreSQL | Alpha | | `full` | ❌ | -| `spiceai` | Spice.ai | Alpha | Arrow Flight | `append`, `full` | ✅ | -| `s3` | S3 | Alpha | Parquet, CSV | `full` | ❌ | -| `dremio` | Dremio | Alpha | Arrow Flight SQL | `full` | ❌ | -| `flightsql` | FlightSQL | Alpha | Arrow Flight SQL | `full` | ❌ | -| `snowflake` | Snowflake | Alpha | Arrow | `full` | ❌ | -| `mysql` | MySQL | Alpha | | `full` | ❌ | -| `clickhouse` | Clickhouse | Alpha | | `full` | ❌ | -| `spark` | Spark | Alpha | Spark Connect | `full` | ❌ | -| `ftp`, `sftp`| FTP/SFTP | Alpha | Parquet, CSV | `full` | ❌ | -| `graphql` | GraphQL | Alpha | GraphQL | `full` | ❌ | -| `odbc` | ODBC | Alpha | ODBC | `full` | ❌ | +| Name | Description | Status | Protocol/Format | Refresh Modes | Supports Inserts | +| --------------- | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- | +| `databricks` | Databricks | Alpha | Spark Connect
S3 / Delta Lake | `full` | ❌ | +| `postgres` | PostgreSQL | Alpha | | `full` | ❌ | +| `spiceai` | Spice.ai | Alpha | Arrow Flight | `append`, `full` | ✅ | +| `s3` | S3 | Alpha | Parquet, CSV | `full` | ❌ | +| `dremio` | Dremio | Alpha | Arrow Flight SQL | `full` | ❌ | +| `flightsql` | FlightSQL | Alpha | Arrow Flight SQL | `full` | ❌ | +| `snowflake` | Snowflake | Alpha | Arrow | `full` | ❌ | +| `mysql` | MySQL | Alpha | | `full` | ❌ | +| `clickhouse` | Clickhouse | Alpha | | `full` | ❌ | +| `spark` | Spark | Alpha | Spark Connect | `full` | ❌ | +| `ftp`, `sftp` | FTP/SFTP | Alpha | Parquet, CSV | `full` | ❌ | +| `graphql` | GraphQL | Alpha | GraphQL | `full` | ❌ | +| `odbc` | ODBC | Alpha | ODBC | `full` | ❌ | +| `http`, `https` | HTTP(s) | Alpha | Parquet, CSV | `full` | ❌ | ## Data Connector Docs diff --git a/spiceaidocs/docs/reference/spicepod/datasets.md b/spiceaidocs/docs/reference/spicepod/datasets.md index 0dafed46..25e5b01c 100644 --- a/spiceaidocs/docs/reference/spicepod/datasets.md +++ b/spiceaidocs/docs/reference/spicepod/datasets.md @@ -86,6 +86,7 @@ Where: - [`flightsql`](../../data-connectors/flightsql.md) - [`snowflake`](../../data-connectors/snowflake.md) - [`ftp`, `sftp`](../../data-connectors/ftp.md) + - [`http`, `https`](../../data-connectors/https.md) - [`clickhouse`](../../data-connectors/clickhouse.md) - [`graphql`](../../data-connectors/graphql.md)