Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documentation for HTTP(s) connector. #302

Merged
merged 6 commits into from
Jul 4, 2024
Merged

Documentation for HTTP(s) connector. #302

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c31b0a2
add http docs
Jeadie Jul 3, 2024
fdd833e
link HTTP doc
Jeadie Jul 3, 2024
e1ceeca
Update https.md
Jeadie Jul 4, 2024
99d5743
update HTTP params
Jeadie Jul 4, 2024
60fda5b
Merge branch 'jeadie/24-07-03/http' of https://github.com/spiceai/doc…
Jeadie Jul 4, 2024
d85be43
PR comments
Jeadie Jul 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions spiceaidocs/docs/data-connectors/https.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
title: 'HTTP(s) Data Connector'
sidebar_label: 'HTTP(s) Data Connector'
description: 'HTTP(s) Data Connector Documentation'
pagination_prev: null
---

The HTTP(s) Data Connector enables federated SQL query against a variety of tabular formatted (e.g. Parquet/CSV) files stored at a HTTP endpoint.

The connector supports HTTP authentication via either `param` and/or `secrets`.

### Parameters
- `http_port`: Optional. Port to create HTTP(s) connection over. Default: 80 and 443 for HTTP and HTTPS respectively.
- `http_username`: Optional. Username to provide connection for HTTP basic authentication. Default: None.
- `http_password`: Optional. Password to provide connection for HTTP basic authentication. Default: None.
- `http_password_key`: Key of the secret that contains the value to use for `http_password`. Default: None.

### Examples
```yaml
datasets:
- from: https://github.com/LAION-AI/audio-dataset/raw/7fd6ae3cfd7cde619f6bed817da7aa2202a5bc28/metadata/freesound/parquet/freesound_parquet.parquet
name: laion_freesound

- from: http://static_username@localhost:3001/report.csv
name: local_report
params:
http_password: BadPa$5w0rd
```
To use a secret for the HTTP password, for example, by env variable
```yaml
datasets:
- from: http://static_username@localhost/report.csv
name: local_report
params:
http_password_key: local_password
http_port: 3001
```
With the associated secret set, for example:
```shell
export SPICE_SECRET_HTTP_LOCAL_PASSWORD="BadPa$5w0rd"
```
31 changes: 16 additions & 15 deletions spiceaidocs/docs/data-connectors/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,21 +11,22 @@ Data Connectors provide connections to databases, data warehouses, and data lake

Currently supported Data Connectors include:

| Name | Description | Status | Protocol/Format | Refresh Modes | Supports Inserts |
| ------------ | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- |
| `databricks` | Databricks | Alpha | Spark Connect <br/> S3 / Delta Lake | `full` | ❌ |
| `postgres` | PostgreSQL | Alpha | | `full` | ❌ |
| `spiceai` | Spice.ai | Alpha | Arrow Flight | `append`, `full` | ✅ |
| `s3` | S3 | Alpha | Parquet, CSV | `full` | ❌ |
| `dremio` | Dremio | Alpha | Arrow Flight SQL | `full` | ❌ |
| `flightsql` | FlightSQL | Alpha | Arrow Flight SQL | `full` | ❌ |
| `snowflake` | Snowflake | Alpha | Arrow | `full` | ❌ |
| `mysql` | MySQL | Alpha | | `full` | ❌ |
| `clickhouse` | Clickhouse | Alpha | | `full` | ❌ |
| `spark` | Spark | Alpha | Spark Connect | `full` | ❌ |
| `ftp`, `sftp`| FTP/SFTP | Alpha | Parquet, CSV | `full` | ❌ |
| `graphql` | GraphQL | Alpha | GraphQL | `full` | ❌ |
| `odbc` | ODBC | Alpha | ODBC | `full` | ❌ |
| Name | Description | Status | Protocol/Format | Refresh Modes | Supports Inserts |
| --------------- | ----------- | ------------ | ----------------------------------- | ---------------- | ---------------- |
| `databricks` | Databricks | Alpha | Spark Connect <br/> S3 / Delta Lake | `full` | ❌ |
| `postgres` | PostgreSQL | Alpha | | `full` | ❌ |
| `spiceai` | Spice.ai | Alpha | Arrow Flight | `append`, `full` | ✅ |
| `s3` | S3 | Alpha | Parquet, CSV | `full` | ❌ |
| `dremio` | Dremio | Alpha | Arrow Flight SQL | `full` | ❌ |
| `flightsql` | FlightSQL | Alpha | Arrow Flight SQL | `full` | ❌ |
| `snowflake` | Snowflake | Alpha | Arrow | `full` | ❌ |
| `mysql` | MySQL | Alpha | | `full` | ❌ |
| `clickhouse` | Clickhouse | Alpha | | `full` | ❌ |
| `spark` | Spark | Alpha | Spark Connect | `full` | ❌ |
| `ftp`, `sftp` | FTP/SFTP | Alpha | Parquet, CSV | `full` | ❌ |
| `graphql` | GraphQL | Alpha | GraphQL | `full` | ❌ |
| `odbc` | ODBC | Alpha | ODBC | `full` | ❌ |
| `http`, `https` | HTTP(s) | Alpha | Parquet, CSV | `full` | ❌ |

## Data Connector Docs

Expand Down
1 change: 1 addition & 0 deletions spiceaidocs/docs/reference/spicepod/datasets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ Where:
- [`flightsql`](../../data-connectors/flightsql.md)
- [`snowflake`](../../data-connectors/snowflake.md)
- [`ftp`, `sftp`](../../data-connectors/ftp.md)
- [`http`, `https`](../../data-connectors/https.md)
- [`clickhouse`](../../data-connectors/clickhouse.md)
- [`graphql`](../../data-connectors/graphql.md)

Expand Down