Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(mode): add mode analytics ingestion source (#3710)
- Loading branch information
1 parent
bd4ecbc
commit 8394fc6
Showing
16 changed files
with
1,995 additions
and
35 deletions.
There are no files selected for viewing
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # see https://datahubproject.io/docs/metadata-ingestion/source_docs/metabase for complete documentation | ||
| source: | ||
| type: "mode" | ||
| config: | ||
| token: 9fa6a90fcd33 | ||
| password: a03bcbc011d6f77c585f5682 | ||
| connect_uri: https://app.mode.com/ | ||
| workspace: "petabloc" | ||
| default_schema: "public" | ||
| owner_username_instead_of_email: False | ||
|
|
||
| # see https://datahubproject.io/docs/metadata-ingestion/sink_docs/datahub for complete documentation | ||
| sink: | ||
| type: "datahub-rest" | ||
| config: | ||
| server: "http://localhost:8080" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| # Mode | ||
|
|
||
| For context on getting started with ingestion, check out our [metadata ingestion guide](../README.md). | ||
|
|
||
| ## Setup | ||
|
|
||
| To install this plugin, run `pip install 'acryl-datahub[mode]'`. | ||
|
|
||
| See documentation for Mode's API at https://mode.com/developer/api-reference/introduction/ | ||
|
|
||
|
|
||
| ## Capabilities | ||
|
|
||
| This plugin extracts Charts, Reports, and associated metadata from a given Mode workspace. This plugin is in beta and has only been tested | ||
| on PostgreSQL database. | ||
|
|
||
| ### Report | ||
|
|
||
| [/api/{account}/reports/{report}](https://mode.com/developer/api-reference/analytics/reports/) endpoint is used to | ||
| retrieve the following report information. | ||
|
|
||
| - Title and description | ||
| - Last edited by | ||
| - Owner | ||
| - Link to the Report in Mode for exploration | ||
| - Associated charts within the report | ||
|
|
||
| ### Chart | ||
|
|
||
| [/api/{workspace}/reports/{report}/queries/{query}/charts'](https://mode.com/developer/api-reference/analytics/charts/#getChart) endpoint is used to | ||
| retrieve the following information. | ||
|
|
||
| - Title and description | ||
| - Last edited by | ||
| - Owner | ||
| - Link to the chart in Metabase | ||
| - Datasource and lineage information from Report queries. | ||
|
|
||
| The following properties for a chart are ingested in DataHub. | ||
|
|
||
| #### Chart Information | ||
| | Name | Description | | ||
| |-----------|----------------------------------------| | ||
| | `Filters` | Filters applied to the chart | | ||
| | `Metrics` | Fields or columns used for aggregation | | ||
| | `X` | Fields used in X-axis | | ||
| | `X2` | Fields used in second X-axis | | ||
| | `Y` | Fields used in Y-axis | | ||
| | `Y2` | Fields used in second Y-axis | | ||
|
|
||
|
|
||
| #### Table Information | ||
| | Name | Description | | ||
| |-----------|------------------------------| | ||
| | `Columns` | Column names in a table | | ||
| | `Filters` | Filters applied to the table | | ||
|
|
||
|
|
||
|
|
||
| #### Pivot Table Information | ||
| | Name | Description | | ||
| |-----------|----------------------------------------| | ||
| | `Columns` | Column names in a table | | ||
| | `Filters` | Filters applied to the table | | ||
| | `Metrics` | Fields or columns used for aggregation | | ||
| | `Rows` | Row names in a table | | ||
|
|
||
| ## Quickstart recipe | ||
|
|
||
| Check out the following recipe to get started with ingestion! See [below](#config-details) for full configuration options. | ||
|
|
||
| For general pointers on writing and running a recipe, see our [main recipe guide](../README.md#recipes). | ||
|
|
||
| ```yml | ||
| source: | ||
| type: mode | ||
| config: | ||
| # Coordinates | ||
| connect_uri: http://app.mode.com | ||
|
|
||
| # Credentials | ||
| token: token | ||
| password: pass | ||
|
|
||
| # Options | ||
| workspace: "datahub" | ||
| default_schema: "public" | ||
|
|
||
| sink: | ||
| # sink configs | ||
| ``` | ||
|
|
||
| ## Config details | ||
|
|
||
| | Field | Required | Default | Description | | ||
| |------------------| -------- |--------------------------|-------------------------------------------------------------------| | ||
| | `connect_uri` | ✅ | `"https://app.mode.com"` | Mode host URL. | | ||
| | `token` | ✅ | | Mode user token. | | ||
| | `password` | ✅ | | Mode password for authentication. | | ||
| | `default_schema` | | `public` | Default schema to use when schema is not provided in an SQL query | | ||
| | `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. | | ||
|
|
||
| See Mode's [Authentication documentation](https://mode.com/developer/api-reference/authentication/) on how to generate `token` and `password`. | ||
|
|
||
| ## Compatibility | ||
|
|
||
| Coming soon! | ||
|
|
||
|
|
||
| ## Questions | ||
|
|
||
| If you've got any questions on configuring this source, feel free to ping us on | ||
| [our Slack](https://slack.datahubproject.io/)! |
Oops, something went wrong.