Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Use tabs for meltano.yml vs meltano config or env #7765

Merged
merged 7 commits into from
Jun 9, 2023
Merged

docs: Use tabs for meltano.yml vs meltano config or env #7765

Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d36ccf4
Use tabs for meltano.yml vs meltano config or env
mjsqu Jun 8, 2023
5a182e8
Uniquely name tab groups
mjsqu Jun 8, 2023
4089f9d
Add script tag for tabs.js
mjsqu Jun 8, 2023
b874c0e
[pre-commit.ci] auto fixes from pre-commit.ci hooks
pre-commit-ci[bot] Jun 8, 2023
0bd18c9
Apply suggestions from code review
tayloramurphy Jun 9, 2023
259c4e6
Merge branch 'main' into main
tayloramurphy Jun 9, 2023
b0862c9
Rename tab groups to the same name and add third env tab
mjsqu Jun 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 97 additions & 24 deletions docs/src/_concepts/plugins.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,15 +172,18 @@ If the catalog does not seem to take effect, you may need to [validate the capab

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs catalog %}
{% tab catalog meltano.yml %}

```yaml{3}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
- name: tap-gitlab
catalog: extract/tap-gitlab.catalog.json
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab catalog config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _catalog <path>
Expand All @@ -195,7 +198,10 @@ meltano config tap-gitlab set _catalog extract/tap-gitlab.catalog.json
export TAP_GITLAB__CATALOG=extract/tap-gitlab.catalog.json

meltano elt tap-gitlab target-jsonl --catalog extract/tap-gitlab.catalog.json

```
{% endtab %}
{% endtabs %}

#### <a name="load-schema-extra"></a>`load_schema` extra

Expand All @@ -213,15 +219,18 @@ It is used as the default value for the [`target-postgres`](https://hub.meltano.

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs load_schema %}
{% tab load_schema meltano.yml %}

```yaml{3}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
- name: tap-gitlab
load_schema: gitlab_data
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab load_schema config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _load_schema <schema>
Expand All @@ -232,7 +241,10 @@ export <EXTRACTOR>__LOAD_SCHEMA=<schema>
meltano config tap-gitlab set _load_schema gitlab_data

export TAP_GITLAB__LOAD_SCHEMA=gitlab_data

```
{% endtab %}
{% endtabs %}

#### `metadata` extra

Expand All @@ -258,7 +270,9 @@ Entity and attribute names can be discovered using [`meltano select --list --all

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs metadata %}
{% tab metadata meltano.yml %}

```yaml{3-8}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
Expand All @@ -271,7 +285,8 @@ extractors:
is-replication-key: true
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab metadata config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _metadata <entity> <key> <value>
Expand All @@ -289,7 +304,10 @@ meltano config tap-postgres set _metadata some_stream_id replication-key created
meltano config tap-postgres set _metadata some_stream_id created_at is-replication-key true

export TAP_POSTGRES__METADATA_SOME_STREAM_ID_REPLICATION_METHOD=FULL_TABLE

```
{% endtab %}
{% endtabs %}

#### `schema` extra

Expand All @@ -315,7 +333,9 @@ This allows you to define a full schema for taps such as [`tap-dynamodb`](https:

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs schema %}
{% tab schema meltano.yml %}

```yaml{3-7}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
Expand All @@ -327,7 +347,8 @@ extractors:
format: date-time
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab schema config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _schema <entity> <attribute> <schema description>
Expand All @@ -344,7 +365,10 @@ meltano config tap-postgres set _metadata some_stream_id created_at type '["stri
meltano config tap-postgres set _metadata some_stream_id created_at format date-time

export TAP_POSTGRES__SCHEMA_SOME_STREAM_ID_CREATED_AT_FORMAT=date

```
{% endtab %}
{% endtabs %}

#### `select` extra

Expand All @@ -366,7 +390,9 @@ selection rules are typically specified using [`meltano select`](/reference/comm

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs select %}
{% tab select meltano.yml %}

```yaml{3-5}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
Expand All @@ -376,7 +402,8 @@ extractors:
- commits.*
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab select config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _select '["<entity>.<attribute>", ...]'
Expand All @@ -392,7 +419,10 @@ export TAP_GITLAB__SELECT='["project_members.*", "commits.*"]'

meltano select tap-gitlab project_members "*"
meltano select tap-gitlab commits "*"

```
{% endtab %}
{% endtabs %}

#### <a name="select-filter-extra"></a>`select_filter` extra

Expand All @@ -417,7 +447,9 @@ selection filers are typically specified using [`meltano elt`](/reference/comman

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs select_filter %}
{% tab select_filter meltano.yml %}

```yaml{6-7}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
Expand All @@ -429,7 +461,8 @@ extractors:
- commits
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab select_filter config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _select_filter '["<entity>", ...]'
Expand All @@ -450,7 +483,10 @@ export TAP_GITLAB__SELECT_FILTER='["!project_members"]'

meltano elt tap-gitlab target-jsonl --select commits
meltano elt tap-gitlab target-jsonl --exclude project_members

```
{% endtab %}
{% endtabs %}

#### `state` extra

Expand All @@ -469,15 +505,18 @@ a state file is typically provided using [`meltano elt`](/reference/command-line

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs state %}
{% tab state meltano.yml %}

```yaml{3}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
extractors:
- name: tap-gitlab
state: extract/tap-gitlab.state.json
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab state config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <extractor> set _state <path>
Expand All @@ -492,7 +531,10 @@ meltano config tap-gitlab set _state extract/tap-gitlab.state.json
export TAP_GITLAB__STATE=extract/tap-gitlab.state.json

meltano elt tap-gitlab target-jsonl --state extract/tap-gitlab.state.json

```
{% endtab %}
{% endtabs %}

### Loaders

Expand Down Expand Up @@ -526,15 +568,18 @@ It is used as the default value for `dbt`'s `target` setting, and should therefo

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs dialect %}
{% tab dialect meltano.yml %}

```yaml{3}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
loaders:
- name: target-example-db
dialect: example-db
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab dialect config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <loader> set _dialect <dialect>
Expand All @@ -545,7 +590,10 @@ export <LOADER>__DIALECT=<dialect>
meltano config target-example-db set _dialect example-db

export TARGET_EXAMPLE_DB__DIALECT=example-db

```
{% endtab %}
{% endtabs %}

#### <a name="target-schema-extra"></a>`target_schema` extra

Expand All @@ -566,7 +614,9 @@ It is used as the default value for `dbt`'s `source_schema` setting.

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs target_schema %}
{% tab target_schema meltano.yml %}

```yaml{5}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
loaders:
Expand All @@ -576,7 +626,8 @@ loaders:
target_schema: $MELTANO_LOAD_DESTINATION_SCHEMA # Value of `destination_schema` setting
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab target_schema config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <loader> set _target_schema <schema>
Expand All @@ -590,7 +641,10 @@ meltano config target-example-db set _target_schema '$MELTANO_LOAD_DESTINATION_S
meltano config target-example-db set _target_schema explicit_target_schema

export TARGET_EXAMPLE_DB__TARGET_SCHEMA=explicit_target_schema

```
{% endtab %}
{% endtabs %}

### Transforms

Expand Down Expand Up @@ -632,7 +686,9 @@ It is included in the default value for `dbt`'s `models` setting: `$MELTANO_TRAN

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs package_name %}
{% tab package_name meltano.yml %}

```yaml{4}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
transforms:
Expand All @@ -641,7 +697,8 @@ transforms:
package_name: facebook_ads
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab package_name config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <transform> set _package_name <name>
Expand All @@ -652,7 +709,10 @@ export <TRANSFORM>__PACKAGE_NAME=<name>
meltano config dbt-facebook-ads set _package_name facebook_ads

export DBT_FACEBOOK_ADS__PACKGE_NAME=facebook_ads

```
{% endtab %}
{% endtabs %}

#### `vars` extra

Expand All @@ -669,7 +729,9 @@ Because these variables are handled by dbt rather than Meltano, [environment var

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs vars %}
{% tab vars meltano.yml %}

```yaml
{% raw %}
Expand All @@ -680,7 +742,8 @@ transforms:
{% endraw %}
```

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab vars config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
{% raw %}
Expand All @@ -693,7 +756,10 @@ meltano config --plugin-type=transform tap-gitlab set _vars schema "{{ env_var('

export TAP_GITLAB__VARS='{"schema": "{{ env_var(''DBT_SOURCE_SCHEMA'') }}"}'
{% endraw %}

```
{% endtab %}
{% endtabs %}
### Orchestrators

<div class="notification is-warning">
Expand Down Expand Up @@ -744,7 +810,9 @@ When a file path's value is `True`, the matching files are considered to be mana

##### How to use

Manage this extra directly in your [`meltano.yml` project file](project#meltano-yml-project-file):
Manage this extra:
{% tabs update %}
{% tab update meltano.yml %}

```yaml{3-4}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved
files:
Expand All @@ -764,7 +832,8 @@ files:
</pre>
</div>

Alternatively, manage this extra using [`meltano config`](/reference/command-line-interface#config) or an [environment variable](/guide/configuration#configuring-settings):
{% endtab %}
{% tab update config %}
tayloramurphy marked this conversation as resolved.
Show resolved Hide resolved

```bash
meltano config <bundle> set _update <path> <true/false>
Expand All @@ -776,7 +845,10 @@ meltano config --plugin-type=files dbt set _update transform/dbt_project.yml fal
meltano config --plugin-type=files dbt set _update profiles/*.yml true

export DBT__UPDATE='{"transform/dbt_project.yml": false, "profiles/*.yml": true}'

```
{% endtab %}
{% endtabs %}

### Utilities

Expand Down Expand Up @@ -905,3 +977,4 @@ call mappings that leverage the same plugin at multiple locations numerous times
# Mask the id if the customer is in the EU region using transform-field mapper.
$ tap-someapi fix-null-country set-region-from-country mask-id-if-eu target-jsonl
```
<script src="/js/tabs.js"></script>