Skip to content

Commit

Permalink
docs: add capabilities reference doc, link to it where applicable (me…
Browse files Browse the repository at this point in the history
  • Loading branch information
aaronsteers committed Aug 31, 2022
1 parent 5068280 commit 3237022
Show file tree
Hide file tree
Showing 5 changed files with 50 additions and 8 deletions.
5 changes: 3 additions & 2 deletions docs/src/_concepts/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,9 @@ In order to use a given package as a plugin in a [project](project),
assuming it meets the requirements of the [plugin type](#types) in question, Meltano needs to know:

1. where to find the package, typically a [pip package](https://pip.pypa.io/en/stable/) identified by its name on [PyPI](https://pypi.org/), public or private Git repository URL, or local directory path,
2. what [settings](/guide/configuration) and other capabilities it supports, and finally
3. what its [configuration](/guide/configuration) should be when invoked.
2. what [settings](/guide/configuration) it supports
3. what [capabilities](/reference/plugin-definition-syntax#capabilities) it supports, and finally
4. what its [configuration](/guide/configuration) should be when invoked.

Together, a package's location (1) and the metadata (2) describing it in terms Meltano can understand make up the **base plugin description**.
In your project, **plugins** extend this description with a specific configuration (3) and a unique name.
Expand Down
4 changes: 4 additions & 0 deletions docs/src/_guide/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,10 +280,14 @@ In prior versions of the Singer spec, the `--properties` option was used instead
If this is the case for a tap, ensure `properties` is set as a [capability](/contribute/plugins) for the tap instead of `catalog`.
Then `meltano elt` will accept the catalog file and will pass it to the tap using the appropriate flag.

For more information, please refer to the [plugin capabilities reference](/reference/plugin-definition-syntax#capabilities).

### Incremental Replication Not Running as Expected

If you're trying to run a pipeline with incremental replication using `meltano elt` but it's running a full sync, ensure that you're passing a [State ID](/getting-started#run-a-data-integration-el-pipeline) via the [`--state-id` flag](/reference/command-line-interface#how-to-use-4).

If using a custom tap, ensure that the tap declares the `state` capability as described in the [plugin capabilities reference](/reference/plugin-definition-syntax#capabilities).

### Testing Specific Failing Streams

When extracting several streams with a single tap, it may be challenging to debug a single failing stream.
Expand Down
4 changes: 1 addition & 3 deletions docs/src/_guide/plugin-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -280,9 +280,7 @@ If it's a tap or target you have developed or are developing yourself,
you'll want to set `pip_url` to either a [Git repository URL](https://pip.pypa.io/en/stable/reference/pip_install/#git) or local directory path.
If you add the `-e` flag ahead of the local path, the package will be installed in [editable mode](https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs).

To find out what `settings` a tap or target supports, reference the README in the repository and/or documentation.
If the `capabilities` a tap supports (executable flags like `--discover` and `--state`) are not described there,
you can try [one of these tricks](/contribute/plugins#how-to-test-a-tap) or refer directly to the source code.
To find out what `settings` a tap or target supports, reference the README in the repository and/or documentation. To learn more about the available plugin capabilities, please refer to the [plugin capabilities reference](/reference/plugin-definition-syntax#capabilities). If the `capabilities` a tap supports (executable flags like `--discover` and `--state`) are not described in the available documentation, you can try [one of these tricks](/contribute/plugins#how-to-test-a-tap) or refer directly to the source code.

This will add a [custom plugin definition](/concepts/project#custom-plugin-definitions) to your [`meltano.yml` project file](/concepts/project#plugins) under the `plugins` property, inside an array named after the plugin type:

Expand Down
43 changes: 40 additions & 3 deletions docs/src/_reference/plugin-definition-syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,17 +57,54 @@ repo: https://github.com/apache/airflow

## `capabilities`

Array of capabilities that the plugin supports.
Array of capabilities that the plugin supports. For example:

```yaml
capabilities:
- catalog
- discover
- state
- about
- stream-maps
```

The full list of defined capabilities is below:

- [`catalog`](#catalog-capability)
- [`properties`](#properties-capability)
- [`discover`](#discover-capability)
- [`state`](#state-capability)
- [`about`](#about-capability)
- [`stream-maps`](#stream-maps-capability)

### `catalog` capability

Declares that the plugin supports stream and property selection using the `--catalog` CLI argument, which is a newer version of the [`--properties` capability](#properties-capability).

Note: The `catalog` capability is a newer version of the [`properties` capability](#properties-capability). Singer taps which support field and stream selection logic should declare the `properties` or `catalog` capability, but not both.

### `properties` capability

Declares that the plugin supports stream and property selection using the `--properties` CLI argument.

Note: The `properties` capability is an older version of the [`--catalog` capability](#catalog-capability). Singer taps which support field and stream selection logic should declare the `properties` or `catalog` capability, but not both.

### `discover` capability

Declares that the plugin can be run with the `--discover` CLI argument, which generates a `catalog.json` file. This is used by Meltano in combination with the `catalog` or `properties` capability to customize the catalog and to apply selection logic.

### `state` capability

Declares that the plugin is able to perform incremental processing using the `--state` CLI option.

Note: This capability must be declared in order to use incremental data replication.

### `about` capability

Declares that the plugin supports a `--about` CLI argument and a paired `--format=json` to optionally print the plugin's metadata in a machine readable format. This capability can be used by users to better understand the capabilities and settings expected by the plugin. It may also be used by Meltano and MeltanoHub codebase to auto-detect behaviors and capabilities.

### `stream-maps` capability

For Singer connectors, declares the ability to perform inline transformations or 'mappings' within the stream. For more details, please see the [Singer SDK Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html) documentation.

## `pip_url`

A string with the plugin's [`pip install`](https://pip.pypa.io/en/stable/cli/pip_install/#options) argument. Can point to multiple packages and include any of the pip install options.
Expand Down
2 changes: 2 additions & 0 deletions docs/src/_tutorials/custom-extractor.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ plugins:
# Can be found in the pyproject.toml of your custom tap under CLI declaration
executable: tap-my-custom-source
capabilities:
# For a reference of plugin capabilities, see:
# https://docs.meltano.com/reference/plugin-definition-syntax#capabilities
- state
- catalog
- discover
Expand Down

0 comments on commit 3237022

Please sign in to comment.