Skip to content

Commit

Permalink
docs: add "variant" prompt and description to custom plugin developme…
Browse files Browse the repository at this point in the history
…nt docs (meltano#7050)

Co-authored-by: Will Da Silva <will@willdasilva.xyz>
Co-authored-by: Edgar R. M. <edgar@meltano.com>
  • Loading branch information
3 people committed Feb 9, 2023
1 parent dcb0e05 commit 14820bb
Showing 1 changed file with 15 additions and 13 deletions.
28 changes: 15 additions & 13 deletions docs/src/_tutorials/custom-extractor.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ You will use cookiecutter to clone the Meltano SDK template repo for implementin

Poetry serves as the dependency manager for the project. If you have used npm before, poetry serves a similar purpose but for Python projects.


## 2. Create a Project Using the Cookiecutter Template

Run the following command to create the project files from the cookiecutter template:
Expand All @@ -75,9 +74,10 @@ cookiecutter https://github.com/meltano/sdk --directory="cookiecutter/tap-templa

After running the above command, you will be prompted to configure your project.

- Type `jsonplaceholder` as your source name
- Type `jsonplaceholder` as your source name.
- Then input your first name and last name.
- You can leave the tap_id and library name as the default suggested names.
- You can leave the `tap_id` and library name as the default suggested names.
- If you are planning to distribute your project on PyPI and the `tap_id` is already in use, you can set a variant name. e.g. 'meltanolabs' or 'pipelinewise'.
- For the stream type, you should select REST, and Custom or N/A for the auth method.
- Finally, you can choose to add a CI/CD template or not. It doesn’t really matter in this case.

Expand All @@ -90,6 +90,8 @@ tap_id [tap-jsonplaceholder]:
library_name [tap_jsonplaceholder]:
variant [None (Skip)]:
Select stream_type:
1 - REST
Expand Down Expand Up @@ -142,7 +144,7 @@ See [Debug A Custom Extractor](https://docs.meltano.com/guide/debugging-custom-e

To configure your custom extractor to consume data from the JSON placeholder, you need to set the **API_URL** and the **streams** that will be replicated.

Open the file ```tap-jsonplaceholder/tap_jsonplaceholder/tap.py``` and replace its content with the content below:
Open the file `tap-jsonplaceholder/tap_jsonplaceholder/tap.py` and replace its content with the content below:

```python
"""jsonplaceholder tap class."""
Expand Down Expand Up @@ -185,8 +187,7 @@ return [stream_class(tap=self) for stream_class in STREAM_TYPES]

```

Then replace the content of ```tap-jsonplaceholder/tap_jsonplaceholder/streams.py``` with the content below:

Then replace the content of `tap-jsonplaceholder/tap_jsonplaceholder/streams.py` with the content below:

```python
"""Stream type classes for tap-jsonplaceholder."""
Expand All @@ -209,12 +210,11 @@ class CommentsStream(jsonplaceholderStream):
).to_dict()
```

The ```tap.py``` file defines the tap settings and the available streams, which is the comments stream in this case. You can find the available stream types in the STREAM_TYPES array.

The ```streams.py``` file configures the comments stream to use the /comments path and also sets the properties of the extracted fields.
The `tap.py` file defines the tap settings and the available streams, which is the comments stream in this case. You can find the available stream types in the STREAM_TYPES array.

Finally, change the url_base in the ```tap-jsonplaceholder/tap_jsonplaceholder/client.py``` file to https://jsonplaceholder.typicode.com.
The `streams.py` file configures the comments stream to use the `/comments` path and also sets the properties of the extracted fields.

Finally, change the value of `url_base` in the `tap-jsonplaceholder/tap_jsonplaceholder/client.py` file to https://jsonplaceholder.typicode.com.

```python
...
Expand Down Expand Up @@ -249,6 +249,7 @@ Execute the command below to run an ELT pipeline that replicates data from the R
```bash
meltano run tap-jsonplaceholder target-jsonl
```

You can inspect the output directory for the extracted JSON data.

Use the command below to get the first five lines of the extracted comments JSON file:
Expand Down Expand Up @@ -305,6 +306,7 @@ Navigate to the parent directory of your custom extractor and run the following
```bash
meltano init
```
The above command will prompt you to enter a project name. You should enter a name like meltano-demo. Afterward, navigate into the newly created project using the below command:
```bash
Expand All @@ -317,7 +319,6 @@ To test the custom extractor as part of your Meltano project, you will need to a
There are two ways you can do this:
## 1. Use The Meltano Add Command
Run the command below to add the extractor as a custom extractor not hosted on MeltanoHub registry:
Expand All @@ -326,9 +327,9 @@ Run the command below to add the extractor as a custom extractor not hosted on M
meltano add --custom extractor tap-jsonplaceholder
```
You will be prompted to input the namespace URL. Choose ```tap-jsonplaceholder```.
You will be prompted to input the namespace URL. Choose `tap-jsonplaceholder`.
Also, choose ```-e /tap-jsonplaceholder``` as the pip_url since you are working with a local extractor project. This should be the full path to your custom extractor.
Also, choose `-e ../tap-jsonplaceholder` as the pip_url since you are working with a local extractor project. This should be the full path to your custom extractor.
Go with the default executable name.
Expand Down Expand Up @@ -437,6 +438,7 @@ To properly expose and configure your settings, you'll need to define them:
- `foo` represents the `{ foo: VALUE }` in the output configuration.
- `foo.a` represents the `{ foo: { a: VALUE } }` in the output configuration.
- **kind**: Represent the type of value this should be, (e.g. `password` for sensitive values or `date_iso8601` for dates).
- **value** (optional): Define a default value for the plugin's setting.
Expand Down

0 comments on commit 14820bb

Please sign in to comment.