Skip to content

Commit

Permalink
docs: add troubleshooting page (meltano#6706)
Browse files Browse the repository at this point in the history
* Add troubleshooting page

* Apply suggestions from code review

Co-authored-by: Taylor A. Murphy <taylor@meltano.com>

* Remove images in troubleshooting docs directory

Co-authored-by: Taylor A. Murphy <taylor@meltano.com>
  • Loading branch information
jared-rimmer and tayloramurphy committed Sep 4, 2022
1 parent ca3fdfd commit cf3566e
Showing 1 changed file with 89 additions and 0 deletions.
89 changes: 89 additions & 0 deletions docs/src/_guide/troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
---
title: Troubleshooting
description: Learn what you can do if you need to troubleshoot
layout: doc
weight: 25
---

If you have a question about Meltano, are having trouble getting it to work, or have any kind of feedback, you can:

<div class="columns is-multiline is-centered">
<div class="column is-half">
<div class="card chiclet">
<div class="card-content has-text-centered">
<div class="block">
<p><span class="icon is-large"><i class="fab fa-github fa-5x"></i></span></p>
<p>Check out the <a href="https://github.com/meltano/meltano/issues/">Meltano issue tracker</a> to see if someone else has already reported the same issue or made the same request. Feel free to weigh in with extra information, or just to make sure the issue gets the attention it deserves.</p>
</div>
</div>
</div>
</div>
<div class="column is-half">
<div class="card chiclet">
<div class="card-content has-text-centered">
<div class="block">
<p><span class="icon is-large"><i class="fab fa-slack fa-5x"></i></span></p>
<p>Join the <a href="https://meltano.com/slack">Meltano Slack workspace</a>, which is frequented by the core team and thousands of community members and data experts. You can ask any questions you may have in here or just chat with fellow users.</p>
</div>
</div>
</div>
</div>
</div>

# Common Troubleshooting Issues

## Log Level Debug

When you are trying to troubleshoot an issue the Meltano logs should be your first port of call.

If you have a failure using Meltano's execution commands (`invoke`, `elt`, `run`, or `test`) or you're experienced general unexpected behavior, you can learn more about what’s happening behind the scenes by setting Meltano’s [`cli.log_level` setting](/reference/settings#clilog_level) to debug, using the `MELTANO_CLI_LOG_LEVEL` environment variable or the `--log-level` CLI option:

```bash
export MELTANO_CLI_LOG_LEVEL=debug

meltano --log-level=debug <command> ...
```

In debug mode, Meltano will log additional information about the environment and arguments used to invoke your components (Singer taps and targets, dbt, Airflow, etc.), including the paths to the generated config, catalog, state files, etc. for you to review.

Here is an example with `meltano elt`:

```
$ meltano --log-level=debug elt tap-gitlab target-jsonl --state-id=gitlab-to-jsonl
meltano | INFO Running extract & load...
meltano | INFO Found state from 2020-08-05 21:30:20.487312.
meltano | DEBUG Invoking: ['demo-project/.meltano/extractors/tap-gitlab/venv/bin/tap-gitlab', '--config', 'demo-project/.meltano/run/tap-gitlab/tap.config.json', '--state', 'demo-project/.meltano/run/tap-gitlab/state.json']
meltano | DEBUG Env: {'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01'}
meltano | DEBUG Invoking: ['demo-project/.meltano/loaders/target-jsonl/venv/bin/target-jsonl', '--config', 'demo-project/.meltano/run/target-jsonl/target.config.json']
meltano | DEBUG Env: {'MELTANO_EXTRACTOR_NAME': 'tap-gitlab', 'MELTANO_EXTRACTOR_NAMESPACE': 'tap_gitlab', 'MELTANO_EXTRACT_API_URL': 'https://gitlab.com', 'MELTANO_EXTRACT_PRIVATE_TOKEN': '', 'MELTANO_EXTRACT_GROUPS': '', 'MELTANO_EXTRACT_PROJECTS': 'meltano/meltano', 'MELTANO_EXTRACT_ULTIMATE_LICENSE': 'False', 'MELTANO_EXTRACT_START_DATE': '2021-03-01', 'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01', 'TARGET_JSONL_DESTINATION_PATH': 'output', 'TARGET_JSONL_DO_TIMESTAMP_FILE': 'False'}
```

## Dump Files Generated by Running Meltano Commands to STDOUT

The [`--dump` flag](/reference/command-line-interface#parameters-1) can be passed to the `meltano invoke` and `meltano elt` commands to dump the content of a pipeline-specific generated file to `STDOUT` instead of actually running the pipeline. Note that adding support for `meltano run` is tracked in [this GitHub issue](https://github.com/meltano/meltano/issues/3072).

This can aid in debugging extractor catalog generation, incremental replication state lookup, and pipeline environment variables.

Supported values are:

- **catalog**: Dump the extractor catalog file that would be passed to the tap’s executable using the `--catalog` option.
- **state**: Dump the extractor state file that would be passed to the tap’s executable using the `--state` option.
- **extractor-config**: Dump the extractor config file that would be passed to the tap’s executable using the `--config` option.
- **loader-config**: Dump the loader config file that would be passed to the target’s executable using the `--config` option.

Like any standard output, the dumped content can be redirected to a file using >, e.g. `meltano elt ... --dump=state > state.json`.

Examples #
```bash
meltano elt tap-gitlab target-postgres --transform=run --state-id=gitlab-to-postgres

meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --full-refresh

meltano elt tap-gitlab target-postgres --catalog extract/tap-gitlab.catalog.json
meltano elt tap-gitlab target-postgres --state extract/tap-gitlab.state.json

meltano elt tap-gitlab target-postgres --select commits
meltano elt tap-gitlab target-postgres --exclude project_members

meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --dump=state > extract/tap-gitlab.state.json
```

0 comments on commit cf3566e

Please sign in to comment.