Merge branch 'main' of https://github.com/BuzzCutNorman/meltano into …

…6467-bin-symbolic-link-windows

.github/workflows/test.yml

@@ -202,6 +202,7 @@ jobs:
         echo "${{ needs.tests.outputs.pytest-results-row-07 }}" >> ${GITHUB_STEP_SUMMARY}
         echo "${{ needs.tests.outputs.pytest-results-row-08 }}" >> ${GITHUB_STEP_SUMMARY}
         echo "${{ needs.tests.outputs.pytest-results-row-09 }}" >> ${GITHUB_STEP_SUMMARY}
+        echo "${{ needs.tests.outputs.pytest-results-row-10 }}" >> ${GITHUB_STEP_SUMMARY}
         echo "${{ needs.tests.outputs.pytest-results-row-11 }}" >> ${GITHUB_STEP_SUMMARY}
         echo "${{ needs.tests.outputs.pytest-results-row-12 }}" >> ${GITHUB_STEP_SUMMARY}
         # echo "${{ needs.tests.outputs.pytest-results-row-13 }}" >> ${GITHUB_STEP_SUMMARY}
@@ -255,3 +256,46 @@ jobs:
 
     - name: Upload coverage report
       uses: codecov/codecov-action@v3.1.0
+
+  mypy:
+    name: "Static type checking"
+    runs-on: ubuntu-latest
+    env:
+      FORCE_COLOR: 1
+
+    steps:
+    - name: Check out the repository
+      uses: actions/checkout@v3.0.2
+
+    - name: Setup Python 3.9
+      uses: actions/setup-python@v4.2.0
+      with:
+        python-version: '3.9'
+        architecture: x64
+        cache: 'pip'
+        cache-dependency-path: 'poetry.lock'
+
+    - name: Install Poetry
+      env:
+        PIP_CONSTRAINT: .github/workflows/resources/constraints.txt
+      run: |
+        pipx install poetry
+        poetry --version
+
+    - name: Upgrade pip
+      env:
+        PIP_CONSTRAINT: .github/workflows/resources/constraints.txt
+      run: |
+        pip install pip
+        pip --version
+
+    - name: Install Nox
+      env:
+        PIP_CONSTRAINT: .github/workflows/resources/constraints.txt
+      run: |
+        pipx install nox
+        pipx inject nox nox-poetry
+
+    - name: Run mypy
+      run: |
+        nox -rs mypy

docs/src/_contribute/style.md

@@ -22,7 +22,8 @@ Python:
 Flake8 is a python tool that glues together `pycodestyle`, `pyflakes`, `mccabe`, and third-party plugins to check the style and quality of python code,
 and `wemake-python-styleguide` is a plugin for Flake8 that offers an extensive set of opinionated rules that encourage clean and correct code.
 
-[MyPy is currently not executed in CI](https://github.com/meltano/meltano/issues/6491). It currently raises many issues when run. We intend to address them over time.
+[MyPy is being adopted incrementally by this codebase](https://github.com/meltano/meltano/issues/6491). If you are adding new code, please use type hints.
+To enable type checks for an existing module, _de-glob_ it in the whitelist in `pyproject.toml` and fix any errors.
 
 Javascript:
 

docs/src/_guide/logging.md

@@ -23,7 +23,7 @@ A logging.yaml contains a few key sections that you should be aware of.
 A few key points to note:
 
 1. Different handlers can use different formats. Meltano ships with [3 formatters](https://github.com/meltano/meltano/blob/main/src/meltano/core/logging/formatters.py):
-   - `meltano.core.logging.console_log_formatter` - A formatter that renders lines for the console, with optional colorization.
+   - `meltano.core.logging.console_log_formatter` - A formatter that renders lines for the console, with optional colorization. When colorization is enabled, tracebacks are formatted with the `rich` python library.
    - `meltano.core.logging.json_log_formatter` - A formatter that renders lines in JSON format.
    - `meltano.core.logging.key_value` - A formatter that renders lines in key=value format.
 2. Different loggers can use different handlers and log at different log levels.
@@ -40,10 +40,10 @@ formatters:
     format: "[%(asctime)s] [%(process)d|%(threadName)10s|%(name)s] [%(levelname)s] %(message)s"
   structured_plain: # log format for structured plain text logs without colored output
     (): meltano.core.logging.console_log_formatter
-    colors: False
+    colors: False # also disables `rich` traceback formatting
   structured_colored: # log format for structured plain text logs WITH colored output
     (): meltano.core.logging.console_log_formatter
-    colors: True
+    colors: True # also enables traceback formatting with `rich`
   key_value: # log format for traditional key=value style logs
     (): meltano.core.logging.key_value_formatter
     sort_keys: False

docs/src/_guide/troubleshooting.md

@@ -0,0 +1,89 @@
+---
+title: Troubleshooting
+description: Learn what you can do if you need to troubleshoot
+layout: doc
+weight: 25
+---
+
+If you have a question about Meltano, are having trouble getting it to work, or have any kind of feedback, you can:
+
+<div class="columns is-multiline is-centered">
+    <div class="column is-half">
+      <div class="card chiclet">
+        <div class="card-content has-text-centered">
+          <div class="block">
+            <p><span class="icon is-large"><i class="fab fa-github fa-5x"></i></span></p>
+            <p>Check out the <a href="https://github.com/meltano/meltano/issues/">Meltano issue tracker</a> to see if someone else has already reported the same issue or made the same request. Feel free to weigh in with extra information, or just to make sure the issue gets the attention it deserves.</p>
+          </div>
+        </div>
+      </div>
+    </div>
+    <div class="column is-half">
+      <div class="card chiclet">
+        <div class="card-content has-text-centered">
+          <div class="block">
+            <p><span class="icon is-large"><i class="fab fa-slack fa-5x"></i></span></p>
+            <p>Join the <a href="https://meltano.com/slack">Meltano Slack workspace</a>, which is frequented by the core team and thousands of community members and data experts. You can ask any questions you may have in here or just chat with fellow users.</p>
+          </div>
+        </div>
+      </div>
+    </div>
+</div>
+
+# Common Troubleshooting Issues
+
+## Log Level Debug
+
+When you are trying to troubleshoot an issue the Meltano logs should be your first port of call.
+
+If you have a failure using Meltano's execution commands (`invoke`, `elt`, `run`, or `test`) or you're experienced general unexpected behavior, you can learn more about what’s happening behind the scenes by setting Meltano’s [`cli.log_level` setting](/reference/settings#clilog_level) to debug, using the `MELTANO_CLI_LOG_LEVEL` environment variable or the `--log-level` CLI option:
+
+```bash
+export MELTANO_CLI_LOG_LEVEL=debug
+
+meltano --log-level=debug <command> ...
+```
+
+In debug mode, Meltano will log additional information about the environment and arguments used to invoke your components (Singer taps and targets, dbt, Airflow, etc.), including the paths to the generated config, catalog, state files, etc. for you to review.
+
+Here is an example with `meltano elt`:
+
+```
+$ meltano --log-level=debug elt tap-gitlab target-jsonl --state-id=gitlab-to-jsonl
+meltano            | INFO Running extract & load...
+meltano            | INFO Found state from 2020-08-05 21:30:20.487312.
+meltano            | DEBUG Invoking: ['demo-project/.meltano/extractors/tap-gitlab/venv/bin/tap-gitlab', '--config', 'demo-project/.meltano/run/tap-gitlab/tap.config.json', '--state', 'demo-project/.meltano/run/tap-gitlab/state.json']
+meltano            | DEBUG Env: {'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01'}
+meltano            | DEBUG Invoking: ['demo-project/.meltano/loaders/target-jsonl/venv/bin/target-jsonl', '--config', 'demo-project/.meltano/run/target-jsonl/target.config.json']
+meltano            | DEBUG Env: {'MELTANO_EXTRACTOR_NAME': 'tap-gitlab', 'MELTANO_EXTRACTOR_NAMESPACE': 'tap_gitlab', 'MELTANO_EXTRACT_API_URL': 'https://gitlab.com', 'MELTANO_EXTRACT_PRIVATE_TOKEN': '', 'MELTANO_EXTRACT_GROUPS': '', 'MELTANO_EXTRACT_PROJECTS': 'meltano/meltano', 'MELTANO_EXTRACT_ULTIMATE_LICENSE': 'False', 'MELTANO_EXTRACT_START_DATE': '2021-03-01', 'TAP_GITLAB_API_URL': 'https://gitlab.com', 'GITLAB_API_TOKEN': '', 'GITLAB_API_GROUPS': '', 'GITLAB_API_PROJECTS': 'meltano/meltano', 'GITLAB_API_ULTIMATE_LICENSE': 'False', 'GITLAB_API_START_DATE': '2021-03-01', 'TARGET_JSONL_DESTINATION_PATH': 'output', 'TARGET_JSONL_DO_TIMESTAMP_FILE': 'False'}
+```
+
+## Dump Files Generated by Running Meltano Commands to STDOUT
+
+The [`--dump` flag](/reference/command-line-interface#parameters-1) can be passed to the `meltano invoke` and `meltano elt` commands to dump the content of a pipeline-specific generated file to `STDOUT` instead of actually running the pipeline. Note that adding support for `meltano run` is tracked in [this GitHub issue](https://github.com/meltano/meltano/issues/3072).
+
+This can aid in debugging extractor catalog generation, incremental replication state lookup, and pipeline environment variables.
+
+Supported values are:
+
+- **catalog**: Dump the extractor catalog file that would be passed to the tap’s executable using the `--catalog` option.
+- **state**: Dump the extractor state file that would be passed to the tap’s executable using the `--state` option.
+- **extractor-config**: Dump the extractor config file that would be passed to the tap’s executable using the `--config` option.
+- **loader-config**: Dump the loader config file that would be passed to the target’s executable using the `--config` option.
+
+Like any standard output, the dumped content can be redirected to a file using >, e.g. `meltano elt ... --dump=state > state.json`.
+
+Examples #
+```bash
+meltano elt tap-gitlab target-postgres --transform=run --state-id=gitlab-to-postgres
+
+meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --full-refresh
+
+meltano elt tap-gitlab target-postgres --catalog extract/tap-gitlab.catalog.json
+meltano elt tap-gitlab target-postgres --state extract/tap-gitlab.state.json
+
+meltano elt tap-gitlab target-postgres --select commits
+meltano elt tap-gitlab target-postgres --exclude project_members
+
+meltano elt tap-gitlab target-postgres --state-id=gitlab-to-postgres --dump=state > extract/tap-gitlab.state.json
+```

noxfile.py

@@ -5,6 +5,7 @@
 import os
 import sys
 from pathlib import Path
+from random import randint
 from textwrap import dedent
 
 try:
@@ -45,6 +46,8 @@ def tests(session: Session) -> None:
         "pytest",
         "pytest-asyncio",
         "pytest-docker",
+        "pytest-order",
+        "pytest-randomly",
         "requests-mock",
     )
 
@@ -55,6 +58,7 @@ def tests(session: Session) -> None:
             "--parallel",
             "-m",
             "pytest",
+            f"--randomly-seed={randint(0, 2**32-1)}",  # noqa: S311, WPS432
             *session.posargs,
             env={"NOX_CURRENT_SESSION": "tests"},
         )
@@ -78,3 +82,21 @@ def coverage(session: Session) -> None:
         session.run("coverage", "combine")
 
     session.run("coverage", *args)
+
+
+@nox_session(python=main_python_version)
+def mypy(session: Session) -> None:
+    """Run mypy type checking.
+
+    Args:
+        session: Nox session.
+    """
+    args = session.posargs or ["src/meltano"]
+
+    session.install(".")
+    session.install(
+        "mypy",
+        "sqlalchemy2-stubs",
+        "types-requests",
+    )
+    session.run("mypy", *args)