Initial docs for new adapter testing framework #1263
Conversation
| @@ -3,6 +3,10 @@ exports.versions = [ | |||
| // version: "1.1", | |||
| // EOLDate: "2023-03-18" | |||
| // }, | |||
| { | |||
| version: "1.1", | |||
| EOLDate: "2024-5-01" | |||
There was a problem hiding this comment.
TBD based on final release date
website/dbt-versions.js
Outdated
| exports.versionedPages = [ | ||
| { | ||
| "page": "docs/contributing/testing-a-new-adapter", | ||
| "firstVersion": "1.1", | ||
| } | ||
| ] |
There was a problem hiding this comment.
This didn't accomplish quite what I'd hoped it would. I was expecting a banner on the page when an older version is selected, and I'm not seeing that
There was a problem hiding this comment.
jk it's working now!
| <VersionBlock lastVersion="1.0"> | ||
|
|
||
| The previous way to test dbt adapter functionality was via a packaged suite of tests, [`pytest-dbt-adapter`](https://github.com/dbt-labs/dbt-adapter-tests). We've since deprecated that approach in favor of the new testing framework outlined in this document. | ||
|
|
||
| </VersionBlock> |
There was a problem hiding this comment.
Is this callback valuable? Or should we pretend like it never existed?
When we're ready, we will want to officially "deprecate" the dbt-adapter-tests repo by adding a note to its README, and then archive it. Similar to what we did for https://github.com/dbt-labs/dbt-integration-tests. (We're running out of names!)
There was a problem hiding this comment.
I think we might want the deprecation mentioned for a while. We wouldn't want to confuse somebody by its sudden disappearance.
✅ Deploy Preview for docs-getdbt-com ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
|
||
| The core testing framework is built using `pytest`, a mature and standard library for testing Python projects. It assumes a basic level of familiarity with the foundational concepts in `pytest`, such as fixtures. If this is your first time using `pytest`, we recommend you read the documentation: https://docs.pytest.org/ | ||
|
|
||
| The [`tests` module](https://github.com/dbt-labs/dbt-core/tree/HEAD/core/dbt/tests) within `dbt-core` includes basic utilities for setting up `pytest` + `dbt`. These are built into `dbt-core`, and used by all functional tests. These also make it possible to write your own tests, using the same utilities. |
There was a problem hiding this comment.
Realizing that the module contents here will actually be an dbt-core feature, for now undocumented, that is available to all project/package maintainers as well (if they care to write pytest cases)
Let's think about where the docs for this piece should live long-term, given that the intended audience is a bit wider than just creators/maintainers of adapter plugin
There was a problem hiding this comment.
We probably want some high level documentation of how to use the standard fixtures and structure a simple test.
| 'host': os.getenv('HOST_ENV_VAR_NAME'), | ||
| 'user': os.getenv('USER_ENV_VAR_NAME'), | ||
| ... | ||
| 'schema': unique_schema # this will be passed in by the testing framework |
There was a problem hiding this comment.
In the Redshift test I specified the unique_schema, but in at least some of the other adapters I didn't. I'm also setting it in the 'dbt_profile_data' fixture, so it's not actually necessary. I'm thinking we should standardize on not supplying it here.
|
|
||
| Then, create a configuration file within your tests directory. In it, you'll want to define all necessary profile configuration for connecting to your data platform in local development and continuous integration. We recommend setting these values with environment variables, since this file will be checked into version control. | ||
|
|
||
| <File name="pytest.ini"> |
There was a problem hiding this comment.
This is supposed to be tests/conftest.py
There was a problem hiding this comment.
@jtcohen6 this content is really solid so I didn't do too much overhauling.
One thing we might do in the future create a landing page for the intro content and then two sub pages for the "Testing your adapter" and "Getting started: basic tests." It might be easier for readers to parse the info and then we could link to a page instead the middle of a page when helping people troubleshoot.
|
|
||
| It should be noted that both of these files are included in the bootstrapped output of the `create_adapter_plugins.py` so when using that script, these files will be included. | ||
|
|
||
| ### Testing your new adapter | ||
|
|
||
| You can use a pre-configured [dbt adapter test suite](https://github.com/dbt-labs/dbt-adapter-tests) to test that your new adapter works. These tests include much of dbt's basic functionality, with the option to override or disable functionality that may not be supported on your adapter. | ||
| This has moved to its own page: ["Testing a new adapter"](testing-a-new-adapter) |
There was a problem hiding this comment.
Yay! Do we need to version the old paragraph for versions below 1.1?
There was a problem hiding this comment.
Good callout:
- The target audience here is developers of adapter integrations, not most dbt end users. I wouldn't expect anyone building a new adapter to build it for any versions older than the latest.
- The "Testing a new adapter" page mentions and links to the old testing suite, so that we acknowledge its existence, while also explaining why we recommend the new approach in its place
|
|
||
| ## Foundation | ||
|
|
||
| dbt-core offers a standard framework for running pre-built functional tests, and for defining new ones. |
There was a problem hiding this comment.
Changing this to be as specific as we can to help non-native English speakers follow along.
| dbt-core offers a standard framework for running pre-built functional tests, and for defining new ones. | |
| dbt-core offers a standard framework for running pre-built functional tests, and for defining your own tests. |
|
|
||
| ::: | ||
|
|
||
| ## Foundation |
There was a problem hiding this comment.
| ## Foundation | |
| ## About testing adapters |
There was a problem hiding this comment.
I'm being a bit coy here. This section of the docs could be relevant to anyone who wants to use the testing framework, including package/project maintainers who want the ability to write pytest cases against their package/project. I'm going to retitle this section About the testing framework.
|
|
||
| The **[`tests` module](https://github.com/dbt-labs/dbt-core/tree/HEAD/core/dbt/tests)** within `dbt-core` includes basic utilities for setting up `pytest` + `dbt`. These are used by all "pre-built" functional tests, and make it possible to quickly write your own tests. | ||
|
|
||
| ### Example |
There was a problem hiding this comment.
What do you think about moving the example to the end and after the Using basic tests (was Getting started: basic tests) section? I feel like more of an intro is helpful context (even for the most baic term "basic test") and those who know all about it will skip ahead.
| ### Example | |
| ## Example using a basic dbt test project |
There was a problem hiding this comment.
Fair, the ordering here is a bit odd; I'm intentionally sowing a seed here for a separate docs page that could exist in the near future. The first section (including this example) could be relevant to programmatic testing of any dbt-based product/project, in a way that's actually broader than the narrower adapter testing use case we're outlining in this page. But, that is our first primary use case, which is why it's all been rolled up into this one page for now.
I'll include some more framing language to make the intention with this section clearer
|
|
||
| :::info | ||
|
|
||
| Previously, we offered a packaged suite of tests for dbt adapter functionality: [`pytest-dbt-adapter`](https://github.com/dbt-labs/dbt-adapter-tests). We are deprecating that suite, in favor of the newer testing framework outlined in this document. The rest of this document assumes that your plugin is compatible with dbt-core v1.1 or newer. |
There was a problem hiding this comment.
So people don't miss this, let's more to a prerequisites section.
| Previously, we offered a packaged suite of tests for dbt adapter functionality: [`pytest-dbt-adapter`](https://github.com/dbt-labs/dbt-adapter-tests). We are deprecating that suite, in favor of the newer testing framework outlined in this document. The rest of this document assumes that your plugin is compatible with dbt-core v1.1 or newer. | |
| Previously, we offered a packaged suite of tests for dbt adapter functionality: [`pytest-dbt-adapter`](https://github.com/dbt-labs/dbt-adapter-tests). We are deprecating that suite, in favor of the newer testing framework outlined in this document. |
|
|
||
| The framework presented above is available for use by anyone who installs `dbt-core`, and who wishes to define their own test cases. | ||
|
|
||
| Building on top of that foundation, we have also built and made available a [package of reusable adapter test cases](https://github.com/dbt-labs/dbt-core/tree/HEAD/tests/adapter), for creators and maintainers of adapter plugins. These test cases cover basic expected functionality, as well as functionality that frequently requires different implementations across databases. |
There was a problem hiding this comment.
| Building on top of that foundation, we have also built and made available a [package of reusable adapter test cases](https://github.com/dbt-labs/dbt-core/tree/HEAD/tests/adapter), for creators and maintainers of adapter plugins. These test cases cover basic expected functionality, as well as functionality that frequently requires different implementations across databases. | |
| We've also built and made available a [package of reusable adapter test cases](https://github.com/dbt-labs/dbt-core/tree/HEAD/tests/adapter), for creators and maintainers of adapter plugins. These test cases cover basic expected functionality, as well as functionality that frequently requires different implementations across databases. |
|
|
||
| If you run into an issue with the core framework, or the basic/optional test cases—or if you've written a custom test that you believe would be relevant and useful for other adapter plugin developers—please open an issue or PR in the `dbt-core` repository on GitHub. | ||
|
|
||
| ## Getting started: basic tests |
There was a problem hiding this comment.
| ## Getting started: basic tests | |
| ## Getting started running basic tests |
|
|
||
| In this example, we'll set up a basic dbt project using `pytest` fixtures, run some simple commands, and ensure they succeed (or fail) as we expect. To that end, we'll be using the `run_dbt` utility defined within the `tests` module of `dbt-core`. | ||
|
|
||
| (For the sake of simplicity, the example assumes that we've already installed and configured `pytest`. We'll show that setup in more detail below.) |
There was a problem hiding this comment.
| (For the sake of simplicity, the example assumes that we've already installed and configured `pytest`. We'll show that setup in more detail below.) | |
| This example assumes that you've already installed and configured `pytest`. You can see how to do this in [Getting started running basic tests](#getting-started-running-basic-tests) |
|
|
||
| The [pytest usage docs](https://docs.pytest.org/how-to/usage.html) offer guidance and a full command reference. In our experience, it can be particularly helpful to use the `-s` flag (or `--capture=no`) to print logs from the underlying dbt invocations, and to step into an interactive debugger if you've added one. You can also use environment variables to set [global dbt configs](global-configs), such as `DBT_DEBUG` (to show debug-level logs). | ||
|
|
||
| ## Testing your adapter |
There was a problem hiding this comment.
I'm flexible on this title just trying to capture what this section involves and it's more background info than steps for testing your adapter.
| ## Testing your adapter | |
| ## Understanding adapter testing |
| 2. Set up and configure pytest | ||
| 3. Define test cases | ||
|
|
||
| ### Install dependencies |
There was a problem hiding this comment.
Maybe we can come back later and number the steps in these procedures. I'm leaving them as-is for now.
jtcohen6
force-pushed
the
new-adapter-testing-framework
branch
from
5711d39
to
4803f74
Compare
jtcohen6
force-pushed
the
new-adapter-testing-framework
branch
from
4803f74
to
a4b63a6
Compare
|
FYI @dataders @runleonarun @gshank I'm going to merge this in so that we can have a first cut live for the community prerelease announcement. Two housekeeping notes:
https://docs.getdbt.com/docs/contributing/testing-a-new-adapter?version=1.1 |
| Those utilities allow you to do three basic things: | ||
| 1. **Quickly set up a dbt "project."** Define project resources via methods such as `models()` and `seeds()`. Use `project_config_update()` to pass configurations into `dbt_project.yml`. | ||
| 2. **Define a sequence of dbt commands.** The most important utility is `run_dbt()`, which returns the [results](dbt-classes#result-objects) of each dbt command. It takes a list of CLI specifiers (subcommand + flags), as well as an optional second argument, `expect_pass=False`, for cases where you expect the command to fail. | ||
| 3. **Validate the results of those dbt commands.** For example, `check_relations_equal()` asserts that two database objects have the same structure and content. You can also write your own `assert` statements, by inspecting the results of a dbt command, or querying arbitrary database objects with `project.run_sql()`. |
There was a problem hiding this comment.
Should we consider add a link to this file so folks can find out the latest functions we provide?
| The **[`tests` module](https://github.com/dbt-labs/dbt-core/tree/HEAD/core/dbt/tests)** within `dbt-core` includes basic utilities for setting up pytest + dbt. These are used by all "pre-built" functional tests, and make it possible to quickly write your own tests. | ||
|
|
||
| Those utilities allow you to do three basic things: | ||
| 1. **Quickly set up a dbt "project."** Define project resources via methods such as `models()` and `seeds()`. Use `project_config_update()` to pass configurations into `dbt_project.yml`. |
There was a problem hiding this comment.
Just synced with @VersusFacit on something related to starting using this testing framework. It would be great if we can highlight two things:
- the concept of pytest fixtures and fixture scope is important when writing tests
- we choose the
classscope for most of our fixtures.
resolves #1256
Description & motivation
Add page: "Testing a new adapter."
Open question: Where should this content live? We want to avoid duplicative work, but also provide the right info to the right people in the right place.
The intended audience for these docs is a person creating a new(ish) adapter plugin for the first time, who wants to stand up initial testing. I think we need other docs on the foundational test framework—going over utilities, fixtures, test classes—for the benefit of OSS contributors (and plugin maintainers) writing new test cases.
Update: I've added a more generic example of a dbt functional test, which I could see going in its own page, &/or in a README here.
Pre-release docs
Now that docs versioning has been merged, I think we should stop doing this: #1261
Checklist
If you added new pages (delete if not applicable):
website/sidebars.js