Releases: dbt-labs/dbt-core
dbt 0.15.2
dbt 0.15.2 (February 2, 2020)
This is a bugfix release.
Features
- Add support for Snowflake OAuth authentication (#2050, #2069)
- Add a -t flag as an alias for
dbt run --target
(#1281, #2057)
Fixes
- Fix for UnicodeDecodeError when installing dbt via pip (#1771, #2076)
- Fix for ability to clean "protected" paths in the
dbt clean
command and improve logging (#2059, #2060) - Fix for dbt server error when
{% docs %}
tags are malformed (#2066, #2067) - Fix for errant duplicate resource errors when models are disabled and partial parsing is enabled (#2055, #2056)
- Fix for errant duplicate resource errors when a resource is included in multiple source paths (#2064, #2065)
Contributors
dbt 0.15.1
dbt 0.15.1 (January 17, 2020)
This is a bugfix release.
Features
- Lazily load database connections (#1584, #1992)
- Support raising warnings in user-space (#1970, #1977) (docs)
- Suppport BigQuery label configuration for models (#1942, #1964) (docs)
- Support retrying when BigQuery models fail with server errors (#1579, #1963) (docs)
- Support sql headers in create table/view statements (#1879, #1967) (docs)
- Support
source snapshot-freshness
command in the dbt server (#2040, #2041) - Support the
-t
shorthand for the--target
flag on the CLI (#1281, #2057)
Fixes
- Fix for catalog generation error when datasets are missing on BigQuery (#1984, #2005)
- Fix for invalid SQL generated when "check" strategy is used in Snapshots with changing schemas (#1797, #2001)
- Fix for gaps in valid_from and valid_to timestamps when "check" strategy is used in Snapshots on some databases (#1736, #1994)
- Fix incorrect thread names in dbt server logs (#1905, #2002)
- Fix for ignored catalog data when user schemas begin with
pg*
on Postgres and Redshift (#1960, #2003) - Fix for poorly defined materialization resolution logic (#1962, #1976)
- Fix missing
drop_schema
method in adapter namespace (#1980, #1983) - Fix incorrect
generated_at
value in the catalog (#1988)
Under the hood
- Fail more gracefully at install time when setuptools is downlevel (#1975, #1978)
- Make the
DBT_TEST_ALT
integration test warehouse configurable on Snowflake (#1939, #1979) - Pin upper bound on
google-cloud-bigquery
dependency to1.24.0
. (#2007) - Remove duplicate
get_context_modules
method (#1996) - Add type annotations to base adapter code (#1982)
Contributors:
dbt 0.15.0
dbt 0.15.0 (November 25, 2019)
Breaking changes
- Support for Python 2.x has been dropped as it will no longer be supported on January 1, 2020
- Compilation errors in .yml files are now treated as errors instead of warnings (#1493, #1751)
- The 'table_name' field field has been removed from Relations
- The existing
compile
andexecute
rpc tasks have been renamed tocompile_sql
andexecute_sql
(#1779, #1798) (docs) - Custom materializations must now manage dbt's Relation cache (docs)
Installation notes:
dbt v0.15.0 uses the psycopg2-binary
dependency (instead of psycopg2
) to simplify installation on platforms that do not have a compiler toolchain installed. pip users should run pip uninstall psycopg2
before upgrading their version of dbt to 0.15.0. If after upgrading you experience segmentation faults, crashes, or installation errors, you can set the DBT_PSYCOPG2_NAME
environment variable to psycopg2
to change the dependency that dbt installs. This may require a compiler toolchain and development libraries.
$ DBT_PSYCOPG2_NAME=psycopg2 pip install dbt
You may also install specific dbt plugins directly by name. This has the advantage of only installing the Python requirements needed for your particular database:
# Installing dbt-snowflake and dbt-bigquery directly bypasses the psycopg2 dependency
$ pip install dbt-snowflake
$ pip install dbt-bigquery
# Installing dbt-postgres and dbt-redshift are also supported
$ pip install dbt-postgres
$ pip install dbt-redshift
Changes
Core
Features
- Add a JSON logger (#1237, #1791) (docs)
- Add structured logging to dbt (#1704, #1799, #1715, #1806)
- Add partial parsing option to the profiles.yml file (#1835, #1836, #1487) (docs)
- Support configurable query comments in SQL queries (#1643, #1864) (docs)
- Support atomic full-refreshes for incremental models (#525, #1682)
- Support snapshot configs in dbt_project.yml (#1613, #1759) (docs)
- Support cache modifications in materializations (#1683, #1770) (docs)
- Support
quote
parameter to Accepted Values schema tests (#1873, #1876) (docs) - Support Python 3.8 (#1886)
- Support filters in sources for
dbt source snapshot-freshness
invocation (#1495, #1776) (docs) - Support external table configuration in yml source specifications (#1784)
- Improve CLI output when running snapshots (#1768, #1769)
Fixes
- Fix for unhelpful error message for malformed source/ref inputs (#1660, #1809)
- Fix for lingering backup tables when incremental models are full-refreshed (#1933, #1931)
- Fix for confusing error message when errors are encountered during compilation (#1807, #1839)
- Fix for logic error affecting the two-argument flavor of the
ref
function (#1504, #1515) - Fix for invalid reference to dbt.exceptions (#1569, #1609)
- Fix for "cannot run empty query" error when pre/post-hooks are empty (#1108, #1719)
- Fix for confusing error when project names shadow context attributes (#1696, #1748)
- Fix for incorrect database logic in docs generation which resulted in columns being "merged" together across tables (#1708, #1774)
- Fix for seed errors located in dependency packages (#1723, #1723)
- Fix for confusing error when schema tests return unexpected results (#1808, #1903)
- Fix for twice-compiled
statement
block contents (#1717, #1719) - Fix for inaccurate output in
dbt run-operation --help
(#1767, #1777) - Fix for file rotation issues concerning the
logs/dbt.log
file (#1863, #1865, #1871) - Fix for missing quotes in incremental model build queries (#1847, #1888)
- Fix for incorrect log level in
printer.print_run_result_error
(#1818, #1823)
Docs
- Show seeds and snapshots in the Project and Database views (docs#37, docs#25, docs#52)
- Show sources in the Database tree view (docs#20, docs#52)
- Show edges in the DAG between models and seeds (docs#15, docs#52)
- Show Accepted Values tests and custom schema tests in the column list for models (docs#52)
- Fix links for "Refocus on node" and "View documentation" in DAG context menu for seeds (docs#52)
Server
- Support docs generation (#1781, #1801)
- Support custom tags (#1822, #1828)
- Support invoking
deps
on the rpc server (#1834, #1837) - Support invoking
run-operation
andsnapshot
on the rpc server (#1875, #1878) - Suppport
--threads
argument tocli_args
method (#1897, #1909) - Support reloading the manifest when a SIGHUP signal is received (#1684, #1699)
- Support invoking
compile
,run
,test
, andseed
on the rpc server ([#1488](https://github.com/fishtown-ana...
dbt 0.14.4
dbt 0.14.4 (November 8, 2019)
This release changes the version ranges of some of dbt's dependencies. These changes address installation issues in 0.14.3 when dbt is installed from pip. You can view the full list of dependency version changes in this commit.
Note: If you are installing dbt into an environment alongside other Python libraries, you can install individual dbt plugins with:
pip install dbt-postgres
pip install dbt-redshift
pip install dbt-snowflake
pip install dbt-bigquery
Installing specific plugins may help mitigate issues regarding incompatible versions of dependencies between dbt and other libraries.
Fixes:
dbt 0.14.3
dbt 0.14.2
dbt 0.14.1
dbt 0.14.1 - (September 3, 2019)
Overview
This release fixes a number of bugs and includes minor improvements to the dbt workflow. These changes include UI improvements for dbt's console output, updates to the dbt Docs website, and general performance improvements.
Note: this release includes an important bugfix for the check
snapshot strategy on Snowflake and BigQuery. See this guide for more information about the issue and resolution.
Links:
Breaking changes
- The undocumented
macros
attribute was removed from thegraph
context variable (#1615)
Features:
- Summarize warnings at the end of dbt runs (#1597, #1654)
- Speed up catalog generation on postgres by using avoiding use of the
information_schema
(#1540) - Docs site updates (#1621)
- Add environment variables for macro debugging flags (#1628, #1629)
- Speed up node selection by making it linear, rather than quadratic, in complexity (#1611, #1615)
- Specify the
application
field in Snowflake connections (#1622, #1623) - Add support for clustering on Snowflake (#634, #1591, #1689) (docs)
- Add support for job priority on BigQuery (#1456, #1673) (docs)
- Add
node.config
andnode.tags
to thegenerate_schema_name
andgenerate_alias_name
macro context (#1700, #1701)
Fixes:
- Fix for reused
check_cols
values in snapshots (#1614, #1709) - Fix for rendering column descriptions in sources (#1619, #1633)
- Fix for
is_incremental()
returning True for models that are not materialized as incremental models (#1249, #1608) - Fix for serialization of BigQuery results which contain nested or repeated records (#1626, #1638)
- Fix for loading seed files which contain non-ascii characters (#1632, #1644)
- Fix for creation of user cookies in incorrect directories when
--profile-dir
or$DBT_PROFILES_DIR
is provided (#1645, #1656) - Fix for error handling when transactions are being rolled back (#1647)
- Fix for incorrect references to
dbt.exceptions
in jinja code (#1569, #1609) - Fix for duplicated schema creation due to case-sensitive comparison (#1651, #1663)
- Fix for "schema stub" created automatically by dbt (#913, #1663)
- Fix for incremental merge query on old versions of postgres (<=9.6) (#1665, #1666)
- Fix for serializing results of queries which return
TIMESTAMP_TZ
columns on Snowflake in the RPC server (#1670) - Fix typo in InternalException (#1640, #1672)
- Fix typo in CLI help for snapshot migration subcommand (#1664)
- Fix for error handling logic when empty queries are submitted on Snowflake (#1693, #1694)
- Fix for non-atomic column expansion logic in Snowflake incremental models and snapshots (#1687, #1690)
- Fix for unprojected
count(*)
expression injected by custom data tests (#1688) - Fix for
dbt run
anddbt docs generate
commands when running against Panoply Redshift (#1479, #1686)
Contributors:
Thanks for your contributions to dbt!
dbt 0.14.0
dbt 0.14.0 - Wilt Chamberlain (July 10, 2019)
Overview
This release adds an rpc server to dbt (docs), replaces Archives with Snapshots (docs, migration guide) and adds new dbt subcommands:
Tests can now be defined with a severity (either warning
or error
) (docs), in addition to a number of other workflow and quality of life improvements.
Links:
Breaking changes
- Stub out adapter methods at parse-time to speed up parsing (#1413)
- Removed support for the
--non-destructive
flag (#1419, #1415) - Removed support for the
sql_where
config to incremental models (#1408, #1351) - Changed
expand_target_column_types
to take a Relation instead of a string (#1478) - Replaced Archives with Snapshots
Features
- Add
run-operation
command which invokes macros directly from the CLI (#1328) (docs) - Add a
dbt ls
command which lists resources in your project (#1436, #467) (docs) - Add Snapshots, an improvement over Archives (#1361, #1175) (docs)
- Add an RPC server via
dbt rpc
(#1301, #1274) (docs) - Made printer width configurable (#1026, #1247) (docs)
- Retry package downloads from the hub.getdbt.com (#1451, #1491)
- Add a test "severity" level, presented as a keyword argument to schema tests (#1410, #1005) (docs)
- Add a
generate_alias_name
macro to configure alias names dynamically (#1363) (docs) - Add a
node
argument togenerate_schema_name
to configure schema names dynamically (#1483, #1463) (docs) - Use
create or replace
on Snowflake to rebuild tables and views atomically (#1101, #1409) - Use
merge
statement for incremental models on Snowflake (#1414, #1307, #1409) (docs) - Add support seed CSV files that start with a UTF-8 Byte Order Mark (BOM) (#1452, #1177)
- Add a warning when git packages are not pinned to a version (#1453, #1446)
- Add logging for
on-run-start
andon-run-end hooks
to console output (#1440, #696) - Add modules and tracking information to the rendering context for configuration files (#1441, #1320)
- Add support for
null
vars, and distinguishnull
vars from unset vars (#1426, #608) - Add support for the
search_path
configuration in Postgres/Redshift profiles (#1477, #1476) (docs (postgres), docs (redshift)) - Add support for persisting documentation as
descriptions
for tables and views on BigQuery (#1031, #1285) (docs) - Add a
--project-dir
path which will invoke dbt in the specified directory (#1549, #1544)
dbt docs Changes
- Add searching by tag name (#32)
- Add context menu link to export graph viz as a PNG (#34)
- Fix for clicking models in left-nav while search results are open (#31)
Fixes
- Fix for unduly long timeouts when anonymous event tracking is blocked (#1445, #1063)
- Fix for error with mostly-duplicate git urls in packages, picking the one that came first. (#1428, #1084)
- Fix for unrendered
description
field as jinja in top-level Source specification (#1484, #1494) - Fix for API error when very large temp tables are created in BigQuery (#1423, #1478)
- Fix for compiler errors that occurred if jinja code was present outside of a docs blocks in .md files (#1513, #988)
- Fix
TEXT
handling on postgres and redshift (#1420, #781) - Fix for compiler error when vars are undefined but only used in disabled models (#1429, #434)
- Improved the error message when iterating over the results of a macro that doesn't exist (#1425, [#1424](https://github.com/fishtown-analyti...
dbt 0.13.1
dbt 0.13.1 (May 13, 2019)
Overview
This release primarily fixes minor bugs introduced in v0.13.0
Features
- Add logging to dbt clean (#1261, #1383, #1391) (@emilieschario)
Fixes
- Add "MaterializedView" relation type to the Snowflake adapter (#1430, #1432) (@adriank-convoy)
- Quote databases properly (#1396, #1402)
- Use "ilike" instead of "=" for database equality when listing schemas (#1411, #1412)
- Pass the model name along in get_relations (#1384, #1388)
dbt docs
- Search by columns (dbt-docs#23) (rmgpinto)
- Support @ selector (dbt-docs#27)
- Fix number formatting on Snowflake and BQ in table stats (dbt-docs#28)
Contributors:
Thanks for your contributions to dbt!
dbt 0.13.0
dbt 0.13.0 - Stephen Girard (March 21, 2019)
Overview
This release provides a stable API for building new adapters and reimplements dbt's adapters as "plugins". Additionally, a new adapter for Presto was added using this architecture. Beyond adapters, this release of dbt also includes Sources which can be used to document and test source data tables. See the full list of features added in 0.13.0 below.
Links:
Installation instructions
dbt v0.13.0 is available on Homebrew, PyPi, and in dbt Cloud.
# Homebrew
$ brew install dbt
# pip
$ pip install dbt
Note: If you're upgrading from an earlier version of dbt using pip
, you may need to install dbt with the -I
flag. This flag is required due to changes in the structure of the dbt package.
$ pip install -I dbt
For full installation instructions, consult the docs
Breaking Changes
- Version 1 schema.yml specs are no longer implemented. Please use the version 2 spec instead (migration guide)
{{this}}
is no longer implemented foron-run-start
andon-run-end
hooks. Use{{ target }}
or anon-run-end
context variable instead (#1176, implementing #878)- A number of materialization-specific adapter methods have changed in breaking ways. If you use these adapter methods in your macros or materializations, you may need to update your code accordingly.
- query_for_existing - removed, use get_relation instead.
- get_missing_columns - changed to take
Relation
s instead of schemas and identifiers - expand_target_column_types - changed to take a
Relation
instead of schema, identifier - get_relation - added a
database
argument - create_schema - added a
database
argument - drop_schema - added a
database
argument
Deprecations
- The following adapter methods are now deprecated, and will be removed in a future release:
- get_columns_in_table - deprecated in favor of get_columns_in_relation
- already_exists - deprecated in favor of get_relation
Features
- Add
source
s to dbt, use them to calculate source data freshness (docs ) (#814, #1240) - Add support for Presto (docs, repo) (#1106)
- Add
require-dbt-version
option todbt_project.yml
to state the supported versions of dbt for packages (docs) (#581) - Add an output line indicating the installed version of dbt to every run (#1134)
- Add a new model selector (
@
) which build models, their children, and their children's parents (docs) (#1156) - Add support for Snowflake Key Pair Authentication (docs) (#1232)
- Support SSO Authentication for Snowflake (docs) (#1172)
- Add support for Snowflake's transient tables (default=true) (docs) (#946)
- Capture build timing data in
run_results.json
to visualize project performance (#1179) - Add CLI flag to toggle warnings as errors (docs) (#1243)
- Add tab completion script for Bash (docs) (#1197)
- Added docs on how to build a new adapter (docs) (#560)
- Use new logo (#1349)
Fixes
- Fix for Postgres character columns treated as string types (#1194)
- Fix for hard to reach edge case in which dbt could hang (#1223)
- Fix for
dbt deps
in non-English shells (#1222) - Fix for over eager schema creation when models are run with
--models
(#1239) - Fix for
dbt seed --show
(#1288) - Fix for
is_incremental()
which should only returnTrue
if the target relation is atable
(#1292) - Fix for error in Snowflake table materializations with custom schemas (#1316)
- Fix errored out concurrent transactions on Redshift and Postgres (#1356)
- Fix out of order execution on model select (#1354, #1355)
- Fix adapter macro namespace issue (#1352, #1353)
- Re-add CLI flag to toggle warnings as errors (#1347)
- Fix release candidate regression that runs run hooks on test invocations (#1346)
- Fix Snowflake source quoting (#1338, #1317, #1332)
- Handle unexpected max_loaded_at types (#1330)
Under the hood
- Replace all SQL in Python code with Jinja in macros (#1204)
- Loosen restrictions of boto3 dependency (#1234)
- Rewrote Postgres introspective queries to be faster on large databases (#1192)
Contributors:
Thanks for your contributions to dbt!
- @patrickgoss #1193
- @brianhartsock #1191
- @alexyer #1232
- @adriank-convoy #1224
- @mikekaminsky #1216
- @vijaykiran #1198, #1199
Stephen Girard
Can you imagine buying the Federal Reserve? That’s how rich Stephen Girard was. The First Bank of the United States, Alexander Hamilton’s brainchild, ran out its charter in 1811 and Girard decided it was a steal. Estimated to be the 4th richest American ever, he just up and bought it.
Not satisfied with one bank, Girard went for more. With the British invading in 1812, Girard put everything on the line and underwrote nearly all the US war bonds. Without this capital, we would likely all have British accents today. In return, the US made Girard a major shareholder of the Second Bank of the United States when it was founded.
Buying government bonds: dépassé.
Buying governments: branché.
Girard, who had no children, was one of America’s biggest philanthropists. He single-handedly endowed the 43-acre Girard College to educate “poor, male, white orphans,” inspired by his own hard-scrabble early life. Thankfully, its target demographic has since broadened—its first African-American student was admitted in 1968 and its first female student was admitte...