Releases: sqlfluff/sqlfluff
[3.0.0a4] - 2023-12-05
Highlights
This release makes a breaking change to the serialized output of the CLI (and by extension, any of the serialized outputs of the API).
-
The serialised output for
sqlfluff lintnow contains more information about the span of linting issues and initial proposed fixes. Beside the new fields, the original fields ofline_posandline_nohave been renamed tostart_line_posandstart_line_no, to distinguish them from the new fields startingend_*. -
The default
annotation_levelset by the--annotation-leveloption on thesqlfluff lintcommand has been changed fromnoticetowarning, to better distinguish linting errors from warnings, which always now have the level ofnotice. This is only relevant when using thegithub-annotationorgithub-annotation-nativeformats.
What’s Changed
- Build out rule and fix serialisation #5364 @alanmcruickshank
- Add warning about github actions annotations limit #5450 @alanmcruickshank
- chore: remove unused line initialization in ParseContext #5448 @gvozdvmozgu
- Refine CLI testing fixture #5446 @alanmcruickshank
- Update black linting #5447 @alanmcruickshank
[3.0.0a3] - 2023-11-29
Highlights
This brings no further breaking changes on top of 3.0.0a2, but instead releases a few of the more minor fixes flowing through while other breaking changes are staged in. In particular there are a few dialect improvements for Snowflake, TSQL, SQLite and Databricks alongside a few further improvements to deployment scripts off the back of earlier changes for 3.x.
What’s Changed
- fix assertion in
test__api__lint_string_specific_exclude_single#5437 @gvozdvmozgu - Databricks CLUSTER BY and OPTIMIZE #5436 @greg-finley
- TSQL: ON DELETE NO ACTION #5434 @greg-finley
- Snowflake dynamic table #5435 @greg-finley
- Support parsing CONSTRAINT definitions when creating Delta Live Tables in SparkSQL/Databricks #5438 @rocwang
- adds few fixes for databricks/sparksql #5431 @markbaas
- TSQL: CREATE USER {FOR|FROM} LOGIN #5426 @greg-finley
- Snowflake Create table order/noorder #5421 @greg-finley
- Simplify Snowflake regexes #5419 @greg-finley
- Permit .* after each tbl_name in multi-table delete syntax #5408 @yoichi
- Fix snowflake add search optimization grant #5412 @jongracecox
- ANSI: Allow combination of UNION clause and WITH clause #5413 @yoichi
- SQLite: Allow block comments to be terminated by end of input #5400 @WittierDinosaur
- Update publish github action to use kebab case #5392 @alanmcruickshank
New Contributors
- @jongracecox made their first contribution in #5412
- @markbaas made their first contribution in #5431
- @rocwang made their first contribution in #5438
- @gvozdvmozgu made their first contribution in #5437
[3.0.0a2] - 2023-11-09
Highlights
The initial 3.0.0a1 release failed to build a docker image, this resolves that issue.
What’s Changed
- Update publish actions and Dockerfile. #5390 @alanmcruickshank
[2.3.5] - 2023-10-27
Highlights
This is a fairly minor release, primarily bugfixes and dialect improvements.
For python API users, there's the addition of a public method on the FluffConfig object allowing the construction of a config object from multiple strings to mimic the effect of nested config files in the CLI.
This release also includes a selection of internal refactoring and reorganisation to support future development work.
This also sees the first contributions by @ShubhamJagtap2000 & @kang8, particularly notable in that both were contributions to SQLFluff documentation! 🎉🎉🏆🎉🎉
What’s Changed
- One (very) small typing improvements #5355 @alanmcruickshank
- Unpick dependencies between modules in
sqlfluff.core#5348 @alanmcruickshank - Resolve SparkSQL re-parsing issue + test validation in test suite. #5351 @alanmcruickshank
- Snowflake: Support ALTER MASKING POLICY #5350 @jmks
- Add a public API for nesting config strings. #5349 @alanmcruickshank
- Update handling of dbt compilation errors #5345 @alanmcruickshank
- Snowflake - Extend Column Default Constraint #5343 @WittierDinosaur
- Fix the dbt anchor link in the realworld documentation #5341 @kang8
- Update README.md #5340 @ShubhamJagtap2000
- Logic to render variants of Jinja templates for more coverage. #5339 @alanmcruickshank
- Templater slicing refactoring of
RawFileSlice#5338 @alanmcruickshank - BigQuery: Support multiple statements in the
BEGIN..EXCEPTION..END#5322 @abdel - Remove codecov traces #5337 @alanmcruickshank
New Contributors
- @ShubhamJagtap2000 made their first contribution in #5340
- @kang8 made their first contribution in #5341
[2.3.4] - 2023-10-17
Highlights
This is a fairly small bugfix release, mostly to resolve a bug introduced in 2.3.3 with commas and LT09. This also includes a couple of additional small performance improvements and some dialect improvements for Oracle, BigQuery and MySQL.
Thanks in particular to @bonnal-enzo who made their first contribution as part of this release 🎉🎉🏆🎉🎉.
What’s Changed
- Commas fix in LT09 #5335 @alanmcruickshank
- UUID Comparisons #5332 @alanmcruickshank
- Two depth map performance improvements #5333 @alanmcruickshank
- Stash parent idx with parent reference #5331 @alanmcruickshank
SettoFrozenSetin segment class_types #5334 @alanmcruickshank- Add support for ANY_VALUE( _ HAVING MIN/MAX _ ) to BigQuery dialect #5321 @bonnal-enzo
- Fix parsing error when using quoted slash in Oracle #5323 @joaostorrer
- Add support for functions and procedures calls via database link in Oracle #5326 @joaostorrer
- Fix parsing error with table name '_' in MySQL #5324 @joaostorrer
New Contributors
- @bonnal-enzo made their first contribution in #5321
[2.3.3] - 2023-10-13
Highlights
There's a lot in this release. Most of it is under the covers and so shouldn't cause any breaking changes for most users. If your use case depends on some of the internals of SQLFluff, you may find some breaking changes. The bigger changes are:
- Python 3.12 support is now official (although older releases may also work as only a few changes were required for full 3.12 support).
- We've done a significant re-write of the parsing engine to remove some unnecessary segment manipulation and get us closer to "single pass" parsing. This changes the internal API being used on any
.match()methods, and also removes theparse_grammarattribute on any dialect segments. We are not aware of any 3rd party libraries which rely on these APIs however and so have not triggered a more major release. These lead to significant performance improvements during parsing. - Standardisation of terminators in the parser, and the introduction of the
ParseModeoption has enabled the removal of theStartsWith,GreedyUntilandEphemeralSegmentparser classes. - Several validation checks have been revised in this release, which should both improve performance (by reducing duplication), but also be more effective in preventing the application of any fixes which would result in unparsable files.
Alongside the big things this also includes a host of bugfixes, dialect improvements and CI/testing improvements.
This release also sees a bumper crop of new contributors, thanks to @dehume, @andychannery, @Kylea650, @robin-alphasophia, @jtbg, @r-petit, @bpfaust, @freewaydev & @abdel who all made the first contibutions in this release! 🎉🎉🎉
What’s Changed
- Oracle space between alias and column reference #5313 @joaostorrer
- Don't apply LT05 on templated rebreak locations #5096 #5318 @alanmcruickshank
- Disable JJ01 unless jinja active #5319 @alanmcruickshank
- Cache the
BaseSegmenthash in reflow #5320 @alanmcruickshank - Better error reporting for invalid macros #5317 @alanmcruickshank
- Add support for begin atomic functions in Postgres #5316 @joaostorrer
- Fix parsing when statement uses plus_sign_join and function in Oracle #5315 @joaostorrer
- Update rule docs with correct config #5314 @alanmcruickshank
- Resolve #5258. More robust algorithm for multiline fix. #5309 @alanmcruickshank
- BigQuery: Add support for
BEGIN..EXCEPTION...ENDblock #5307 @abdel - Refine placement of metas around templated blocks #5294 @alanmcruickshank
- Extend ruff checking to docstring rules #5302 @alanmcruickshank
- Fix for strange TSQL bugs #5306 @alanmcruickshank
- Staging PR for #5282 #5305 @alanmcruickshank
- Resolve some strange whitespace indentation behaviour #5292 @alanmcruickshank
- Simplify
_process_lint_result#5304 @alanmcruickshank - Performance improvement on segment comparison #5303 @alanmcruickshank
- Refactor LT09 #5299 @alanmcruickshank
- Change drop function to allow DropBehaviourGrammar with space after function name #5295 @joaostorrer
- Resolve click import options on autocomplete #5293 @alanmcruickshank
- Updated docstrings with missing args/returns/etc info, added missing docstrings, minor formatting fixes. #5278 @freewaydev
- Use ruff rule I replace isort #5289 @zhongjiajie
- Snowflake: Parse ALTER DATABASE statement #5284 @jmks
- Snowflake: Parse ALTER ACCOUNT statements #5283 @jmks
- Snowflake: create AlterProcedureStatementSegment #5291 @moreaupascal56
- Rewrite of matching interface #5230 @alanmcruickshank
- Follow noqa in block comments #5133 @daviewales
- Fix insert on conflict with function in Postgres #5286 @joaostorrer
- Add support for Pivot and Unpivot clauses in Oracle #5285 @joaostorrer
- Adding "create table as" for greenplum dialect #5173 @bpfaust
- Update CI to python 3.12 #5267 @alanmcruickshank
- Snowflake: Add CreateResourceMonitorStatementSegment & AlterResourceMonitorStatementSegment #5272 @moreaupascal56
- [TSQL] Add create fulltext index statement segment class #5274 @r-petit
- Snowflake: Add CreateSequenceStatementSegment & AlterSequenceStatementSegment #5270 @moreaupascal56
- Add CommaSegment to AlterWarehouseStatementSegment SET clause #5268 @moreaupascal56
- Snowflake: Parse EXECUTE IMMEDIATE clause #5275 @jmks
- TSQL: Add missing
HISTORY_RETENTION_PERIODsequence to the table option segment #5273 @r-petit - Snowflake: Fix ScalingPolicy and WarehouseType Refs in WarehouseObjectProperties and use ObjectReferenceSegment in AlterWarehouseStatementSegment #5264 @moreaupascal56
- Finish the removal of
GreedyUntil#5263 @alanmcruickshank - Add support for date operations with intervals in Oracle #5262 @joaostorrer
- Change RawSegment
typetoinstance_types#5253 @alanmcruickshank - Revise MatchableType -> Matchable #5252 @alanmcruickshank
- Unnest fixing and re-address validation triggers #5249 @alanmcruickshank
- Resolves #5174: Snowflake alter table constraint #5247 @andychannery
- Bring together the generic segments #5243 @alanmcruickshank
- minor: update docs with correct link to airflow ds_filter #5244 @jtbg
- #5245 - Snowflake dialect: Adds support for variable definitions in scripting blocks #5246 @robin-alphasophia
- Introduce "word" segment #5234 @alanmcruickshank
- #5239 Added (basic) support for properly linted Snowflake scripting #5242 @robin-alphasophia
- Allow Snowflake pipe integration to be a quoted or unquoted #5241 [@Kylea650](https://g...
[2.3.2] - 2023-09-10
Highlights
Much of this release is internal optimisations and refactoring. We're in the process of upgrading some quite old code in the parser, most of which should not be visible to end users (apart from perhaps some performance improvements!).
This release also allows missing template variables in the placeholder templater to be automatically filled with the name of the variable rather than raising an error (see: #5101).
Beyond that this includes some dialect improvements for DuckDB, SparkSQL, Snowflake, Redshift & Postgres.
Thanks particularly to @shyaginuma, @Fullcure3, @adilkhanekt & @pilou-komoot who made their first contributions as part of this release. 🎉🎉🎉
What’s Changed
- Allow not specifying parameters names when using placeholder templater #5101 @shyaginuma
- Update coverage job to run in the right conditions #5183 @alanmcruickshank
- Duckdb: UNION BY NAME #5176 @greg-finley
- Output coverage report direct to PR #5180 @alanmcruickshank
- Upgrades to the parse fixture generation script #5182 @alanmcruickshank
- Refactor of Sequence match #5177 @alanmcruickshank
- Simplify Greedy Match #5178 @alanmcruickshank
- Quality of life improvements on parse fixture script #5179 @alanmcruickshank
- Lift and shift matching algorithms #5170 @alanmcruickshank
- Capitalise boolean values in example configs, for consistency #5175 @pilou-komoot
- Pull terminator setting up into the base grammar #5172 @alanmcruickshank
- Flip the if in sequence and un-nest #5171 @alanmcruickshank
- SparkSQL: Support CACHE TABLE without query #5165 @reata
- Remove configurable
enforce_whitespace_preceding_terminator#5162 @alanmcruickshank - Adding optional sequence block for columns parsing in Snowflake external tables #5157 @adilkhanekt
- SparkSQL: Support ALTER TABLE SET LOCATION without partition spec #5168 @reata
- Tighten terminators on
Delimited#5161 @alanmcruickshank terminator>terminatorson StartsWith #5152 @alanmcruickshank- Redshift: Support SELECT INTO #5159 @reata
- Duckdb: Integer division #5154 @greg-finley
terminator>terminatorson Delimited grammar #5150 @alanmcruickshank- Tests for unparsable sections #5149 @alanmcruickshank
- Un-nest the delimited match method #5147 @alanmcruickshank
- Grammar .copy() assert no unexpected kwargs #5148 @alanmcruickshank
- Postgres: CLUSTER #5146 @greg-finley
- Postgres alter policy #5138 @Fullcure3
New Contributors
- @Fullcure3 made their first contribution in #5138
- @adilkhanekt made their first contribution in #5157
- @pilou-komoot made their first contribution in #5175
- @shyaginuma made their first contribution in #5101
[2.3.1] - 2023-08-29
Highlights
This release is primarily a performance release, with most major changes aimed at the linting and fixing phases of operation. Most of the longest duration rules (excepting the layout rules) should see noticeable speed improvements.
Alongside those changes, there are a selection of bugfixes and dialect improvements for Oracle, PostgreSQL, Snowflake & TSQL.
What’s Changed
- Postgres: Update returning with alias #5137 @greg-finley
- Reduce copying on _position_segments (improves
fix) #5119 @alanmcruickshank - Import rationalisation #5135 @alanmcruickshank
- Select Crawler Refactor: Part 3 #5115 @alanmcruickshank
- Add support for comparison operators with space in Oracle #5132 @joaostorrer
- Snowflake support for bracketed query after
EXCEPT#5126 @ulixius9 - Treatment of null literals. #5099 #5125 @alanmcruickshank
- Allow double-quoted parameters in create procedure #5131 @greg-finley
- Fix coverage & mypy #5134 @alanmcruickshank
- Ensure Unparsable can be given position. #5117 @alanmcruickshank
- Reduce copying in LintFix instantiation #5118 @alanmcruickshank
- Optimise crawl behaviour of JJ01 #5116 @alanmcruickshank
- Simplify rules with improvement to SegmentSeeker #5113 @alanmcruickshank
- Refactor AM07 #5112 @alanmcruickshank
- Select Crawler Refactor: Part 2 #5110 @alanmcruickshank
- Add support to Hierarchical Queries in Oracle #5108 @joaostorrer
- ✅ Strict MyPy for sqlfluff.core.parser #5107 @alanmcruickshank
- Free up pydocstyle again #5109 @alanmcruickshank
- Postgres: Allow CREATE TABLE INHERITS with no new columns #5100 @greg-finley
- Strict mypy in parser.segments #5094 @alanmcruickshank
- Select Crawler Refactor: Part 1 #5104 @alanmcruickshank
- RF01 & recursive_crawl improvements #5102 @alanmcruickshank
- fix new more restrictive tox #5103 @alanmcruickshank
- Make Day as Non Reserved Keyword #5062 @ulixius9
[2.3.0] - 2023-08-14
Highlights
This release brings one new dialect, two new rules and some changes to the CLI:
- We now support the trino dialect. This is a first version of support, so do post any issues on GitHub in the usual way. This was also the first contribution to the project from @efung 🏆.
ST09/structure.join_condition_order: Which checks whether tables referenced inJOINclauses are referenced in the order of their definition. By default this means that in theONclause, the column referencing the table in theFROMclause should come before the column referencing the table in theJOINclause (e.g.... FROM a JOIN b on a.c = b.c). This rule was also the first contribution to the project from @thibonacci 🏆.AL08/aliasing.unique.column: Which checks that column aliases and names are not repeated within the sameSELECTclause. This is normally an error as it implies the same column has been imported twice, or that two expressions have been given the same alias.- The
--profileroption onsqlfluff parsehas been removed. It was only present on theparsecommand and notlintorfix, and it is just as simple to invoke the pythoncProfilerdirectly. - The
--recursecli option andsqlfluff.recurseconfiguration option have both been removed. They both existed purely for debugging the parser, and were never used in a production setting. The improvement in other debugging messages when unparsable sections are found means that this option is no longer necessary.
Along side these more significant changes this also includes:
- Performance optimisations for
AL04,AL05,AM04,RF01&ST05which cumulatively may save up to 30% on the total time spend in the linting phase for some projects. - Dialect improvements for Oracle & TSQL.
What’s Changed
- Remove IdentitySet #5093 @alanmcruickshank
- Stricter typing in smaller sqlfluff.core.parser #5088 @alanmcruickshank
- Preliminary support of Trino dialect #4913 @efung
- Rename ST09 #5091 @alanmcruickshank
- TSQL: Fix Clustered Index asc/desc #5090 @greg-finley
- Parent references and more efficient path_to #5076 @alanmcruickshank
- New Rule: AL08 - column aliases must be unique #5079 @alanmcruickshank
- Add support for fetch first row(s) only in Oracle #5089 @joaostorrer
- Fix bug around quoted identifiers for ST09 #5087 @thibonacci
- Add strict typing to the templating tracer #5085 @WittierDinosaur
- Remove recurse config #5065 @alanmcruickshank
- ✅ Strictly type dialect #5067 @pwildenhain
- Add new rule ST09: Joins should list the table referenced earlier (default)/later first #4974 @thibonacci
- Remove the internal cProfiler option #5081 @alanmcruickshank
- Optimisation on select analysis #5082 @alanmcruickshank
New Contributors
- @thibonacci made their first contribution in #4974
- @efung made their first contribution in #4913
[2.2.1] - 2023-08-09
Highlights
This is primarily a bugfix release for 2.2.0 which introduced a bug in the exit_code returned by linting commands which ignored errors while setting processes > 1.
In addition to that this release introduces bugfixes for:
- Errors raised by two specific
dbtexceptions. - Issues with unwanted logging output when using
-f yamlor-f jsonalongside thedbttemplater.
This also introduces dialect improvements for Oracle and for LIMIT clauses.
Thanks also to @adityapat3l who made their first contribution as part of this release! 🎉🎉🎉
What’s Changed
- Split apart the grammar tests #5078 @alanmcruickshank
- Resolve pickling of errors #5066 #5074 @alanmcruickshank
- Better context based tracking #5064 @alanmcruickshank
- fixing limit handling for bracketed arithmathic operations #5068 @adityapat3l
- Never run in multiprocessing mode with only 1 file. #5071 @alanmcruickshank
- Add dbt 1.6 tests #5073 @alanmcruickshank
- Handle two kinds of dbt errors more gracefully #5072 @alanmcruickshank
- Try to silence dbt logging #5054 #5070 @alanmcruickshank
- Move
_prune_optionswithin_longest_trimmed_match. #5063 @alanmcruickshank - Fix issue 4998 - Add backup and auto refresh grammar to redshift materialized view #5060 @adityapat3l
- Add mypy strict typing for sqlfluff.core.rules #5048 @pwildenhain
- ⬆️ Bump mypy version in pre-commit #5055 @pwildenhain
- Add SQL Plus bind variable support (Oracle) #5053 @joaostorrer
New Contributors
- @adityapat3l made their first contribution in #5060