Merge branch 'main' into drop-function

.github/workflows/ci-tests.yml

@@ -208,9 +208,15 @@ jobs:
     - name: Run the version test
       run: |
         sqlfluff --version
-    - name: Run a simple select parse test
+    - name: Run a simple select parse test via stdin
       run: |
         echo "select 1" | sqlfluff parse -
-    - name: Run a simple select lint test
+    - name: Run a simple select lint test via stdin
       run: |
         echo "select 1" | sqlfluff lint -
+    - name: Run a simple select parse test via file
+      run: |
+        sqlfluff parse <(echo "select 1")
+    - name: Run a simple select lint test via file
+      run: |
+        sqlfluff lint <(echo "select 1")

CONTRIBUTING.md

@@ -75,36 +75,28 @@ changes.
 
 ### Developing and Running SQLFluff Locally
 
-To use your local development branch of SQLFluff, I recommend you use a virtual
-environment. e.g:
-
+The simplest way to set up a development environment is to use `tox`.
+First ensure that you have tox installed (windows users may have to replace `python3` with `py`):
 ```shell
-python3 -m venv .venv
-source .venv/bin/activate
+python3 -m pip install -U tox
 ```
 
-> `python3 -m venv .venv` creates a new virtual environment (in current working
-> directory) called `.venv`.
-> `source .venv/bin/activate` activates the virtual environment so that packages
-> can be installed/uninstalled into it. [More info on venv](https://docs.python.org/3/library/venv.html).
-
-Once you are in a virtual environment, run:
-
+A virtual environment can then be created and activated by running:
 ```shell
-pip install -U tox
-pip install -Ur requirements.txt -Ur requirements_dev.txt
-pip install -e .
+tox -e py --devenv .venv
+source .venv/bin/activate
 ```
+(The `py` environment defaults to the python version used
+to install tox, however any version you want can be installed
+by replacing `py` with `py37`, `py39`, `dbt020-py38`, etc. If
+you are planning development on or using the dbt templater
+you may wish to chose one of the dbt environments.)
 
-> `pip install tox` installs the `tox` testing package. It is not part of the project
-> dependencies as the test suite runs through `tox`.
-
-> `pip install -Ur requirements.txt -Ur requirements_dev.txt` installs the project dependencies
-> as well as the dependencies needed to run linting, formatting, and testing commands. This will
-> install the most up-to-date package versions for all dependencies (-U).
+Windows users should call `.venv\Scripts\activate` rather than `source .venv/bin/activate`.
 
-> `pip install -e .` installs the package using a link to the source code so that any changes
-> which you make will immediately be available for use.
+This virtual environment will already have the package installed in editable mode for you, as well as
+`requirements_dev.txt` and `plugins/sqlfluff-plugin-example`. Additionally if a dbt virtual environment
+was specified, you will also have `dbt-core`, `dbt-postgres`, and `plugins/sqlfluff-templater-dbt` available.
 
 ### Developing plugins
 
@@ -122,10 +114,9 @@ pip install -e plugins/sqlfluff-templater-dbt/.
 
 ### Testing
 
-To test locally, SQLFluff uses `tox`, which means you can build locally using...
+To test locally, SQLFluff uses `tox`. The test suite can be run via:
 
 ```shell
-pip install tox
 tox
 ```
 
@@ -135,31 +126,31 @@ Python version, so you can always specify a particular environment. For example,
 if you are developing in Python 3.8 you might call...
 
 ```shell
-tox -e generate-fixture-yml,py38,linting
+tox -e generate-fixture-yml,py38,linting,mypy
 ```
 
 ...or if you also want to see the coverage reporting...
 
 ```shell
-tox -e generate-fixture-yml,cov-init,py38,cov-report,linting
+tox -e generate-fixture-yml,cov-init,py38,cov-report,linting,mypy
 ```
 
-> NB: The `cov-init` task clears the previous test results, the `py36` environment
-> generates the results for tests in that Python version and the `cov-report-nobt`
+> NB: The `cov-init` task clears the previous test results, the `py38` environment
+> generates the results for tests in that Python version and the `cov-report`
 > environment reports those results out to you (excluding dbt).
 
-You can also run specific tests only by making use of `pytest -k` to match test.
-For example below will only run the tests for rule L012 which is much faster while
-working on an issue, before running full tests at the end:
+`tox` accepts `posargs` to allow you to refine your test run, which is much
+faster while working on an issue, before running full tests at the end.
+For example, you can run specific tests by making use of the `-k` option in `pytest`:
 
 ```
-pytest -k L012 -v test
+tox -e py38 -- -k L012 test
 ```
 
-Alternatively, you can also run tests from specific directory or file only:
+Alternatively, you can also run tests from a specific directory or file only:
 ```
-pytest test/cli
-pytest test/cli/commands_test.py
+tox -e py38 -- test/cli
+tox -e py38 -- test/cli/commands_test.py
 ```
 
 To run the dbt-related tests you will have to explicitly include these tests:

docs/source/indentation.rst

@@ -49,7 +49,7 @@ So, without further ado, here are the principles we think apply to indentation:
 
 2. **Line Length**. Long lines are hard to read and many SQL guidelines
    include a line length restriction. This is (of course) configurable, but
-   the default is 80 characters (in line with the `Fishtown SQL style guide`_.)
+   the default is 80 characters (in line with the `dbt Labs SQL style guide`_.)
 
 3. **Bracket behaviour**. For brackets there are three accepted ways:
 
@@ -158,7 +158,7 @@ of the :code:`JOIN` expression.
 
 Semantically, a :code:`JOIN` expression is part of the :code:`FROM` expression
 and therefore would be expected to be indented. However according to many
-of the most common SQL style guides (including the `Fishtown SQL style guide`_
+of the most common SQL style guides (including the `dbt Labs SQL style guide`_
 and the `Mozilla SQL style guide`_) the :code:`JOIN` keyword is expected to at
 the same indent as the :code:`FROM` keyword. By default, *SQLFluff* sides with
 the current consensus, which is to *not* indent the :code:`JOIN` keyword,
@@ -213,6 +213,6 @@ to indentation. However, if you have other versions of indentation which are
 supported by published style guides, then please submit an issue on GitHub
 to have that variation supported by *SQLFluff*.
 
-.. _`Fishtown SQL style guide`: https://github.com/fishtown-analytics/corp/blob/master/dbt_coding_conventions.md#sql-style-guide
+.. _`dbt Labs SQL style guide`: https://github.com/dbt-labs/corp/blob/master/dbt_style_guide.md
 .. _`Mozilla SQL style guide`: https://docs.telemetry.mozilla.org/concepts/sql_style.html#joins
 .. _`Baron Schwartz`: https://www.xaprb.com/blog/2006/04/26/sql-coding-standards/

requirements.txt

@@ -14,6 +14,5 @@ appdirs
 # dataclasses backport for python 3.6
 dataclasses; python_version < '3.7'
 chardet
-types-chardet
 toml
 tqdm

requirements_dev.txt

@@ -17,3 +17,6 @@ mypy
 typing_extensions
 types-toml
 types-pkg_resources
+types-chardet
+# Requests is required for the util script
+requests

setup.py

@@ -121,6 +121,8 @@ def read(*names, **kwargs):
         "tblib",
         # For handling progress bars
         "tqdm",
+        # To get the encoding of files.
+        "chardet",
     ],
     entry_points={
         "console_scripts": [

src/sqlfluff/core/default_config.cfg

@@ -89,3 +89,7 @@ forbid_subquery_in = join
 [sqlfluff:rules:L047]  # Consistent syntax to count all rows
 prefer_count_1 = False
 prefer_count_0 = False
+
+[sqlfluff:rules:L052]  # Semi-colon formatting approach.
+semicolon_newline = False
+require_final_semicolon = False

src/sqlfluff/core/rules/config_info.py

@@ -103,6 +103,20 @@
             "Should alias have an explict AS or is implicit aliasing required?"
         ),
     },
+    "semicolon_newline": {
+        "validation": [True, False],
+        "definition": (
+            "Should semi-colons be placed on a new line after the statement end?"
+        ),
+    },
+    "require_final_semicolon": {
+        "validation": [True, False],
+        "definition": (
+            "Should final semi-colons be required? "
+            "(N.B. forcing trailing semi-colons is not recommended for dbt users "
+            "as it can cause issues when wrapping the query within other SQL queries)"
+        ),
+    },
 }