Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: replace pylint, isort, black with ruff (DEV-2972) #653

Merged
merged 96 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
eb44d20
install ruff
jnussbaum Nov 22, 2023
993857a
fix ruff errors
jnussbaum Nov 22, 2023
a85a2a4
fix more ruff errors
jnussbaum Nov 22, 2023
dd5130f
toml
jnussbaum Nov 22, 2023
1cac1e9
poetry remove pylint
jnussbaum Nov 23, 2023
2f3ce30
poetry remove black
jnussbaum Nov 23, 2023
70c1b59
pre-commit: replace black by ruff
jnussbaum Nov 23, 2023
4f61fc8
tests on push: replace black & pylint by ruff
jnussbaum Nov 23, 2023
8b7e6b0
update readme
jnussbaum Nov 23, 2023
d4d72e1
add pyupgrade to ruff rules
jnussbaum Nov 23, 2023
d88cd97
mention correct vscode extension id
jnussbaum Nov 23, 2023
a32236f
remove isort
jnussbaum Nov 23, 2023
f170aaa
update code quality tools docs
jnussbaum Nov 23, 2023
1f366d1
adapt batches
jnussbaum Nov 23, 2023
1361cf7
make it work with basic ruleset
jnussbaum Nov 23, 2023
b924adc
ignore unused imports in all __init__.py files
jnussbaum Nov 23, 2023
97a14c1
replace some pylint ignores by their ruff equivalent
jnussbaum Nov 23, 2023
186f0af
migrate broad-exception-caught to BLE001
jnussbaum Nov 27, 2023
254af50
update link text
jnussbaum Nov 27, 2023
f240412
enable isort
jnussbaum Nov 27, 2023
a5ed974
remove "unnecessary" pylint broad-exception-caught (why aren't they f…
jnussbaum Nov 27, 2023
09a64b5
remove unnecessary pylint ignore in __init__.py
jnussbaum Nov 27, 2023
1dca867
Merge branch 'main' into wip/ruff
jnussbaum Nov 27, 2023
48307e5
UP031 pyupgrade: printf-string-formatting
jnussbaum Nov 27, 2023
424dc4b
remove unused pylint ignore
jnussbaum Nov 27, 2023
5cf1496
replace pylint missing-function-docstring by D102
jnussbaum Nov 27, 2023
0194660
get rid of too-many-instance-attributes
jnussbaum Nov 27, 2023
780fe31
get rid of too-many-public-methods (still experimental in ruff)
jnussbaum Nov 27, 2023
502b1d4
add RUF100: unused-noqa
jnussbaum Nov 27, 2023
cb25549
get rid of too-few-public-methods
jnussbaum Nov 27, 2023
92264d5
migrate undocumented-public-class
jnussbaum Nov 27, 2023
9369256
remove unused ignore
jnussbaum Nov 27, 2023
16eb23b
remove unused ignore
jnussbaum Nov 27, 2023
0a5236e
get rid of no-member
jnussbaum Nov 27, 2023
eb26891
D103 pydocstyle: undocumented-public-function
jnussbaum Nov 27, 2023
5af27f4
remove unused ignore
jnussbaum Nov 27, 2023
8d59de1
"redefined-outer-name" is not available in ruff
jnussbaum Nov 27, 2023
1a5c271
get rid of protected-access
jnussbaum Nov 27, 2023
dc48a89
get rid of redefined-outer-name
jnussbaum Nov 27, 2023
f0988f0
expression-not-assigned doesn't exist in ruff
jnussbaum Nov 27, 2023
5694e21
B023 function-uses-loop-variable
jnussbaum Nov 27, 2023
1aef81e
ARG flake8-unused-arguments
jnussbaum Nov 27, 2023
a552da1
get rid of protected-access and import-outside-toplevel
jnussbaum Nov 27, 2023
9423c5f
no implicit reexport in __init__.py
jnussbaum Nov 27, 2023
bd6e715
Revert "no implicit reexport in __init__.py"
jnussbaum Nov 27, 2023
a17d34a
add msg to ignore codes
jnussbaum Nov 27, 2023
8a28043
edit
jnussbaum Nov 27, 2023
d684b55
don't ignore redefined-loop-name globally
jnussbaum Nov 27, 2023
ed480ac
use ruff names instead of pylint names
jnussbaum Nov 27, 2023
24557e0
resolve redefined-builtin
jnussbaum Nov 27, 2023
3fcb92b
add D417 undocumented-param
jnussbaum Nov 27, 2023
d2ab06d
don't ignore PLR0915 too-many-statement globally
jnussbaum Nov 27, 2023
8566849
don't ignore too-many-branches globally
jnussbaum Nov 27, 2023
71df0a2
don't ignore too-many-return-statements globally
jnussbaum Nov 27, 2023
be67d8f
don't ignore collapsible-else-if globally
jnussbaum Nov 27, 2023
25b93b0
don't ignore global-statement globally
jnussbaum Nov 27, 2023
ea0cfaa
edit
jnussbaum Nov 27, 2023
0aab716
activate all RUF rules
jnussbaum Nov 27, 2023
04ac49e
implement ISC
jnussbaum Nov 27, 2023
5ad405f
add YTT and ASYNC
jnussbaum Nov 27, 2023
70461d0
add INP
jnussbaum Nov 27, 2023
9d701aa
add ICN
jnussbaum Nov 27, 2023
7778e09
add PIE
jnussbaum Nov 27, 2023
5b6b4ef
add PT
jnussbaum Nov 27, 2023
2b02d88
add TID, ARG, PGH
jnussbaum Nov 27, 2023
7085ae4
add ISC001 to ignore list
jnussbaum Nov 28, 2023
f85e5fe
remove pydocstyle from coming-soon-linters
jnussbaum Nov 28, 2023
01f4c8e
ignore INP001 in entire testdata dir / rearrange pyproject.toml
jnussbaum Nov 28, 2023
b6e2449
resolve builtin-attribute-shadowing
jnussbaum Nov 28, 2023
a7833fc
resolve merge conflicts ?
jnussbaum Nov 28, 2023
5f04cd0
Merge branch 'main' into wip/ruff
jnussbaum Nov 28, 2023
d4bff53
Merge branch 'main' into wip/ruff
jnussbaum Dec 6, 2023
3c85f60
generate poetry.lock
jnussbaum Dec 6, 2023
f05b2d2
resolve/ignore some ruff problems
jnussbaum Dec 6, 2023
0aa4d4c
remove pylint ignore comments
jnussbaum Dec 6, 2023
1804aa9
adapt CI output format
jnussbaum Dec 6, 2023
47f24fb
restructure: less ignore comments in the code, more in the CI
jnussbaum Dec 6, 2023
664568d
reformat
jnussbaum Dec 6, 2023
fc73d61
Merge branch 'main' into wip/ruff
jnussbaum Dec 6, 2023
0f59bd8
fix
jnussbaum Dec 7, 2023
8ceccea
revert irrelevant reformatting
jnussbaum Dec 7, 2023
8e37bec
remove ignore comments
jnussbaum Dec 7, 2023
5dd1731
Merge branch 'main' into wip/ruff
jnussbaum Dec 13, 2023
043bab4
fix/ignore new violations
jnussbaum Dec 13, 2023
7c52ea5
add config details for vscode extension
jnussbaum Dec 13, 2023
3f39e31
move PLR0913 and PLR2004 from toml to pre-commit + tests-on-push
jnussbaum Dec 13, 2023
97efa0f
put file-wide ignores below imports
jnussbaum Dec 13, 2023
8acbc3f
move PT009 from toml to pre-commit + tests-on-push
jnussbaum Dec 13, 2023
044ee1b
move PT027 from toml to ignore-comments
jnussbaum Dec 13, 2023
c4aadb1
remove PT009 from pre-commit and tests-on-push
jnussbaum Dec 13, 2023
11f38ae
Merge branch 'main' into wip/ruff
jnussbaum Dec 13, 2023
e5cc343
remove unused ignore
jnussbaum Dec 13, 2023
8b0a035
fix PLW2901
jnussbaum Dec 13, 2023
51d729a
fix PLW2901
jnussbaum Dec 13, 2023
2885760
remove ignore comments for A002 (builtin-argument-shadowing)
jnussbaum Dec 13, 2023
2e31d3a
don't ignore import_script.py
jnussbaum Dec 13, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 7 additions & 6 deletions .github/workflows/tests-on-push.yml
Expand Up @@ -63,14 +63,15 @@ jobs:
- name: Install Python dependencies
run: poetry install
# run tests
- name: Linting with Pylint
run: poetry run pylint src test
- name: Linting with ruff
run: >
poetry run ruff check .
--output-format=github
--ignore=A002,A003,B023,D101,D102,D103,PLR0911,PLR0912,PLR0913,PLR0915,PLR2004,PLR5501,PLW0603
- name: Formatting with ruff
run: poetry run ruff format .
- name: Linting with mypy
run: poetry run mypy .
- name: Linting with black
run: poetry run black --check .
- name: Linting with isort
run: poetry run isort . --check --diff
- name: Linting with darglint
run: poetry exec darglint

Expand Down
32 changes: 20 additions & 12 deletions .pre-commit-config.yaml
@@ -1,14 +1,27 @@
# Configuration of https://pre-commit.com/

repos:
- repo: https://github.com/psf/black-pre-commit-mirror # faster than https://github.com/psf/black
rev: 23.9.1

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.6
hooks:
- id: black
# It is recommended to specify the latest version of Python supported by your project here,
# or alternatively use pre-commit's default_language_version,
# see https://pre-commit.com/#top_level-default_language_version
language_version: python3.11
- id: ruff
args: [
--ignore=A002,
--ignore=A003,
--ignore=B023,
--ignore=D101,
--ignore=D102,
--ignore=D103,
--ignore=PLR0911,
--ignore=PLR0912,
--ignore=PLR0913,
--ignore=PLR0915,
--ignore=PLR2004,
--ignore=PLR5501,
--ignore=PLW0603,
]
- id: ruff-format

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
Expand Down Expand Up @@ -42,8 +55,3 @@ repos:
hooks:
- id: markdownlint
args: [--config, .markdownlint.yml, --ignore, CHANGELOG.md]

- repo: https://github.com/PyCQA/isort
rev: 5.11.5
hooks:
- id: isort
32 changes: 14 additions & 18 deletions README.md
@@ -1,12 +1,11 @@
[![](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)
[![](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![](https://img.shields.io/badge/Python%20code%20style-black-000000.svg)](https://github.com/psf/black)
[![](https://img.shields.io/badge/Python%20linting-pylint-darkgreen)](https://github.com/pylint-dev/pylint)
[![](https://img.shields.io/badge/Python%20typing-mypy-darkgreen)](https://github.com/python/mypy)
[![](https://img.shields.io/badge/Markdown%20linting-markdownlint-darkgreen)](
https://github.com/igorshubovych/markdownlint-cli)
[![](https://img.shields.io/badge/Link%20validation-markdown%20link%20validator-darkgreen)](
https://www.npmjs.com/package/markdown-link-validator)
[![](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![](https://img.shields.io/pypi/l/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)
](https://github.com/astral-sh/ruff)
[![](https://img.shields.io/badge/mypy-blue)](https://github.com/python/mypy)
[![](https://img.shields.io/badge/markdownlint-darkgreen)](https://github.com/igorshubovych/markdownlint-cli)
[![](https://img.shields.io/badge/markdown%20link%20validator-darkgreen)
](https://www.npmjs.com/package/markdown-link-validator)


# DSP-TOOLS - DaSCH Service Platform Tools
Expand Down Expand Up @@ -166,17 +165,15 @@ please make sure you use the same code style rules as we do.
We use the following linters:

- [MarkdownLint](https://github.com/igorshubovych/markdownlint-cli) (configured in `.markdownlint.yml`)
- [Black](https://pypi.org/project/black/) (configured in `pyproject.toml`)
- [mypy](https://pypi.org/project/mypy/) (configured in `pyproject.toml`)
- [pylint](https://pypi.org/project/pylint/) (configured in `pyproject.toml`)
- [isort](https://pypi.org/project/isort/) (configured in `pyproject.toml`)
- [ruff](https://pypi.org/project/ruff/) (configured in `pyproject.toml`)
- [darglint](https://pypi.org/project/darglint/) (configured in `.darglint`)

These linters are integrated in the GitHub CI pipeline,
so that every pull request is checked for code style violations.

In addition, there are [pre-commit hooks](#pre-commit-hooks)
that run Black and MarkdownLint locally before every commit.
that run Ruff and MarkdownLint locally before every commit.
This prevents you from committing code style violations.
Pre-commit is contained in the dependencies,
but before the first use, the hooks must be installed with `pre-commit install`
Expand All @@ -189,17 +186,16 @@ Depending on your IDE the installation is as follows.

- **MarkdownLint** can be installed as an extension (`davidanson.vscode-markdownlint`),
and be configured in the VSCode settings.
- **Black** can be set as formatter in the `ms-python.python` extension.
To do so, set `"python.formatting.provider": "black"` in the VSCode `settings.json`.
Alternatively, `ms-python.black-formatter` can be installed as an extension.
- **mypy** can be installed as an extension (`matangover.mypy`),
and be configured in the VSCode settings.
- This extension is different from the mypy functionality
of Microsoft's Python extension `ms-python.python`,
which only lints each file separately,
leading to incomplete type checking.
- **pylint** can be installed as an extension (`ms-python.pylint`),
and be configured in the VSCode settings.
- **ruff** can be installed as an extension (`charliermarsh.ruff`),
and be configured in the VSCode settings:
- `settings.json` > `[python]` > `"editor.defaultFormatter": "charliermarsh.ruff"`
- `settings.json` > `"ruff.format.args": ["--line-length=120"]`

If configured correctly, the style violations will be listed in the "Problems" tab.

Expand Down
10 changes: 3 additions & 7 deletions docs/developers/code-quality-tools/code-quality-tools.md
Expand Up @@ -12,17 +12,13 @@ The decision to use this set of tools is based on the information in the followi
| --------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------ |
| [General formatting](./general-formatting.md) | [EditorConfig](https://EditorConfig.org/) | `.editorconfig` |
| | [MarkdownLint](https://github.com/igorshubovych/markdownlint-cli/) | `.markdownlint.yml` |
| [Python formatting](./python-formatting.md) | [Black](https://pypi.org/project/black/) | `pyproject.toml` |
| [Python docstring formatting](./python-docstring-formatting.md) | [pydocstyle](https://pypi.org/project/pydocstyle/) * | |
| | [darglint](https://pypi.org/project/darglint/) | `.darglint` |
| [Python formatting](./python-formatting.md) | [Ruff](https://pypi.org/project/ruff/) | `pyproject.toml` |
| [Python docstring formatting](./python-docstring-formatting.md) | [darglint](https://pypi.org/project/darglint/) | `.darglint` |
| [Python type checking](./python-type-checking.md) | [mypy](https://pypi.org/project/mypy/) | `pyproject.toml` |
| [Python linting](./python-linting.md) | [Ruff](https://pypi.org/project/ruff/) * | |
| | [pylint](https://pypi.org/project/pylint/) ** | `pyproject.toml` |
| [Python linting](./python-linting.md) | [Ruff](https://pypi.org/project/ruff/) | `pyproject.toml` |
| [Security checks](./security.md) | [Dependabot](https://docs.github.com/en/code-security/dependabot/) | `.github/dependabot.yml` |
| | [CodeQL](https://codeql.github.com/) | GitHub settings |
| | [Gitleaks](https://gitleaks.io/) * | `.gitleaks.toml` |
| | [Bandit](https://pypi.org/project/bandit/) | `.github/workflows/bandit.yml` |

(*) coming soon

(**) will perhaps become redundant, as Ruff matures
5 changes: 5 additions & 0 deletions docs/developers/code-quality-tools/python-formatting.md
Expand Up @@ -24,6 +24,11 @@ Black aims for readability and reducing git diffs.
Black is an easy-to-use tool, with sensible and useful defaults.
Its style is very elegant.

### [Ruff](https://pypi.org/project/ruff/)

Besides being a linter, Ruff is also an auto-formatter.
It is designed as a drop-in replacement for Black.

### [yapf](https://pypi.org/project/yapf/)

Auto-formatter that can be configured to support different styles.
Expand Down
5 changes: 4 additions & 1 deletion docs/developers/code-quality-tools/python-linting.md
Expand Up @@ -34,7 +34,8 @@ Unlike Pyflakes, Flake8 can be configured.
The fastest, most promising and most exciting linter, although still in beta.
Ruff is a linter with autofix support.
It has near-parity with Flake8, and they are working on implementing pylint rules as well
(see [How does Ruff compare to Pylint?](https://beta.ruff.rs/docs/faq/#how-does-ruff-compare-to-pylint)).
(see [How Does Ruff's Linter Compare to Pylint?](
https://docs.astral.sh/ruff/faq/#how-does-ruffs-linter-compare-to-pylint)).
Ruff can be used to replace

- Flake8 (wrapper around Pyflakes, pycodestyle, and McCabe) plus dozens of plugins
Expand All @@ -48,6 +49,8 @@ Ruff can be used to replace
- pyupgrade (automated upgrade to newer python syntax)
- autoflake (automated removal of unused imports or variables)

Additionally, Ruff's formatter can be used to replace Black.

### [Vulture](https://pypi.org/project/vulture/)

Static code analyzer that finds unused code in Python programs.
Expand Down
17 changes: 8 additions & 9 deletions docs/index.md
@@ -1,12 +1,11 @@
[![](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)
[![](https://img.shields.io/pypi/pyversions/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![](https://img.shields.io/badge/Python%20code%20style-black-000000.svg)](https://github.com/psf/black)
[![](https://img.shields.io/badge/Python%20linting-pylint-darkgreen)](https://github.com/pylint-dev/pylint)
[![](https://img.shields.io/badge/Python%20typing-mypy-darkgreen)](https://github.com/python/mypy)
[![](https://img.shields.io/badge/Markdown%20linting-markdownlint-darkgreen)](
https://github.com/igorshubovych/markdownlint-cli)
[![](https://img.shields.io/badge/Link%20validation-markdown%20link%20validator-darkgreen)](
https://www.npmjs.com/package/markdown-link-validator)
[![](https://img.shields.io/pypi/v/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![](https://img.shields.io/pypi/l/dsp-tools.svg)](https://pypi.org/project/dsp-tools/)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)
](https://github.com/astral-sh/ruff)
[![](https://img.shields.io/badge/mypy-blue)](https://github.com/python/mypy)
[![](https://img.shields.io/badge/markdownlint-darkgreen)](https://github.com/igorshubovych/markdownlint-cli)
[![](https://img.shields.io/badge/markdown%20link%20validator-darkgreen)
](https://www.npmjs.com/package/markdown-link-validator)

# DSP-TOOLS Documentation

Expand Down