Python: Replace `isort`, `pylint`, `prettier`, `pyupgrade` & `flake8` with `ruff` #8408

MehulBatra · 2023-08-27T15:31:50Z

We will be using Rust for faster CICD
It's written in rust for faster linting , sorting & typechecking it will be used as a part of the pre-commit hook & git runner replacing Isort, pylint, prettier, pyupgrade and flake8 and will help in the faster development cycle by pointing errors, bugs, stylistic errors, and suspicious constructs faster than ever.

Resolves #8294

…ing of python projects

MehulBatra · 2023-08-27T15:38:28Z

Linked to issue: #8294

python/pyproject.toml

Co-authored-by: Fokko Driesprong <fokko@apache.org>

MehulBatra · 2023-08-27T19:03:26Z

I saw flake8 also being used we can remove that as well, as Ruff can take care of flake rules by default

… were exlcuded in flake in ruff aswell

Fokko · 2023-08-29T08:06:33Z

@MehulBatra I think we can also remove pyupgrade:

Fokko

Some small comments, can you take a look at those @MehulBatra? Apart from that, it looks great! There are probably more rules that we want to enable, but we can do that in a follow up PR. I like ruff a lot.

python/pyproject.toml

python/.pre-commit-config.yaml

python/pyproject.toml

Co-authored-by: Fokko Driesprong <fokko@apache.org>

Fokko

Something looks really off 🤔

python/pyiceberg/avro/file.py

python/pyiceberg/avro/reader.py

Fokko

Some small nits, but apart from that it looks great! Almost there

python/.pre-commit-config.yaml

python/pyiceberg/transforms.py

python/tests/avro/test_file.py

python/tests/catalog/test_glue.py

python/pyproject.toml

Fokko · 2023-09-01T14:09:44Z

python/build-module.py

@@ -17,6 +17,7 @@

 import os
 import shutil
+


This line surprises me since they are both internal packages.

It separated import and from statements mostly due to property
lines-between-types = 1

https://beta.ruff.rs/docs/settings/#isort-lines-between-types
I can try setting it up 0

Co-authored-by: Fokko Driesprong <fokko@apache.org>

python/pyproject.toml

python/.pre-commit-config.yaml

python/pyiceberg/avro/file.py

python/.pre-commit-config.yaml

python/tests/catalog/test_glue.py

MehulBatra · 2023-09-01T15:19:11Z

python/build-module.py

@@ -17,6 +17,7 @@

 import os
 import shutil
+


It separated import and from statements mostly due to property
lines-between-types = 1

…t from

MehulBatra · 2023-09-22T15:28:59Z

python/tests/avro/test_decoder.py

@@ -24,8 +24,7 @@
 from unittest.mock import MagicMock, patch

 import pytest
-
-from pyiceberg.avro.decoder import BinaryDecoder, StreamingBinaryDecoder, new_decoder
+from pyiceberg.avro.decoder import BinaryDecoder, StreamingBinaryDecoder, new_decoder  # type: ignore


It was throwing Module "pyiceberg.avro.decoder" has no attribute "new_decoder" while running mypy check in commit hook hence added

# type: ignore comment to tell mypy to ignore this error

MehulBatra · 2023-09-22T15:44:55Z

python/pyiceberg/io/pyarrow.py:1164:16: E721 Do not compare types, use isinstance()
if not isinstance(self.current_max, str):
python/pyiceberg/io/pyarrow.py:1169:16: E721 Do not compare types, use isinstance()
changed it to
if not isinstance(self.current_max, bytes):

MehulBatra · 2023-09-22T15:46:51Z

for ruff isort properties added properties in toml:
known-first-party = ["pyiceberg", "tests"]
section-order = ["future", "standard-library", "third-party", "first-party", "local-folder"]

Fokko · 2023-09-22T15:52:27Z

python/pyiceberg/io/pyarrow.py:1164:16: E721 Do not compare types, use isinstance()

This is a great catch already, thanks for fixing this!

MehulBatra · 2023-09-22T16:18:59Z

fixed the mypy_path config
Mypy will now use the relative path to determine the fully qualified module name

Fokko · 2023-09-22T20:18:15Z

Thanks again @MehulBatra for working on this, great to have this in!

alexander-beedie · 2023-09-26T09:40:54Z

Nice one :)

[FEAT] replacing isort,pylint with ruff(made in rust) for faster lint…

cf4ee89

…ing of python projects

github-actions bot added the python label Aug 27, 2023

Fokko reviewed Aug 27, 2023

View reviewed changes

python/pyproject.toml Outdated Show resolved Hide resolved

Fokko reviewed Aug 27, 2023

View reviewed changes

python/pyproject.toml Show resolved Hide resolved

Update python/pyproject.toml

0d55687

Co-authored-by: Fokko Driesprong <fokko@apache.org>

MehulBatra requested a review from Fokko August 27, 2023 18:42

remove flake8 and ruff already covering it and ignore the rules which…

a2d2f7f

… were exlcuded in flake in ruff aswell

MehulBatra changed the title ~~[FEAT] replacing isort, pylint with ruff~~ [FEAT] replacing isort, pylint & flake8 with ruff Aug 27, 2023

MehulBatra added 4 commits August 28, 2023 12:37

upgrading to newer version of ruff

5a81b9f

remove prettier as this can be covered by ruff

1267280

linting check

4e2c746

linting check

f5189e5

MehulBatra changed the title ~~[FEAT] replacing isort, pylint & flake8 with ruff~~ [FEAT] replacing isort, pylint, prettier & flake8 with ruff Aug 28, 2023

Fokko changed the title ~~[FEAT] replacing isort, pylint, prettier & flake8 with ruff~~ Python: Replace isort, pylint, prettier & flake8 with ruff Aug 29, 2023

Fokko changed the title ~~Python: Replace isort, pylint, prettier & flake8 with ruff~~ Python: Replace isort, pylint, prettier & flake8 with ruff Aug 29, 2023

Fokko reviewed Aug 29, 2023

View reviewed changes

python/pyproject.toml Outdated Show resolved Hide resolved

python/.pre-commit-config.yaml Outdated Show resolved Hide resolved

python/pyproject.toml Show resolved Hide resolved

MehulBatra and others added 2 commits August 29, 2023 13:52

Update python/pyproject.toml

6c13849

Co-authored-by: Fokko Driesprong <fokko@apache.org>

replace pyupgrade with ruff extension of pyupgrade

f1baa0a

MehulBatra changed the title ~~Python: Replace isort, pylint, prettier & flake8 with ruff~~ Python: Replace isort, pylint, prettier, pyupgrade & flake8 with ruff Aug 29, 2023

MehulBatra added 2 commits August 29, 2023 14:08

reformat as per ruff

3a208c9

rearrange ruff order

1022969

Fokko reviewed Aug 29, 2023

View reviewed changes

python/pyiceberg/avro/file.py Outdated Show resolved Hide resolved

python/pyiceberg/avro/reader.py Outdated Show resolved Hide resolved

ruff isort change detect same package

3afd3ed

MehulBatra force-pushed the python_ruff_for_faster_linting branch from 666f891 to 3afd3ed Compare August 30, 2023 17:20

exclude dev/provision.py from linting

1a18771

Fokko reviewed Aug 30, 2023

View reviewed changes

python/.pre-commit-config.yaml Show resolved Hide resolved

python/pyiceberg/transforms.py Show resolved Hide resolved

python/tests/avro/test_file.py Outdated Show resolved Hide resolved

python/tests/catalog/test_glue.py Outdated Show resolved Hide resolved

one line difference between first and third party

6de6668

MehulBatra and others added 5 commits August 31, 2023 12:56

lint build-module.py

f915b00

ruff isort known first party

1e1477d

ruff isort known first party

cc44f7d

ruff isort known first party

98857de

Merge branch 'master' into python_ruff_for_faster_linting

2436d09

Fokko reviewed Sep 1, 2023

View reviewed changes

MehulBatra and others added 2 commits September 1, 2023 20:50

Update python/pyproject.toml

7e9b5ca

Co-authored-by: Fokko Driesprong <fokko@apache.org>

rolling back pyarrow change

b5e315b

MehulBatra commented Sep 6, 2023

View reviewed changes

MehulBatra and others added 3 commits September 11, 2023 23:20

lines-between-types is set to 0 to avoid gap between direct and impor…

86f7167

…t from

resolve conflict

8714505

Merge branch 'master' into python_ruff_for_faster_linting

a6e3aa6

MehulBatra commented Sep 22, 2023

View reviewed changes

add pyiceberg as known first party for sort order

b487ca1

Fokko approved these changes Sep 22, 2023

View reviewed changes

add pyiceberg as known first party for sort order

f309938

Fokko merged commit 4f22dd8 into apache:master Sep 22, 2023
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python: Replace `isort`, `pylint`, `prettier`, `pyupgrade` & `flake8` with `ruff` #8408

Python: Replace `isort`, `pylint`, `prettier`, `pyupgrade` & `flake8` with `ruff` #8408

MehulBatra commented Aug 27, 2023 •

edited

MehulBatra commented Aug 27, 2023

MehulBatra commented Aug 27, 2023 •

edited

Fokko commented Aug 29, 2023

Fokko left a comment

Fokko left a comment

Fokko left a comment

Fokko Sep 1, 2023

MehulBatra Sep 1, 2023

MehulBatra Sep 11, 2023

MehulBatra Sep 1, 2023

MehulBatra Sep 22, 2023 •

edited

MehulBatra commented Sep 22, 2023

MehulBatra commented Sep 22, 2023 •

edited

Fokko commented Sep 22, 2023

MehulBatra commented Sep 22, 2023

Fokko commented Sep 22, 2023

alexander-beedie commented Sep 26, 2023

Python: Replace isort, pylint, prettier, pyupgrade & flake8 with ruff #8408

Python: Replace isort, pylint, prettier, pyupgrade & flake8 with ruff #8408

Conversation

MehulBatra commented Aug 27, 2023 • edited

MehulBatra commented Aug 27, 2023

MehulBatra commented Aug 27, 2023 • edited

Fokko commented Aug 29, 2023

Fokko left a comment

Choose a reason for hiding this comment

Fokko left a comment

Choose a reason for hiding this comment

Fokko left a comment

Choose a reason for hiding this comment

Fokko Sep 1, 2023

Choose a reason for hiding this comment

MehulBatra Sep 1, 2023

Choose a reason for hiding this comment

MehulBatra Sep 11, 2023

Choose a reason for hiding this comment

MehulBatra Sep 1, 2023

Choose a reason for hiding this comment

MehulBatra Sep 22, 2023 • edited

Choose a reason for hiding this comment

MehulBatra commented Sep 22, 2023

MehulBatra commented Sep 22, 2023 • edited

Fokko commented Sep 22, 2023

MehulBatra commented Sep 22, 2023

Fokko commented Sep 22, 2023

alexander-beedie commented Sep 26, 2023

Python: Replace `isort`, `pylint`, `prettier`, `pyupgrade` & `flake8` with `ruff` #8408

Python: Replace `isort`, `pylint`, `prettier`, `pyupgrade` & `flake8` with `ruff` #8408

MehulBatra commented Aug 27, 2023 •

edited

MehulBatra commented Aug 27, 2023 •

edited

MehulBatra Sep 22, 2023 •

edited

MehulBatra commented Sep 22, 2023 •

edited