Skip to content

Fix artifact for MyPy checks#23094

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-artifact-mypy-errors
Apr 19, 2022
Merged

Fix artifact for MyPy checks#23094
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-artifact-mypy-errors

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented Apr 19, 2022

Occasionally MyPy detects errors which have not been detected
before. This is likely caused by having too many files passed
to MyPY. If the number of files to pass to MyPy is too big,
pre-commit will automatically split the list of files into
several "mypy" commands. If we are unlucky the list of files
will cause MyPy to detect slightly different errors.

We split the mypy checks to be run separately for airflow core
and airflow providers to limit the list of files to be shorter.

We are also preparing for splitting off providers so this is
good idea in general.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@github-actions github-actions Bot added the full tests needed We need to run full set of tests for this PR to merge label Apr 19, 2022
@github-actions
Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Occasionally MyPy detects errors which have not been detected
before. This is likely caused by having too many files passed
to MyPY. If the number of files to pass to MyPy is too big,
pre-commit will automatically split the list of files into
several "mypy" commands. If we are unlucky the list of files
will cause MyPy to detect slightly different errors.

We split the mypy checks to be run separately for airflow core
and airflow providers to limit the list of files to be shorter.

We are also preparing for splitting off providers so this is
good idea in general.
@potiuk potiuk force-pushed the fix-artifact-mypy-errors branch from c3c8a95 to 922d1f7 Compare April 19, 2022 18:31
@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Apr 19, 2022

Hey @ashb (one more comment) - I think I figured out why the MyPy "instability" happens and updated the PR to fix it permanently.

I think it has to do with the number of files and the fact that it's semi-random how pre-commit splits files if it decides to split long list of files in two commands.

This PR now makes the split "consistent" - it splits the files to check into "core" and "providers" buckets. Which sooner or later we will have to do anyway when/if we split repos.

@ashb
Copy link
Copy Markdown
Member

ashb commented Apr 19, 2022

Ohh right yeah that would explain it!

@potiuk
Copy link
Copy Markdown
Member Author

potiuk commented Apr 19, 2022

Passed static checks. Merging.

@potiuk potiuk merged commit c897ccd into apache:main Apr 19, 2022
@potiuk potiuk deleted the fix-artifact-mypy-errors branch April 19, 2022 19:12
@jedcunningham jedcunningham added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Apr 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) full tests needed We need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants