Release airflow upgrade_check as separate package

[turbaszek]

Description

Release airflow upgrade_check as an installable package in addition to command in 1.10.13/14 release.

Use case / motivation

In this way, users won't have to upgrade Airflow version only to use this tool. The upgrade check currently should be compatible with any 1.10.X version.

Users that will use this tool have already installed Airflow and doing pip install airflow_upgrade_check should be not a problem.

Related Issues

N/A

[mik-laj]

This may be a problem with database migrations. This tool would then have to contain copies of the newest migrations.

[vikramkoka]

Just to make sure that I understand the scope of this clearly, does this include the following elements:

1. Upgrade checks to be run from 1.10.x to say if this is compatible with 2.0,
2. Documentation to support above and including the overall guide for upgrading to 2.0, and
3. Upgrade utils to automate as much as possible of the migration steps.

Is this understanding correct?

[potiuk]

Comment from my side as promised:

I think we can use a similar process I implemented for docker image separation. I have a working solution here: https://github.com/potiuk/airflow-docker

The "main" branch there is just synchronizing the repo. The "master" and "v1-10-stable" are just mirrors of "master" and "v1-10-stable" from airflow (filtered to only contain The Dockerfile + relevant files). It extracts only the relevant paths/commits. So if we structure the "upgrade-check" properly, it can extract only the "upgrade-check" relevant stuff to this repo.

Benefits:

• We do not have to manually synchronize v1-10 airflow code to such a project (it happens automatically).
• We can continue running full tests and whole CI checks in airflow - where we have all the pre-commits, tests, etc.
• The synchronization is scheduled and runs in "pull" mode from airflow's public repo, so there is no need for additional authentication (and secrets).

Caveats:

• for now the "main" branch contains the .github/workflow schedule and the code that synchronizes the repos. It does not contain the actual Dockerfile + scripts. This is because the scheduled workflows must reside in the "main" branch and I did not want to mix the "sync" code with the actual Dockerfiles. However, it should be entirely possible to have a single "main branch" containing both the "sync" code and the "upgrade-check" code.

[potiuk]

We do not have to follow this route of course - this is just a proposal, that I am happy to discuss if you think it's worth pursuing

[kaxil]

Any updates on this one @turbaszek @dimberman ?

[kaxil]

Meeting - 19 Nov 2020

Attendees: Ash, Vikram, Jarek, Tomasz, Daniel, Kaxil

• We will continue using v1-10-stable for upgrade_checks and the PRs targetted towards it.

• We will support Airflow >=1.10.13 for upgrade_checks.

• CI needs to be updated to test/support multiple versions.

• Idea: Add a "version" field for checks to show supported Airflow version.

• Ash to work on packaging and automation this/next week.

• Add a check when using the plugin to see if a newer version of upgrade_check is available. Error / Warning ??? (This can be a Rule in upgrade-check itself)

• If we include "--apply" flag that automatically fixes the code, we need to flag and WARN that to users that there will be False positives as Bowler is known to fail in some-cases (TypeHints). Have users confirm before apply any change and show the code diff that is being changed -- same as what Bowler does.

• Where we don't automate, just verify we have it documented

Release Management:

• Have setup.py / setup.cfg inside airflow/upgrade_check folder (Ash to figure out details)

[ashb]

I'm working on the release process part for this now.

[ashb]

Release created, but still some tidy up of docs/process left to do so it is repeatable.

[turbaszek]

@ashb should we close the issue?