Describe provider versioning changes with semver

[potiuk]

We agreed that provider package versioning needs to be done with semver. We need to work out the way on how this automated versioning would work from those perspectives:

• contributor
• maintainer
• release manager

Short documentation about this would be great, before we implement that, so that we know what could be automated and what should be manual work.

We need to agree on:

• how to increment the values when we release each package
• based on what criteria
• how the contributors should mark the PRs as bugfixes/features (needed for automation)
• how maintainers should verify it
• whether release manager needs to do some verification
• decide how we release sources - all together or as separate set of sources per package : Separate.
• manage cross provider dependencies
• tagging mechanism: separate tag per providers
• how we approach relating patchlevels for minor versions and minor versions for major ones (say we have 2.0.0 and 1.1.0 of providers out there - how do we approach relating 1.1.1 with bugfixes).

[potiuk]

@ashb - since you seem to be the biggest proponent of the Semver proposal, I think it would be great that you describe the process or at least propose it. There is no big hurry with that, as this will only be needed when we eventually release the final 2.0 packages together with 2.0 release, but it would be great to at least think and propose how the provess will look like from those three perspectives and where we can automate what.

It would be great to discuss it upfront before someone attempts to implement it. Would you be ok with leading that part ?

[ashb]

@ashb - since you seem to be the biggest proponent of the Semver proposal, I think it would be great that you describe the process or at least propose it. There is no big hurry with that, as this will only be needed when we eventually release the final 2.0 packages together with 2.0 release, but it would be great to at least think and propose how the provess will look like from those three perspectives and where we can automate what.

It would be great to discuss it upfront before someone attempts to implement it. Would you be ok with leading that part ?

Sure! I'll start on this on Monday or Tuesday

[potiuk]

Once agreed the process here we will implement it in #11427

[ashb]

• decide how we release sources - all together or as separate set of sources per package

Per provider, and containing just the sources for that provider (i.e. airflow/providers/X/* -- essentially what we would get if doing setup.py sdist) -- the whole point of separating them out is to be able to have different release cycles for different providers.

[potiuk]

• decide how we release sources - all together or as separate set of sources per package

Per provider, and containing just the sources for that provider (i.e. airflow/providers/X/* -- essentially what we would get if doing setup.py sdist) -- the whole point of separating them out is to be able to have different release cycles for different providers.

Yep. Agree this is probably the best way. It will be also easier to version the source packages then. We'd have to remove providers from Airflow sdist completely, but that's easy.

[potiuk]

Hello everyone here. I wanted to propose some vision of the 'separate provider' versioning that we should implement as part of cleanup and release Google Provider.

We are already planning to release 2.0.0 provider for Google as it will contain breaking changes due to non-compatible Python API 2.0 libraries. So we will have a chance to test the extreme version,

My proposal:

1. We should add 'current version' of the library to provider.yaml. This might be the top "version" name in the list of versions simply. This way we avoid duplication.

2. For every released version of provider we add a tag in the repo (currently this means 60 tags): Example: 'providers-google-1.0.0'.

3. We slightly change the way we generate release notes. We will simply get the list of commits from the providers-google-<PREVIOUS_VERSION> tag.

4. We remove all the generated CHANGES_* and README_* files completely and instead of keeping them in the repo, we will generate them automatically when the documentation is generated and place it as part of the 'per-provider' documentation.

5. Similarly as we currently do with Airflow's UPDATING.md, we keep the information about breaking changes in the ADDIITONAL_INFO.md (which is already there for some packages). This additional info is automatically part of README.md. We should make sure that any breaking changes or new features added are reflected in the ADDITIONAL_INFO.md during PR Review. This should be author's responsibility to describe it and reviewer's responsibility to point out that it should be updated when we see new feature/backwards incompatible changes. We should add them in "master" version and just before release we decide what is the new version and we update it in a separate commit. We can even rename ADDITIONAL_INFO to UPDATING to keep consistency or rename it to something different. I am open for proposals. This will also become a better "readable" description of what has changed between different versions.

6. We will add a tool/view/smth to show each provider together with ADDITIONAL_CHANGES and list of affecting commit in a single summary page, so that we can decide when to release which providers.

I think this is a minimal but pretty comprehensive set of things that we have to implement in order start releasing providers independently and more frequently and if we agree that this looks good, I can split it into issues and start implementing it.

Let me know what you think @kaxil @ashb @mik-laj @turbaszek @XD-DENG others interested :).

[ashb]

Sounds good. Two minor comments

1. We can use the existing version field in the provide.yaml for this can't we?
2. I think for the 1.0.0 I did providers/ (rather than providers-) prefix. We should be consistent, and either continue to use / or go back and recreate the 1.0.0. Same for upgrade-check tags.

Otherwise in total agreement.

[potiuk]

• We can use the existing version field in the provide.yaml for this can't we?

Yep. We currently have the 'versions' field which is an array of versions and we use it to keep list of available versions of the provider (we can use it to select version in the generated docs like in Airflow docs). Sorted in descending order, so we can add the right version as the first entry at the moment we prepare for release.

    "versions": {
      "description": "List of available versions in Pypi. Sorted descending according to release date.",
      "type": "array",
      "items": {
        "type": "string"
      }
    },

• I think for the 1.0.0 I did providers/ (rather than providers-) prefix. We should be consistent, and either continue to use / or go back and recreate the 1.0.0. Same for upgrade-check tags.

We have to recreate it anyway because we want to have a separate tag for each provider (yeah a lot of tags). But I am also fine with keeping it as:

providers-google/1.0.0
providers-apache-hive/1.0.0

I think this is a nice convention to separate version with '/'

[potiuk]

And one more thing. I believe for the new versions of providers we can completely skip the list of added/moved operators in README. Those are useful for Backport providers and possibly for 1.0.0 version, but I think for all the future releases of providers the "ADDITIONAL_INFO.md" content (or renamed to UPDATING) is much more important, and the documentation we have covers the list of classes etc. much better. So we should have link to the documentation here and list of changes from ADDITIONAL_INFO.

I think list of commits in each version is useful though. The fact that you can get from the README directly to the commit and see the description of it is really nice (but there is no need to show all the commits which were there for the 1.0.0 version).

[potiuk]

This has been spli into separate issues #13271 #13272 #13273 #13274 #13275 #13276 #13277 so I am closing this issue.