Version mismatch for azure-mgmt-cdn and azure-mgmt-compute

[glaubitz]

Hi!

PyPi lists 1.0.0 as the latest stable version of azure-mgmt-cdn while azure-mgmt still depends on ~= 0.30.0 meaning that the 1.0.0 is considered incompatible. Similarly, there is azure-mgmt-compute in version 2.0.0 while azure-mgmt still depends on ~= 1.0.0.

Are the newer versions of -cdn and -compute not supposed to be used yet despite being tagged as stable on PyPi? I ask because we usually pick the latest stable versions from PyPi for packaging in SUSE.

Adrian

[lmazuel]

Hi @glaubitz

azure-mgmt and azure are bundle meta-packages, that we want to publish every 6 months ish. Since they have dozen of packages, I can't rev a major version each time one of them is updated to a major version (or I'll reach v42.0.0 pretty soon :)).
It's why we document single installation on the documentation:
https://docs.microsoft.com/python/api/overview/azure/virtualmachines

But we still publish bundle meta-packages for ease of use.

I see myself these meta-packages a little like a Ubuntu/Debian cycle (sorry I don't know if SUSE follow the same cycle :)).

Note that we didn't introduce any packages incompatibilities since June 2016 ish (and msrest 0.4.x series)

[glaubitz]

Hi @lmazuel

The problem with this approach is that it makes downstream packaging harder as I can no longer just pull the latest stable version of each azure component from PyPI and package it for a distribution. I have to make sure I only pull the versions that are listed in the setup.py of the meta packages azure and azure-mgmt as otherwise these meta packages will have dependencies that cannot be fulfilled due to the use of the ~= operator which introduces conflicts with higher major versions as the one specified.

Thus, I suggest either to not use the ~= operator when you don't need it - since you mention you didn't introduce any incompatibilities since June 2016 - and just replace it with >=. Or update the versions in the setup.py of azure and azure-mgmt to match the latest stable versions available on PyPi.

Another alternative would be to not mark any packages on PyPi as stable until the azure and ``azure-mgmthave updated the version numbers in theirsetup.py```.

The current situation is definitely not optimal.

[lmazuel]

Hi @glaubitz

I see. First, just to be sure I'm clear, when I say "n't introduce any packages incompatibilities since June 2016", I don't mean there is no breaking changes from one version of the same package to the next major version, I say that any packages of two services starting June 2016 can be installed side by side without issue (example, azure-mgmt-network 0.30.0rc5 and azure-mgmt-compute 2.0.0)

On my side, I want to be sure that if someone depends on azure~=2.0, he never can get a breaking change, this will never happen. But at the same time, I need to release each latest service team release once available (e.g. CLI being updated with new features).

So, at the Suse level, you are packaging azure 2.0.0 correct? So each time I release a package what should be installed using ~= (a minor release), you want to update the SUSE package with that version (just to be sure I understand your need, before talking solution to your need :))

[glaubitz]

Hi @lmazuel

I see. First, just to be sure I'm clear, when I say "n't introduce any packages incompatibilities since June 2016", I don't mean there is no breaking changes from one version of the same package to the next major version, I say that any packages of two services starting June 2016 can be installed side by side without issue (example, azure-mgmt-network 0.30.0rc5 and azure-mgmt-compute 2.0.0)

Yes, that's how I understood it. My current issue is with the azure-mgmt meta package.

On my side, I want to be sure that if someone depends on azure~=2.0, he never can get a breaking change, this will never happen. But at the same time, I need to release each latest service team release once available (e.g. CLI being updated with new features).

Ok, I understand the reasoning here.

So, at the Suse level, you are packaging azure 2.0.0 correct? So each time I release a package what should be installed using ~= (a minor release), you want to update the SUSE package with that version (just to be sure I understand your need, before talking solution to your need :))

What I need is a clear distinction of what packages can be shipped together and which not. It is really very confusing for a downstream project if you have the latest version of a meta package which does not pull in the latest version of all stable packages.

As far as I know, PyPi can mark packages as "pre-release". So, what I would suggest is to not mark packages as stable in PyPi until you have updated enough packages to release a new version of the meta packages.

We have over 40 packages from the Azure SDK in SUSE (see: https://build.opensuse.org/project/show/Cloud:Tools:azuresdkbreak), so coordinating all of them can be quite tricky. I unfortunately made the mistake that I picked up the new versions of azure-mgmt-cdn and azure-mgmt-compute for submission to the repositories not noticing that the azure-mgmt depends on azure-mgmt-cdn ~= 0.30.3 and azure-mgmt-compute ~= 1.0.0. Now I'm in the situation that I'm not sure whether I need to downgrade these packages or not.

Is there already an estimated release date for the new SDK which depends on the newer versions of the individual packages?

[lmazuel]

What I need is a clear distinction of what packages can be shipped together and which not.

Any packages that depends on msrestazure 0.4.x can be shipped together without any problem (roughly everything released after June 2016)

As far as I know, PyPi can mark packages as "pre-release". So, what I would suggest is to not mark packages as stable in PyPi until you have updated enough packages to release a new version of the meta packages.

Problem is, when I release azure-mgmt-compute, it is stable :/. I have users that just care about this particular package and don't use azure or azure-mgmt, and using a preview package won't be fair for them.

I just got a long discussion with @brettcannon about this about the azure story. We think that by essence there is no good way to maintain a semantic versioning of a meta-package like azure. We should see this package as a quickstart alias to simplify the initial setup. Trying to pin versions inside just does not work, and creates regular problems.

Our current suggestion is then to stop pinning version in azure and assume his alias role (this is mostly what people asks from us actually). This means:

• We release azure 3.0 with no pinned dependencies. Azure 3.0 is a an alias list and nothing more.
• We release azure 3.(x+1) versions only on new packages coming.
• We don't planned 3.x.y release (bug fix release)
• We guarantee strict semantic versioning at the service level
• Then, the correct way to pin your app, is to pip freeze the service dependency, not azure. Note that already today, pip freeze would have give you specific version of packages anyway.
• Packages that are not considered stable (meaning: we expect regular breaking changes) are not in azure and must be installed by name.
• Packages in azure are considered stable enough, so we don't plan a lot of breaking changes releases.
• We don't touch to azure 2.0, since we took the commitment that it will never breaks (hence 3.0)

Thoughts: @irl @rjschwei @glaubitz

[lmazuel]

FYI rejected ideas:

• Rev major version of azure at each service release. Will grow too fast, and it's unfair if a small service change a comma and big service like compute does not change
• Rev every six months. Cause of this issue, azure is behind.
• Rev "ubuntu like" with version "17.04". If it's every month, does not tell you what is breaking and it's time consuming. If it's every 6 months, go back to square 2
• Release 3.0.0rc1 at the same time of 2.0.0, keep 3.0.0rc1 not pinned. In 6 months, release 3.0.0 pinned, and release 4.0.0rc1 unpinned etc. This force people to use --pre and it's false: we are NOT installing preview packages.

[rjschwei]

This works for us, strict semantic versioning at the component level.

Thanks,
Robert