Python 3 compatibility

[wido]

A lot of Python code inside CloudStack is still Python 2 and we have just 10 months left before Python 2 is no longer supported: https://pythonclock.org/

Various items need to be looked at and checked for Python 3 compatibility. A list, but probably not complete:

• KVM Security Groups
• Xen Security Groups
• Virtual Router scripts (addressed in Update SystemVM scripts from python2 to python3 #3730)
• Various setup tools
• build tools (addressed in some minor changes to make the build run with python 3 #4030)
• marvin framework and smoke tests (Marvin3 #4727)
• marvin component test sets (component test ports/fixes in python3 #5082)
• packaging (cloud.spec) (Suggestions on Python processing of cloud.spec file in packaging directory #4513 is asking for it)

These are isolated projects and can usually be taken care of separately and also properly tested.

Creating this ticket so it can be referred to and also this is out there.

[rohityadavcloud]

Is there an argument to say, these may be implemented purely in Java (if possible)? Currently, many of these scripts esp on KVM are forked via the cloudstack-agent and called.
For VR, I played with GRPC and I think it can be secured against the current CA-framework introduced in 4.11; here's a PoC: shapeblue#44
I'll propose more details on dev@ once I can put together design/idea details.

[falcon78921]

@rhtyd @wido I would love to help with moving over to Python 3, if that's still the plan.

[wido]

Although I agree with @rhtyd that a lot can be done in Java, it's not always the easiest. Python is sometimes a lot easier to develop and talk directly with the system for executing commands.

We haven't started on this @falcon78921 , but any help is welcome! Just pick a Python script and start working on it.

[rohityadavcloud]

@wido changing the env dependency is a big step, python3.x is not available on all distros by default. I think it's better to discuss on dev@ for future steps. In my proposal, for the VR for example I've proposed that we move to a lightweight agent and as PoC I've shared a Go-Java approach based on grpc.

I think the best time would be to discuss/start doing things after 4.13, as we've decided to deprecate CentOS6 after 4.13.

[wido]

@rhtyd Python 3 is available on ALL distros. It has been for a very long time. It's on RHEL, CentOS, Debian and Ubuntu. No need to install anything :)

We can always talk about this, but the fact is that per January Python 2 is EOL and it's not so super difficult to write things in Python 3.

I'm not against an Agent in the VR for example, sounds like a good thing!

[PaulAngus]

There are also knock-on effects, ie I suspect that the chances of running Python3 on XenServer 6.x is near zero.

[wido]

@PaulAngus Python 2 is EOL in 2020, so any plans on what Xen is going to do?

But this doesn't stop us to fix other Python code before that time.

[rohityadavcloud]

I think we'll need to deprecate all the XS 6.x, which we can discuss and aim to do in 4.14 and onwards. XS7 or xcp-ng 7.x with CentOS7 based dom0 may be more easy to support python3.

[andrijapanicsb]

guys, any update on this one? I don't see it coming to 4.14 due to a needed effort?

[rohityadavcloud]

Given the time it won't be possible to completely do it and test within 4.14, let's do it against 4.15.

[wido]

@GabrielBrascher Has done work on this and he can comment on how far he got.

[GabrielBrascher]

@rhtyd @andrijapanicsb I am testing and re-working with PR #3730 right now. However, considering the tight schedule, it might be better to keep such changes to the next release indeed.

[rohityadavcloud]

@DaanHoogland can you review that's remaining and being targetted for 4.16; I suppose except for component tests and VR scripts all are 100% complete? Even component tests are all migrated to py3? cc @nvazquez @sureshanaparti

[DaanHoogland]

yes, component tests are migrated and #5082 is open for fixing (some of) them. The VR is not done and there are some obscure tests in test/integration/testpaths and some in test/integration/plugins
nothing else that I know of.

[nvazquez]

Thanks @DaanHoogland - PR 5082 is merged but there are still some component tests to migrate?
@GabrielBrascher are you still working on this PR for VR code? #4807

[GabrielBrascher]

@nvazquez I need to roll a couple more tests.
Let's have it in the next release, for 4.16 it might be risky to have it.
Thanks!

[DaanHoogland]

Thanks @DaanHoogland - PR 5082 is merged but there are still some component tests to migrate?
@GabrielBrascher are you still working on this PR for VR code? #4807

not to migrate but to fix. component tests have been neglected way too long.

[DaanHoogland]

VR python scripts are not addressed and may not before 4.17. for now I'm keeping this open, just for the sake of the VR scripts.

[DaanHoogland]

as there are no plans for doing this in 4.18 I'm moving this to 'unplanned'.

[rohityadavcloud]

I think at some point our systemvmtemplate based OS (Debian 11) will EOL, so our runway may be limited by that. For this reason we may need to migrate the VR scripts to py3 or rework the VR agent approach.

Thanks @wido all, I think most of the items are done except migration of the VR scripts for which I've created a new ticket - #7500