Enable support for both Python 2.x and 3.x

[brianredbeard]

As subsequent changes are no longer forthcoming to the 2.x branch of Python projects should begin the process of cross compatibility. This issue is being opened to collect outstanding issues and note blockers, should other users attempt to run Iris with Python 3.x.

[jrgp]

It will be interesting to see which iris dependencies don't work on 3.x.

[brianredbeard]

As of right now it looks like the basic components (between iris and oncall) will be:

• moving from python-ldap to pyldap or ldap3. pyldap (in my brief testing) has worked as a drop-in replacement for python-ldap.
• Fixing some print statements & importing from __future__ import print_function
• Being more intentional about when/where/why unicode is used. Currently there is a mix of unicode and non-unicode strings. One answer would be to use future to treat all strings as unicode literals with from __future__ import unicode_literals.
• Avoid using byte-strings as literals (currently done in a few spots, most notably in using falcon )
• Review the use of requests instead of urllib. urllib becomes a consolidation of functionality from urllib, urllib2, urlparse, robotparser, and a number of other modules re-namespaced under urllib. If moving to a different tool like requests is to be avoided, there are a number of ways that this can be supported by wrapping the imports in a try except block or using aliases - http://python-future.org/compatible_idioms.html#urllib-module
• Utilize python3 style iterators for custom classes (e.g. use assert next(slaves) versus assert slaves.next() )

Additionally, there are a few situations where one of the following could be likely

• Making a choice between accepting some inefficiency in Python 2.x vs utilizing the module six (which is already required for some of the other dependencies) or future. As future is the most simple way of handling some of the print functions, this is likely. This would affect how dictionary iteration,

It looks like a lot of this can be successfully be done using some of the various Python 2->3 conversion tools, but would be useful to make the changes scoped to one type at a time with a burn-in period.

[jrgp]

Some of those we can do, but we're still discussing internally the pros/cons of moving from 2.7 to 3.x.

To avoid relying on compatibility layers (six/future/etc) we feel it'd be best to only support one major version of python instead of trying to support both.

[houqp]

If we want to stick with Python, then it's better to go all in to py3. I can see us benefit a lot from the static type annotation.

@brianredbeard thoughts on porting Iris/oncall to Go? Internally we have talked about doing a go rewrite multiple times. But I am not sure how customers will feel about the rewrite since Iris is in a very stable state right now.

[brianredbeard]

@jrgp I totally understand. I've on all of my various utilities I've been undergoing the process of moving 2->3 for all of my scripts.

@houqp I would be all for a Go version. There are positives and negatives to it to be noted.

Positives:

• Easier deployment - The fact that binaries are statically compiled and (in most cases) cross platform can widen the audience.
• Smaller deployments - My Iris / Oncall image (still in flight) is coming in at approximately 100MB with all dependencies. (That being said, I've spent a long time optimizing this workflow). Applications built with Go will not have to worry about package managers, system libraries, etc.
• Removes some dependencies (related to the previous comment)
• Unified stack providing good mechanism for traffic serving and thread management
• Better separation of build/run stages

Negatives:

• Will possibly change your workflow
• Tests will need to be re-written & test frameworks are a bit
• Go dependency management is a PITA

[jrgp]

@brianredbeard regarding those Go negatives:

• Workflow (in terms of build -> deploy) might not change that much
• Part of why we focused so much on the e2etest.py is because those would be reusable in a go rewrite
• Go dep management is pretty fragmented. We developed an internal solution using gradle/ivy for our Go apps, but we aren't going to open source that tooling.

We've had lots of internal talks about doing a golang rewrite (iris was almost written in go), it would just take time and the existing py2.7 implementation works well enough.

@houqp : I'm not sure if it would be good to use py3's static type checking as that incurs performance overhead which we probably wouldn't want for parts like sender. (per @fellyns)

[webermanish]

Is Iris fully functional for python 3.x ?

[diegocepedaw]

@webermanish the py3 branch is fully python 3 and is in fact the branch we are currently developing on. We hadn't merged it with the main branch because it breaks python 2 compatibility but we will probably do so soon.

[webermanish]

@diegocepedaw does that mean py3 branch code is stable and can be used in production.

[diegocepedaw]

@webermanish, yes, that branch is what we currently use in production.