Improved Python Wrapping

[johnhaddon]

We have a few issues with the way we have wrapped classes in Python to date, using the IECorePython::Wrapper class :

• We pay the overhead of wrapping a class even when we don't end up deriving from it in Python. Many classes (nearly all the Parameters) are wrapped so they can be derived from in Python, but this is a very rare use case - most Parameters are just used directly and most Parameter instances are not of Python derived types.
• When a class is wrapped, all its wrapped methods must acquire the GIL and enter Python to see if there is an override there. In many cases there is not. Entering Python is slow, and acquiring the GIL destroys parallelism.
• When a class is wrapped, the C++ class must hold a reference to the Python half of its implementation. This creates a circular reference which must be broken by garbage collection. This has overheads in both memory and runtime.
• Wrapping classes must use ugly macros like IECOREPYTHON_RUNTIMETYPEDWRAPPERFNS to repeat the same old overrides time and time again.

This pull request implements a new style of wrapping which solves all these problems. In its constructor, it determines whether the instance is of a Python subclass or not, and provides this information via an isSubclassed() method. This can be called without acquiring the GIL, and is used to avoid looking for method overrides when that is not necessary. When an instance isn't subclassed we also avoid making a circular reference, and avoid all garbage collection overhead. The new wrappers also automatically provide the appropriate virtual overrides, with no need for macros.

There are two potential downsides to this change :

• We can no longer inject overrides for methods into non-subclassed Python objects. To override a method you must derive from the base class as you would in C++. This isn't terribly Pythonic, but it is necessary to enabled this optimisation, and this optimisation is very necessary.
• We can no longer guarantee a one-to-one relationship between a C++ object and a Python object for non-subclassed objects. We've always had a problem whereby two non-wrapped Python objects could refer to the same C++ object, and we use the "a.isSame( b )" workaround where normally one would use "a is b". Because wrapped classes hold onto their Python half, we can guarantee a one-to-one mapping, but now we only guarantee that when a wrapped object is subclassed in Python. Otherwise we would have to pay the garbage collection penalty. I have some ideas as to how this could be solved, and solved for non-wrapped objects too, but that will be something for the future.

Right now, Cortex has both styles of wrapping available, and all existing classes are wrapped using the old style - nothing has changed for Cortex users. I'm about to put in a pull request for Gaffer which uses the new style, as it's critical for performance there. In that pull request you'll see some of the gotchas with the new approach, and how to deal with them. Once we've transitioned over to Gaffer using the new style, and got any problems ironed out we can move Cortex to the new style too, and remove all the old deprecated code - but I suggest we don't do that till the next major version.

[ldmoser]

Great work John! I'm looking forward to apply the new wrappers in Cortex 9!
Andrew, do you want to look at it as well?

[johnhaddon]

Thanks for spotting that error Lucio - I've fixed it along with a couple of other small tidy ups.