ILWIS Python API TechDoc

Introduction

This document is meant for developers, who want to extend the ILWIS Objects Python API. This Python API is based on SWIG and created in C++ using QtCreator, it is heavily recommended, that you have at least a general knowledge about these subjects. General documentation about ILWIS Objects can be found in ILWIS-Next-Generation-(Architecture) and ILWIS-Objects-overview.

In this document we will go over the set-up and idea behind the Python API's development, and we will also talk about how to set up the development environment. The last part of this document will be about how you can actually expand or edit the Python API.

Please note, that this document was created for Windows environment. Nevertheless it should as well be applicable for other development environments. Be aware of the difference in \ and / when dealing with paths to files and directories. A small error in those can create strange and hard to track down bugs.

Note that the whole set-up is tested only for 32-bit binaries. Make sure all mentioned packages are installed as 32-bit version!

Installation

The development environment for the ILWIS Objects Python API is based on the QtCreator IDE and a Python 3 installation combined with the toolbox that actually made the creation of this API possible: SWIG.

In the following paragraphs we will go over the set-up and configuration of the development environment. It is however assumed that you have a copy of the ILWIS source code and the project folder (with the libraries and resources etc.) in your working directory. If not the current version of the source code can be found in our repositories on GitHub (IlwisCore, IlwisConnectors), external libraries and resources are until now only available on our project folder on Google Drive^[TODO].

Qt

ILWIS Objects is created in QtCreator because of its multi-platform advantage over visual studio, this however means that in order to run and compile ILWIS Objects you need a Qt installation. And of course the ILWIS Objects project has to be configured correctly within Qt this however will be done in another chapter.

We have to install QtCreator. It is recommended to install all packages, this avoids the risk of missing a vital package later on.

Once installed we have to set up the ILWIS Objects projects within QtCreator:

1. Select projects from the <quick menu><build and run><build>

2. Open all current Project files (<File><OpenProject>), the most important are:

   1. <your Ilwis folder>\projects\IlwisCore\core.pro

   2. <your Ilwis folder>\projects\IlwisCore\Ilwisscript.pro

   3. <your Ilwis folder>\projects\IlwisCore\projectionimplproj4.pro

   4. <your Ilwis folder>\projects\IlwisCore\rasteroperations.pro

   5. <your Ilwis folder>\projects\IlwisCore\internalconnector.pro

   6. <your Ilwis folder>\projects\IlwisConnectors\gdalconnector.pro

   7. <your Ilwis folder>\projects\IlwisConnectors\Ilwis3connector.pro

   8. <your Ilwis folder>\projects\IlwisConnectors\baseoperations.pro

   9. <your Ilwis folder>\projects\IlwisConnectors\pythonapi.pro

3. Common practice is to change the build directory to the temp sub-folder of projects directory via <Projects (configureation)><General><Build directory:> like this: <your ILWIS working directory>\projects\temp\<Qt-project name>

Python

The Python API is compiled for python 3.0 and higher^[1], which means we recommend you install the most recent version. Again it is recommended to do a full installation.

SWIG

Description from the SWIG website:

"SWIG is an interface compiler that connects programs written in C and C++ with scripting languages such as Perl, Python, Ruby, and Tcl. It works by taking the declarations found in C/C++ header files and using them to generate the wrapper code that scripting languages need to access the underlying C/C++ code. In addition, SWIG provides a variety of customization features that let you tailor the wrapping process to suit your application."

SWIG does not require an actual installation, it only needs to be downloaded and unzipped in a folder of your choice. For later command line usage of SWIG it might be handy to add the path to the swig.exe to your system path environment. Whenever we refer to use the swig.exe, you can as well simply refer to that executable using a full path, so no need to change your system environment.

Configuration

Make sure all other ILWIS project are properly configured (including additional deployment make-steps like make install and already compiled.

1. Set the pythonapi.pro as Active Project (<in Projects-sidebar><context menu on pythonapi.pro><Set as Active Project>)

2. The project configuration of the pythonapi.pro assumes Python 3.3.X to be installed in the standard path C:\Python33\, if you chose another install directory or version of Python, please correct the line PYTHONDIR = C:/Python33 in the project file to refer to the correct directory.

3. In the Build view of your pythonapi project configuration (<Ctrl>+5) add the following custom build step (<Build Steps><Add Build Step><Custom Process Step>) and move it up to be the first build step before qmake and make:

   Command : swig.exe
   Arguments: -python -c++ ilwisobjects.i (pay attention to spaces)
   Working directory: %{sourceDir}\pythonapi

4. In the Run view of your pythonapi project configuration (<Ctrl>+5) add a make step (<Deployment><Add Deploy Step><Make>) with the following argument: install.

For the run-configuration of the final Python and SWIG parts I refer to the README file of the Python API, this file explains, how to tell python, where our extension ilwisobjects.pyd can find the ilwiscore.dll and all the derived dependencies. One approach is to adjust the system PATH environment.

1. After proceeding through the README file, your run-configuration can be set up like this:

   Executable: python

Arguments: test.py
Working directory: %{sourceDir}\pythonapi

Implementation

In this chapter the actual design behind the Python API is explained, as well as the important details within the implementation. This is important as there have been made some, at first, not completely obvious or even logical choices. The usual cause of those choices is some bugs with Qt-headers and SWIG. This had to be solved by creating more different types of headers, which made some things a little bit complex.

The general idea

The Python API is made using SWIG to wrap C++ headers, and this way generating a library usable by python code. It is however not as easy as it looks to wrap the ILWIS header files with SWIG. Because Python on windows is compiled using MSVC++ (Microsoft visual studio) and ILWIS is compiled using gcc for multiplatform purposes. This creates an interesting problem with headers during the SWIG compilation: MSVC imports the standard libraries because it gets called first, which causes gcc to not find its required libraries. This required us to create our own C++ header which tells SWIG exactly what to import from where.

The layout of the Python API. Blue is python, green is Qt gray is homemade

These new headers are generally copied from the original, with full path imports. Most of these classes are renamed by adding pythonapi_ in front of it they can be found in the pythonapi project.

The python classes

In order to actually convert certain things only found in C++ to python we needed to create some help classes in the pythonapi project, these classes are: pycontainer, pyerror and pyvariant.

Pycontainer

The translation between list and or vectors in c to python native lists by SWIG is not working as it should, so a work around was needed, this work around is the pycontainer class. It does this by creating a tuple of those lists and passing this on to the python part of the code. The currently implemented tuples are string, int (64 and 32), double and long .

Pyerror

There was need to translate the errors thrown by the Ilwis system, the pyerror class is meant to solve this. It translates the exceptions thrown by the c code to exceptions useable by the python code.

Extending

Extending the Python API will mainly be done by adding more Ilwis classes to its functionality. Adding a class to the Python API has to be done by creating a wrapper class for it so SWIG can actually compile it.

Creating a wrapper class

In order to add a class to the Python API you need to create a wrapper class for it, this is needed to avoid the problem with the swig headers. A wrapper class consists of a header and a cpp file. The Header file should have a member with a pointer to an object of the actual Ilwis class, the following step by step breakdown will go over all the details. For examples we refer to the actual source code, make sure to actually compare the original and the wrappers classes next to each other, you should also take a look at the example.

• The first thing to do is the creation of the actual wrapper class, we adapted the following naming scheme: pythonapi_<class>. The wrapper class requires both a header and a source file.
• The next thing to do would be copying the header file from the original to the wrapper version. Important is to remove all the imports from this file, also important is the declaration of a namespace in which you declare the class you are wrapping.
• The next thing to do is adding a pointer to the original class (If possible try to use an unique pointer, else use an shared pointer). This should be an internal member, either protected or private depending on the class.
• Afterwards you have to redirect the function calls to the correct Ilwis class, you can actually refer to those Ilwis classes, but it requires relative path imports. (recommended you study the code for examples)
• Last thing to do Is register the class in the SWIG interface file (ilwisobjects.i)

From here you have a wrapper class for the class you wanted to add to the Python API, It is however recommended you test it by adding an additional test to the python test (in python).

Extra extending

For some cases it might not be enough to just create a wrapper class, this can be because it there are unique errors that are thrown, but it might also be because there is a list of a different type. It is also possible that the SWIG interface file is incomplete with just the import. The big majority of those problems will be solved by one of the following.

PyContainer

In order to add a new tuple you need to add a new method to pycontainer. This method should look like this:

bool setTupleItem(PyObject *tuple, int i, const <your type> value)
	PyObject *v = <Python function for reading a value of this type>(value);
	if (!v) {
		Py_DECREF(tuple);
		tuple = NULL;
		return false;
	}
	PyTuple_SET_ITEM(tuple, i, v);
	return true;
}

The function you should use to get the pyobject are generally retrieved from PyLong, PyFloat and the likes. An example for Strings is: PyUnicode_FromString(value);

Pyerror

Adding a new error type the Python API is relatively simple. You simply add a new else if statement to the excisting if else loop in the translate_exception_type method of pyerror which should look like this:

else if(typeid(e) == typeid(<your Qt/Ilwis exception>)){
	return <the corresponding python exception type>;
}

You can find the possible python errors in the errors.h file inside your python install directory.

Pyvariant

If you want to extend the pyvariant file you need to add a method of the following make-up

<corresponding QType> PyVariant::__<Python type>__(){
if(!this->_data->canConvert(QVariant::<corresponding QType>))
	throw std::domain_error(QString("PyVariant cannot convert '%1' to int").arg(this->_data->toString()).toStdString());
else{
	bool ok = false;
	<corresponding QType> ret = this->_data->to<corresponding QType>(&ok);
	if (!ok)
		throw std::domain_error(QString("PyVariant cannot convert '%1' to <type> ").arg(this->_data->toString()).toStdString());
		return ret;
	}
}

Details about the construction of these kinds of methods and essentially all information about the c interface of python can be found here.

SWIG interface file

For certain classes it is required you edit the SWIG interface file within the Python API project (it is called objects.i). Generally this edit of the interface file is with static python code, which is required to solve the problem in the typeless object from the do method. This is usually done by inserting a static python method in the interface file, it is also possible that adding certain templates or definitions is needed.

Currently we have a to<Class> function in a couple of the Python API classes, these are needed to cast the object* pointer from the do method as mentioned before, which means extending the interface file is only needed when the particular class can be a return type of the do function.

Python code that is going to be added has to be inserted in the relevant class, this can be done using the following code block:

%extend pythonapi::<class to be inserted in> {
%insert("python") %{<python code>%}}

This is mainly used to create typecasting possibilities within the python interface, think about the do method that returns an object*, which within python is not cast-able. The templating done within this interface is generally used for simple base classes like Box2D etc. More details about the possibilities of this interface file can be found here.

Example: FeatureCoverage

In this example we will explain how to add FeatureCoverage to the Python API. Obviously this class already has a wrapper class and is working correctly, but the example may be relevant.

The first that has to be done is creating the header and the source file, following the correct naming method these files should be pythonape_featurecoverage.h and pythonape_featurecoverage.c. The first thing that should be done in the header file is creating a typdef of the actual featurecoverage class, you should however not import it. In our case:

namespace Ilwis {
    class FeatureCoverage;
    typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}

Once this is done we start to work in the pythonapi namespace, in this namespace we have to define the feature coverage class, in certain cases it is possible we also need to define a return class if this is needed for some of the access methods we will add next. Our header now looks like this:

namespace Ilwis {
    class FeatureCoverage;
    typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}
namespace pythonapi {
    class FeatureCoverage : public Coverage;
}

Next up is the adding of our access methods, because we want the python side of the code to able to actually use this class. Also important is a pointer to an instance of the actual Ilwis Featurecoverage, we need this to get the actual functionality inside our wrapper. Which access methods you add depends on the functionality you want the Python API to have. In this example we implement 2 different constructors, a method to get a feature iterator of the coverage, a query for the feature count and a new feature function. Additionally we need to implement a toFeatureCoverage method in this example, this helps casting the Object* in the do function to the correct type. Our header now looks like this:

namespace Ilwis {
	class FeatureCoverage;
	typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}

namespace pythonapi {
	class FeatureIterator;

	class FeatureCoverage : public Coverage{
		friend class FeatureIterator;
		friend class Engine;
		private:
			FeatureCoverage(Ilwis::IFeatureCoverage* coverage);
		public:
			FeatureCoverage();
			FeatureCoverage(std::string resource);
			FeatureIterator __iter__();
			unsigned int featureCount() const;
			Feature newFeature(Geometry& geometry);
			static FeatureCoverage* toFeatureCoverage(Object *obj);
	};
}

The only thing left to do is the actual implementation of these access methods, and adding the relevant code to the swig interface file. A great example of this is in the source file the above example, pythonapi_featurecoverage.c.

Footnotes

[1]: Not yet compiled with define Py_LIMITED_API (stable ABI). See Issue #25.
[TODO]: DLLs and other resources files are not yet available for everyone. A file sharing solution needs to be found.