ILWIS Python API TechDoc
This document is meant for developers who want to extend or expand the Ilwis 4 python API. This python API is based on SWIG and created in C++ using QTCreator, it is heavily recommended you have at least a general knowledge about these subjects. It is also recommended you have a general idea about the general functionality of Ilwis itself. This information can be found in the actual Ilwis 4 documentation.
The goal of this document is to educate the reader about the Python API of Ilwis 4. We will go over the setup and idea behind its development, and we will also talk about how to set up the environment for yourself. The last part of this document will be about how you can actually expand or edit the python API yourself as a developer.
Please note the difference in \\ and / when dealing with paths to files and directories. A small error in those can create strange and hard to track down bugs.
In order to actually produce thing for the Ilwis 4 Python API it is required you install a couple of programs. These programs form you working environment and are necessary to actually perform certain operations. This working environment exists of an editor(QT) and a Python 3 installation combined with the toolbox that actually made the creation of this API possible: SWIG.
In the following paragraphs we will go over the setup and configuration of the working environment it is however assumed you have the Ilwis source code and the project folder (with the libraries etc.), if not the most recent version of the code can be found on the 52 North GitHub and the project folder on Google Drive.
Ilwis 4 is created in QTCreator because of its multiplatform advantage over visual studio, this however means that in order to run and compile Ilwis 4 you need a QT installation. And of course The Ilwis 4 project has to be configured correctly within QT this however will be done in another chapter.
First thing first: we have to install QTCreator. It is recommended to install all packages, this avoids the risk of missing a vital package later on.
Once installed we have to setup the Ilwis 4 Projects within QT:
- Select projects from the
<quick menu><build and run><build> -
<File><OpenProject>and do this for:<your Ilwis folder>\projects\IlwisConnectors\gdalconnector.pro<your Ilwis folder>\projects\IlwisConnectors\Ilwis3connector.pro<your Ilwis folder>\projects\IlwisConnectors\pythonapi.pro<your Ilwis folder>\projects\IlwisConnectors\baseoperations.pro<your Ilwis folder>\projects\IlwisCore\core.pro<your Ilwis folder>\projects\IlwisCore\Ilwisscript.pro<your Ilwis folder>\projects\IlwisCore\projectionimplproj4.pro<your Ilwis folder>\projects\IlwisCore\rasteroperations.pro<your Ilwis folder>\projects\IlwisCore\internalconnector.pro
Make sure the build directory is: <your Ilwis>\projects\temp\core\Debug
- You can use the standard settings for now. You should see the configuration chapter for the actual configurations. 1. Done, wheeeeeeee!
The python API is compiled for python 3.0 and higher1, which means we recommend you install the most recent version. Again it is recommended to do a full installation. It might also be useful to install a python editor for the creation and eventual running of test scripts.
From the description on the swig site:
"SWIG is an interface compiler that connects programs written in C and C++ with scripting languages such as Perl, Python, Ruby, and Tcl. It works by taking the declarations found in C/C++ header files and using them to generate the wrapper code that scripting languages need to access the underlying C/C++ code. In addition, SWIG provides a variety of customization features that let you tailor the wrapping process to suit your application."
SWIG does not require an actual installation, it only needs to be downloaded and unzipped in the correct folder. Make sure you download the correct version for your operating system. Right now we use SWIG 2.12 as at the moment of writing this is the most recent version.
If you want to be able to actually compile your Python API you will have to configure all the projects correctly within QT. Also there have to be made a couple of additions to your path environment, since you want to run SWIG during the compilation.
In order to edit the configurations inside QT you have to select project from the quick menu. Inside this menu you can select your projects up top and below that you can select which settings to adjust. The majority of our configurations are done in the Build&Run menu, it is however important to note that this menu exists of 2 sub menus: “Build” and “run”. The required configurations can all be done within those 2 sub menus unless stated otherwise.
-
In the pythonapi project add a custom build step(it has to be the first build step):
Command :swig.exe
Arguments:-python -c++ ilwisobjects.i(pay attention to spaces)
Working directory:<your Ilwis folder>\projects\IlwisConnectors\pythonapi
You can find this under<build><build steps><add build step>. Make sure this is the first step that is taken, by moving it all the way up. -
In the pythonapi projects add a make step with the following argument:
install.
You can find this under<run><deployment><add deploy step>make step.
For the configuration of the Python and SWIG parts I refer to the user README of the Python API.
In this chapter the actual design behind the Python API is explained, as well as the important details within the implementation. This is important as there have been made some, at first, not completely obvious or even logical choices. The usual cause of those choices is some bugs with QT-headers and SWIG. This had to be solved by creating more different types of headers, which made some things a little bit complex.
The python API is made using SWIG to wrap C++ headers, and this way generating a library usable by python code. It is however not as easy as it looks to wrap the ILWIS header files with SWIG. Because Python on windows is compiled using MSVC++ (Microsoft visual studio) and ILWIS is compiled using gcc for multiplatform purposes. This creates an interesting problem with headers during the SWIG compilation: MSVC imports the standard libraries because it gets called first, which causes gcc to not find its required libraries. This required us to create our own C++ header which tells SWIG exactly what to import from where.
.png)
The layout of the Python API. Blue is python, green is QT gray is homemade
These new headers are generally copied from the original, with full path imports. Most of these classes are renamed by adding pythonapi_ in front of it they can be found in the pythonapi project.
In order to actually convert certain things only found in C++ to python we needed to create some help classes in the pythonapi project, these classes are: pycontainer, pyerror and pyvariant.
The translation between list and or vectors in c to python native lists by SWIG is not working as it should, so a work around was needed, this work around is the pycontainer class. It does this by creating a tuple of those lists and passing this on to the python part of the code. The currently implemented tuples are string, int (64 and 32), double and long .
There was need to translate the errors thrown by the Ilwis system, the pyerror class is meant to solve this. It translates the exceptions thrown by the c code to exceptions useable by the python code.
Extending the python API will mainly be done by adding more Ilwis classes to its functionality. Adding a class to the Python API has to be done by creating a wrapper class for it so SWIG can actually compile it.
In order to add a class to the python API you need to create a wrapper class for it, this is needed to avoid the problem with the swig headers. A wrapper class consists of a header and a cpp file. The Header file should have a member with a pointer to an object of the actual Ilwis class, the following step by step breakdown will go over all the details. For examples we refer to the actual source code, make sure to actually compare the original and the wrappers classes next to each other, you should also take a look at the example.
- The first thing to do is the creation of the actual wrapper class, we adapted the following naming scheme:
pythonapi_<class>. The wrapper class requires both a header and a source file. - The next thing to do would be copying the header file from the original to the wrapper version. Important is to remove all the imports from this file, also important is the declaration of a namespace in which you declare the class you are wrapping.
- The next thing to do is adding a pointer to the original class (If possible try to use an unique pointer, else use an shared pointer). This should be an internal member, either protected or private depending on the class.
- Afterwards you have to redirect the function calls to the correct Ilwis class, you can actually refer to those Ilwis classes, but it requires relative path imports. (recommended you study the code for examples)
- Last thing to do Is register the class in the SWIG interface file (
ilwisobjects.i)
From here you have a wrapper class for the class you wanted to add to the python API, It is however recommended you test it by adding an additional test to the python test (in python).
For some cases it might not be enough to just create a wrapper class, this can be because it there are unique errors that are thrown, but it might also be because there is a list of a different type. It is also possible that the SWIG interface file is incomplete with just the import. The big majority of those problems will be solved by one of the following.
In order to add a new tuple you need to add a new method to pycontainer. This method should look like this:
bool setTupleItem(PyObject *tuple, int i, const <your type> value)
PyObject *v = <Python function for reading a value of this type>(value);
if (!v) {
Py_DECREF(tuple);
tuple = NULL;
return false;
}
PyTuple_SET_ITEM(tuple, i, v);
return true;
}The function you should use to get the pyobject are generally retrieved from PyLong, PyFloat and the likes. An example for Strings is: PyUnicode_FromString(value);
Adding a new error type the python API is relatively simple. You simply add a new else if statement to the excisting if else loop in the translate_exception_type method of pyerror which should look like this:
else if(typeid(e) == typeid(<your QT/Ilwis exception>)){
return <the corresponding python exception type>;
}You can find the possible python errors in the errors.h file inside your python install directory.
If you want to extend the pyvariant file you need to add a method of the following make-up
<corresponding QType> PyVariant::__<Python type>__(){
if(!this->_data->canConvert(QVariant::<corresponding QType>))
throw std::domain_error(QString("PyVariant cannot convert '%1' to int").arg(this->_data->toString()).toStdString());
else{
bool ok = false;
<corresponding QType> ret = this->_data->to<corresponding QType>(&ok);
if (!ok)
throw std::domain_error(QString("PyVariant cannot convert '%1' to <type> ").arg(this->_data->toString()).toStdString());
return ret;
}
}Details about the construction of these kinds of methods and essentially all information about the c interface of python can be found here.
For certain classes it is required you edit the SWIG interface file within the Python API project (it is called objects.i). Generally this edit of the interface file is with static python code, which is required to solve the problem in the typeless object from the do method. This is usually done by inserting a static python method in the interface file, it is also possible that adding certain templates or definitions is needed.
Currently we have a to<Class> function in a couple of the python API classes, these are needed to cast the object* pointer from the do method as mentioned before, which means extending the interface file is only needed when the particular class can be a return type of the do function.
Python code that is going to be added has to be inserted in the relevant class, this can be done using the following code block:
%extend pythonapi::<class to be inserted in> {
%insert("python") %{<python code>%}}This is mainly used to create typecasting possibilities within the python interface, think about the do method that returns an object*, which within python is not cast-able. The templating done within this interface is generally used for simple base classes like Box2D etc. More details about the possibilities of this interface file can be found here.
In this example we will explain how to add FeatureCoverage to the Python API. Obviously this class already has a wrapper class and is working correctly, but the example may be relevant.
The first that has to be done is creating the header and the source file, following the correct naming method these files should be pythonape_featurecoverage.h and pythonape_featurecoverage.c. The first thing that should be done in the header file is creating a typdef of the actual featurecoverage class, you should however not import it. In our case:
namespace Ilwis {
class FeatureCoverage;
typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}Once this is done we start to work in the pythonapi namespace, in this namespace we have to define the feature coverage class, in certain cases it is possible we also need to define a return class if this is needed for some of the access methods we will add next. Our header now looks like this:
namespace Ilwis {
class FeatureCoverage;
typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}
namespace pythonapi {
class FeatureCoverage : public Coverage;
}Next up is the adding of our access methods, because we want the python side of the code to able to actually use this class. Also important is a pointer to an instance of the actual Ilwis Featurecoverage, we need this to get the actual functionality inside our wrapper. Which access methods you add depends on the functionality you want the python API to have. In this example we implement 2 different constructors, a method to get a feature iterator of the coverage, a query for the feature count and a new feature function. Additionally we need to implement a toFeatureCoverage method in this example, this helps casting the Object* in the do function to the correct type. Our header now looks like this:
namespace Ilwis {
class FeatureCoverage;
typedef IlwisData<FeatureCoverage> IFeatureCoverage;
}
namespace pythonapi {
class FeatureIterator;
class FeatureCoverage : public Coverage{
friend class FeatureIterator;
friend class Engine;
private:
FeatureCoverage(Ilwis::IFeatureCoverage* coverage);
public:
FeatureCoverage();
FeatureCoverage(std::string resource);
FeatureIterator __iter__();
unsigned int featureCount() const;
Feature newFeature(Geometry& geometry);
static FeatureCoverage* toFeatureCoverage(Object *obj);
};
}The only thing left to do is the actual implementation of these access methods, and adding the relevant code to the swig interface file. A great example of this is in the source file the above example, pythonapi_featurecoverage.c.