Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python wrappings #1528

merged 98 commits into from Feb 15, 2021

Add python wrappings #1528

merged 98 commits into from Feb 15, 2021


Copy link

@phcerdan phcerdan commented Nov 13, 2020

PR Description

DGtal is wrapped to python using pybind11, and uploaded regularly to pypi for all platforms (Linux, Windows, MacOS)
and multiple python version (from 3.5 to latest) using azure-pipelines.

pip install dgtal
import dgtal


Due to the heavy templated nature of DGtal, only a subset of types
are available in Python. All the types in Z2i and Z3i namespaces are wrapped, they are defined in StdDefs.h.

Steps to add a new wrap:

Imagine we want to add wrappings for the class HyperRectDomain from the kernel module.

  • Inside the folder wrap/kernel add the files:

    • HyperRectDomain_types_py.h, to add (and reuse later) the specific types of HyperRectDomain that are going to be wrapped into python.
    #include "Common_types_py.h" // For DGtal::Python::Integer
    namespace DGtal {
        namespace Python {
            using Z2i  = DGtal::SpaceND<2, DGtal::Python::Integer>;
            using DomainZ2i = DGtal::HyperRectDomain <Z2i>;
        } // namespace Python
    } // namespace DGtal
    • HyperRectDomain_py.cpp. This is the file to compile where we define the function init_HyperRectDomain.
      Using the types from HyperRectDomain_types_py we wrap that type with the template function declare_HyperRectDomain.
    #include "dgtal_pybind11_common.h"
    #include "HyperRectDomain_types_py.h" // For DGtal::Python::DomainZ2i
    #include "HyperRectDomain_declare_py.h"
    void init_HyperRectDomain(py::module & m) {
      auto py_class_DomainZ2i = declare_HyperRectDomain<DGtal::Python::DomainZ2i>(m, "DomainZ2i");
    • HyperRectDomain_declare_py.h. This is where the templated function declare_HyperRectDomain is defined, and where the actual pybind11 wrappings are done.
  • Add a call to init_HyperRectDomain to the init_dgtal_kernel function inside the kernel_init.cpp.
    If you are adding a new module, i.e. foo, the module foo has to be added to the parent init file dgtal_init_py.cpp in the wrap folder.
    All this modularity allows parallel compilation to speed up the build process.

  • Add the .cpp files to the apropiate CMakeLists.txt.
    In this case we add HyperRectDomain_py.cpp to wrap/kernel/CMakeLists.txt.
    In the case we are adding a new module, use add_subdirectory(foo) in wrap/CMakeLists.txt

  • Add python tests for your new wraps.
    Add the file to wrap/test/kernel, and using
    pytest exercise all the constructors, functions and access to data members exposed in your wrap.

    Also, add the test file to wrap/test/kernel/CMakeLists.txt to allow ctest to discover and execute the test.
    You can then ctest -R test_Hyper* -V to execute all the tests in that file.
    Please remember that pytest requires that your test functions start with test_, for example def test_constructors():.
    Also, pytest allows to parametrize your test functions, allowing to test the wrappings for multiple types automatically.
    See existing examples to guide you in the process.

Classes wrapped


  • PointVector: Point2D, Point3D, RealPoint2D, RealPoint3D
  • HyperRectDomain: DomainZ2i, DomainZ3i
  • DigitalSetBySTLVector: DigitalSetZ2i, DigitalSetZ3i
  • KhalimskyPreSpaceND: PreCell2D, SPreCell2D, KPreSpace2D (and 3D)
  • KhalimskySpaceND: Cell2D, SCell2D, KSpace2D (and 3D)
  • MetricAdjacency: 2D: Adj4, Adj8, 3D: Adj6, Adj18, Adj26
  • DigitalTopology: DT4_8, DT8_4, DT6_18, DT18_6, DT6_26, DT26_6
  • Object: Object4_8, Object8_4, Object6_18, Object18_6, Object6_26, Object26_6


  • Provide bridge from numpy to ImageContainersBySTLVector.
    Check for usage, in a nutshell:
    Use external library (ITK, opencv) and use their numpy bridge.
    import itk
    import dgtal
    itk_image = itk.imread(image_filename)
    itk_np_array = itk.GetArrayViewFromImage(itk_image)
    # itk_np_array = itk_np_array.astype('int32') # if conversion is needed.
    ImageContainer = dgtal.images.ImageContainerByVector2DUnsignedChar
    dgtal_image = ImageContainer(itk_np_array)
  • [NA] Wrap internal DGtal IO classes. Discarded for now (@dcoeurjo)


  • CubicalComplex CubicalComplex2D, CubicalComplex3D (using std::unordered_map as CellContainer).
  • VoxelComplex: VoxelComplex (3D-only)

Factory functions

Factory functions are defined in to ease the construction of the wrapped types:

  • Point: dgtal.Point(dim, dtype, data)
    dgtal.Point will return a dgtal.kernel.Point2D if dim == 2 and dtype == 'int32', or a dgtal.kernel.Point3D
    if dim == 3 and dtype == 'float'. The data argument will be passed to the constructor, accepting list,
    tuples and numpy.arrays of the right type and dimension.
    Also: dgtal.Point(data=[2,3]) will return a dgtal.kernel.Point2D, dimension will be inferred.
  • Domain: dgtal.Domain(lower_bound, upper_bound)
  • DigitalSet: dgtal.DigitalSet(domain)
  • KPreSpace: dgtal.KPreSpace(dim)
  • PreCell: dgtal.PreCell(dim, point), SPreCell: dgtal.PreCell(dim, point, positive)
  • KSpace: dgtal.KSpace(dim)
  • MetricAdjacency: dgtal.MetricAdjacency(dim, max_norm).
    Example: adj26 = dgtal.MetricAdjacency(dim=3, max_norm=3)
  • DigitalTopology: dgtal.DigitalTopology(foreground, background, properties)
  • Object: dgtal.Object(topology, domain, point_set, connectedness)
  • ImageContainer: dgtal.ImageContainer(dtype, domain, data, lower_bound_ijk)

Dev notes

Building wraps

Compile DGtal with option DGTAL_WRAP_PYTHON=ON.
This will create a folder in your build tree named dgtal.
From the build folder this will work: python -c 'import dgtal', to use the package from the build tree elswhere, create a virtualenviroment:

mkvirtualenv dgtal-build

And then create a file named: dgtal.pth in ~/.virtualenvs/dgtal-build/lib/python3.X/site-packages with the content:


Now you can use workon dgtal-build from anywhere, and python -c 'import dgtal' will work.
Remember that if you modify and rebuild DGtal wrappings, and you are in a ipython session, or jupyter, you will need to restart or reload
the ipython kernel for the changes to be loaded.

Releasing to pypi


It is recommended that you setup a virtual environment for deploy, so azure-cli and twine do not have to be installed in the system python.

workon dgtal-build

For azure download

For pypi upload

  • Setup a personal access token in pypi (or know the password!).
  • pip install twine for easy upload.

Download the wheels from Azure (Linux/MacOS only)

First, login into azure if you haven't already, using your PersonalAccessToken.

az devops login --organization


Then, go to the pipeline PythonDeploy, and select the commit from where you want the wheels.
In the URL, grab the buildId number, in this case: 246.

With that id, use the script located in dgtal-src/wrap/deploy/

/path/ 246

All the wheels for all platforms and all python versions will be downloaded to /tmp/dist.

Upload to pypi

 python -m twine upload /tmp/dist/* --verbose 

For testing that the package complies with all requisites, you might want to first upload to test.pypi.

python -m twine upload --repository-url /tmp/dist/* --verbose 

Increase version in

After manual deployment to pypi, don't forget to increase the version in wrap/deploy/

A package with the same version cannot be uploaded to pypi (i.e. cannot override).

The provided python script can increment the version automatically:

./wrap/deploy/ --no-write

incrementing version component: patch | version_index: 2
current_version:  0.0.1
final_version:  0.0.2
Not writing to any file, remove -n or --no-write argument if wanted.

By default the script increase the patch version. You can increment bigger components with -c minor or -c major. Use --help for other options.

Remove the --no-write option to commit the new version to the file


  • Unit-test of your feature with pytest.
  • Doxygen documentation of the code completed (classes, methods, types, members...)
  • Documentation module page added or updated.
  • New entry in the added.
  • No warning raised in Debug cmake mode (otherwise, Travis C.I. will fail).
  • All continuous integration tests pass (Travis & appveyor)

@phcerdan phcerdan force-pushed the pybind11 branch 9 times, most recently from 213807e to 2672faf Compare November 18, 2020 13:42
@phcerdan phcerdan force-pushed the pybind11 branch 8 times, most recently from 3722e47 to 0a2cbf4 Compare December 1, 2020 08:52
@phcerdan phcerdan force-pushed the pybind11 branch 13 times, most recently from 5e32111 to 3c05bb4 Compare December 9, 2020 14:20
Because there is a construct method, the base method is not inherited
Start adding factory functions to `` to ease the construction of the wrapped types:
Add `dgtal.Point(dim, dtype, data)`:

`dgtal.Point` will return a `dgtal.kernel.Point2D` if `dim == 2` and `dtype == 'int32'`, or a `dgtal.kernel.Point3D`
if `dim == 3` and `dtype == 'float'`. The `data` argument will be passed to the constructor, accepting list,
tuples and `numpy.array`s of the right type and dimension.
dgtal.Domain(dim=2, lower_bound=dgtal.Point(dim=2).zero,
Also include PreCell and SPreCell helpers.
 Accepting dim, and max_norm parameters
Change py::arg("props") for "properties" in `DigitalTopology` for consistency.
Also add static method: `adjacency_pair_string` to the python wrappings
of Digital Topology.

MetricAdjacency `__str__` now shows typestr (Adj8, Adj26, etc.)
Works with np.arrays as well.
The check of valid dimensions, only 2 or 3 allowed is still in place.
The problem was with CowPtr.
Solved using `const TT`, which force CowPtr to avoid the copy.

Same solution for `topology()` method.
In `foreground()`, and `background()` methods.
These `MetricAdjacency` members are references and non-copyable.

RuntimeError: return_value_policy = copy, but type DGtal::MetricAdjacency<DGtal::SpaceND<2u, int>, 2u, 2u> is non-copyable!

Set `py::return_value_policy::reference_internal` instead of automatic (it was doing copies)
Accepts domain, or data as a numpy.array

Test factory added in test_ImageContainerBySTLVector
CI failed with the following error:

warning: argument 'set' of command @param is not found in the argument list of
DGtal::VoxelComplex< TKSpace, TCellContainer >::construct(typename TDigitalSet)

It's a bug in Doxygen, workorounded to pass CI
./wrap/deploy/ --no-write

incrementing version component: patch | version_index: 2
current_version:  0.0.1
final_version:  0.0.2
Not writing to any file, remove -n or --no-write argument if wanted.
Copy link


@dcoeurjo dcoeurjo self-requested a review February 15, 2021 11:45
Copy link

all green.. thanks a lot @phcerdan

@dcoeurjo dcoeurjo merged commit 8ca3c8a into DGtal-team:master Feb 15, 2021
@phcerdan phcerdan deleted the pybind11 branch May 4, 2021 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

2 participants