Building issues pycapnp v1.3.0 vs capnproto v0.10.3

[sfavazza]

Hello there,

I am evaluating pycapnp in a docker container but I cannot successfully build the generated code for the calculator example.

Steps to reproduce:

• build a test image with the following Dockerfile:

FROM python:3.8
ARG USER=cooluser
ARG HOME=/home/${USER}
RUN useradd -ms /bin/bash ${USER}


# requirements for cap'n proto
RUN apt-get update && apt-get install -y \
    autoconf \
    automake \
    libtool \
    pkg-config \
    cmake

# install "cap'n proto" from source (on master specific commit)
ARG TRGT_TAG="v0.10.3"
ARG BUILD_DIR=/opt/capnproto
RUN git clone https://github.com/capnproto/capnproto.git ${BUILD_DIR} \
    && cd ${BUILD_DIR} \
    && git checkout ${TRGT_TAG} \
    && cd c++ \
    && autoreconf -i \
    && ./configure \
    && make -j6 \
    && make install

RUN mkdir /data && chown ${USER}:${USER} /data

USER ${USER}

ENV PATH=${HOME}/.local/bin:$PATH

RUN cd ${HOME} \
    && pip install virtualenv \
    && virtualenv .pyvenv \
    && pip install \
    jinja2 \
    'cython<3'

ENV PATH=${HOME}/.pyvenv/bin:$PATH

WORKDIR /data
ENTRYPOINT [ "bash" ]

• mkdir ~/pycapnp-build-test
• docker run --rm -ti -v ~/pycapnp-build-test:/data
• from within the docker shell:

git clone https://github.com/capnproto/pycapnp.git .
pip install .
cd examples
capnpc -oc++ calculator.capnp
capnpc -ocython calculator.capnp
python setup_capnp.py build_ext --inplace

I got the following error:

Warning: passing language='c++' to cythonize() is deprecated. Instead, put "# distutils: language=c++" in your .pyx or .pxd file(s)
Compiling calculator_capnp_cython.pyx because it changed.
[1/1] Cythonizing calculator_capnp_cython.pyx
/home/cooluser/.pyvenv/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /data/examples/calculator_capnp_cython.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...
        def __set__(self, value):
            self._set_literal(value)
    cpdef _get_previousResult(self):
        return self.thisptr_child.getPreviousResult()
        
    cpdef _set_previousResult(self, interface_t value):
                                   ^
------------------------------------------------------------

calculator_capnp_cython.pyx:174:36: 'interface_t' is not a type identifier

Error compiling Cython file:
------------------------------------------------------------
...
        self.thisptr_child = (<C_DynamicStruct_Builder>struct.thisptr).asCalculator_Expression_call()
    
    cpdef _get_function(self):
        return self.thisptr_child.getFunction()
        
    cpdef _set_function(self, interface_t value):
                             ^
------------------------------------------------------------

calculator_capnp_cython.pyx:324:30: 'interface_t' is not a type identifier
Traceback (most recent call last):
  File "setup_capnp.py", line 37, in <module>
    ext_modules=cythonize('*_capnp_cython.pyx', language="c++")
  File "/home/cooluser/.pyvenv/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1115, in cythonize
    cythonize_one(*args)
  File "/home/cooluser/.pyvenv/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1238, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: calculator_capnp_cython.pyx

I tried with the cap'n proto library version v0.10.3 because of the v1.3.0 (2023-01-26) changelog, though trying with more recent cap'n proto versions (namely v1.0.0 and v1.0.1) gives the same result.

Any idea where the interface_t type is defined?

[tobiasah]

Have you tried the latest beta release (2.0.0.b1)?

[LasseBlaauwbroek]

You are trying to use an incomplete attempt at implementing a capnp->cython compiler. This code was never completed and has never worked.

[sfavazza]

Have you tried the latest beta release (2.0.0.b1)?

I get 1648 lines on the stderr, for brevity I report the first chunk and the last one:

/pyvenv/lib/python3.8/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /data/modules/pycapnp/examples/calculator_capnp_cython.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Error compiling Cython file:
------------------------------------------------------------
...


import capnp
import calculator_capnp

from capnp.includes.types cimport *
^
------------------------------------------------------------

calculator_capnp_cython.pyx:22:0: 'capnp/includes/types.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...

import capnp
import calculator_capnp

from capnp.includes.types cimport *
from capnp cimport helpers
^
------------------------------------------------------------

calculator_capnp_cython.pyx:23:0: 'capnp.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...

import capnp
import calculator_capnp

from capnp.includes.types cimport *
from capnp cimport helpers
^
------------------------------------------------------------

calculator_capnp_cython.pyx:23:0: 'capnp/helpers.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
import capnp
import calculator_capnp

from capnp.includes.types cimport *
from capnp cimport helpers
from capnp.includes.capnp_cpp cimport DynamicValue, Schema, VOID, StringPtr, ArrayPtr, Data
^
------------------------------------------------------------

calculator_capnp_cython.pyx:24:0: 'capnp/includes/capnp_cpp.pxd' not found

Last chunk:

Error compiling Cython file:
------------------------------------------------------------
...
            return self._get_function()
        def __set__(self, value):
            self._set_function(value)
    cpdef _get_params(self):
        cdef DynamicValue.Builder temp = self.thisptr_child.getParams()
        return to_python_builder(temp, self._parent)
              ^
------------------------------------------------------------

calculator_capnp_cython.pyx:335:15: 'to_python_builder' is not a constant, variable or function identifier

Error compiling Cython file:
------------------------------------------------------------
...

    def to_dict(self, verbose=False, ordered=False):
        ret = {
        
        
        'function': _to_dict(self.function, verbose, ordered),
                   ^
------------------------------------------------------------

calculator_capnp_cython.pyx:356:20: '_to_dict' is not a constant, variable or function identifier

Error compiling Cython file:
------------------------------------------------------------
...
        
        
        'function': _to_dict(self.function, verbose, ordered),
        
        
        'params': _to_dict(self.params, verbose, ordered),
                 ^
------------------------------------------------------------

calculator_capnp_cython.pyx:359:18: '_to_dict' is not a constant, variable or function identifier
Traceback (most recent call last):
  File "setup_capnp.py", line 37, in <module>
    ext_modules=cythonize('*_capnp_cython.pyx', language="c++")
  File "/pyvenv/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1115, in cythonize
    cythonize_one(*args)
  File "/pyvenv/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1238, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: calculator_capnp_cython.pyx

Though I just realized that the steps I performed are automatically performed in background (namely the code generation steps with capnpc), so I think the approach I reported here needs more configuration (it would explain why the compilation fails).

To run the example was enough to simply perform:

python3 async_calculator_server.py 0.0.0.0:1234  &
python3 async_calculator_client.py 0.0.0.0:1234

Feel free to close this issue!