[Python Bindings] error while printing a python generated module in c++

[ravil-mobile]

Hi all,

I would like to build a small EDSL. The idea is to have the dsl frontend in python and handle lowering in c++.
Here is a very small example which I wrote to reproduce the error:

from mlir.ir import Context, Module, get_dialect_registry
from mlir.ir import InsertionPoint, Location, F32Type
from mlir.dialects import func, arith
from mlir.dialects import arith
from edsl_cpp import entry


def main():
  registry = get_dialect_registry()
  context = Context()
  context.append_dialect_registry(registry)
  module = None
  with context:
    module = Module.create(Location.unknown())
    with InsertionPoint(module.body), Location.unknown():
      fp_type = F32Type.get()
      function = func.FuncOp("test", ([fp_type, fp_type, fp_type], [fp_type]))
    
      with InsertionPoint(function.add_entry_block()) as block:
        one = function.arguments[0]
        two = function.arguments[1]
        mult_res = arith.MulFOp(one, two)
    
        three = function.arguments[2]
        res = arith.AddFOp(mult_res, three)
        func.ReturnOp([res])

  entry(module._CAPIPtr)


if __name__ == "__main__":
  main()

#include <pybind11/pybind11.h>
#include "mlir/CAPI/IR.h"
#include "mlir-c/IR.h"
#include "mlir-c/RegisterEverything.h"
#include "mlir-c/Bindings/Python/Interop.h"
#include "mlir/CAPI/Support.h"
#include <stdexcept>

PYBIND11_MODULE(edsl_cpp, m) {
  m.doc() = "edsl python bindings";
  m.def("entry", [](pybind11::object capsule){
    MlirModule mlirModule = mlirPythonCapsuleToModule(capsule.ptr());
    if (mlirModuleIsNull(mlirModule)) {
      throw std::runtime_error("empty module");
    }
    MlirContext context = mlirModuleGetContext(mlirModule);
    if (mlirContextIsNull(context)) {
      throw std::runtime_error("empty context");
    }
    auto module = unwrap(mlirModule);
    module->dump();
  });
}

If necessary, I can share the whole source code. Anyway, when I am executing the python script I get the following error:

dialect has no registered type printing hook
UNREACHABLE executed at <path-to-llvm>/llvm-project/mlir/include/mlir/IR/Dialect.h:111!

Does anybody have an idea what is missing in my example?
I can dump a module in python without any problem by something is wrong in the cpp-part.

I am using 461274b81d8641eab64d494accddc81d7db8a09e commit but I bet it is going to be the same with the main branch.

[llvmbot]

@llvm/issue-subscribers-mlir-python

Author: None (ravil-mobile)

Hi all,

I would like to build a small EDSL. The idea is to have the dsl frontend in python and handle lowering in c++.
Here is a small example:

from mlir.ir import Context, Module, get_dialect_registry
from mlir.ir import InsertionPoint, Location, F32Type
from mlir.dialects import func, arith
from mlir.dialects import arith
from edsl_cpp import entry


def main():
  registry = get_dialect_registry()
  context = Context()
  context.append_dialect_registry(registry)
  module = None
  with context:
    module = Module.create(Location.unknown())
    with InsertionPoint(module.body), Location.unknown():
      fp_type = F32Type.get()
      function = func.FuncOp("test", ([fp_type, fp_type, fp_type], [fp_type]))
    
      with InsertionPoint(function.add_entry_block()) as block:
        one = function.arguments[0]
        two = function.arguments[1]
        mult_res = arith.MulFOp(one, two)
    
        three = function.arguments[2]
        res = arith.AddFOp(mult_res, three)
        func.ReturnOp([res])

  entry(module._CAPIPtr)


if __name__ == "__main__":
  main()

#include <pybind11/pybind11.h>
#include "mlir/CAPI/IR.h"
#include "mlir-c/IR.h"
#include "mlir-c/RegisterEverything.h"
#include "mlir-c/Bindings/Python/Interop.h"
#include "mlir/CAPI/Support.h"
#include <stdexcept>

PYBIND11_MODULE(edsl_cpp, m) {
  m.doc() = "edsl python bindings";
  m.def("entry", [](pybind11::object capsule){
    MlirModule mlirModule = mlirPythonCapsuleToModule(capsule.ptr());
    if (mlirModuleIsNull(mlirModule)) {
      throw std::runtime_error("empty module");
    }
    MlirContext context = mlirModuleGetContext(mlirModule);
    if (mlirContextIsNull(context)) {
      throw std::runtime_error("empty context");
    }
    auto module = unwrap(mlirModule);
    mlir::AsmState state(module);
    module->dump();
  });
}

If necessary, I can share the whole source code. Anyway, when I am executing the python script I get the following error:

dialect has no registered type printing hook
UNREACHABLE executed at <path-to-llvm>/llvm-project/mlir/include/mlir/IR/Dialect.h:111!

Does anybody have an idea what is missing in my example?
I can dump a module in python without any problem by something is wrong in the cpp-part.

[makslevental]

You're trying to print outside of the with context - just indent entry(module._CAPIPtr).

[ravil-mobile]

You're trying to print outside of the with context - just indent entry(module._CAPIPtr).

Indenting entry(module._CAPIPtr) to the right results in the same error.

Actually, I create a context outside of the with statement. So, context is a local variable of main. It is supposed to be destroyed at the end of the function

[makslevental]

This kind of stuff

  m.def("entry", [](pybind11::object capsule){
    MlirModule mlirModule = mlirPythonCapsuleToModule(capsule.ptr());
    ...
    auto module = unwrap(mlirModule);
    module->dump();

is "sharp edges". It can be done but takes some careful coordination. I can help you debug further but I'm gonna need to see more of the source. Feel free to ping me on the discord (same handle).

[ravil-mobile]

This kind of stuff

  m.def("entry", [](pybind11::object capsule){
    MlirModule mlirModule = mlirPythonCapsuleToModule(capsule.ptr());
    ...
    auto module = unwrap(mlirModule);
    module->dump();

is "sharp edges". It can be done but takes some careful coordination. I can help you debug further but I'm gonna need to see more of the source. Feel free to ping me on the discord (same handle).

Thanks @makslevental,

You can find the source code here: https://github.com/ravil-mobile/edsl-mlir/tree/main

[makslevental]

Well for starters, dialect_libs and extension_libs in

target_link_libraries(edsl_cpp PUBLIC
  ${dialect_libs}
  ${extension_libs}
  MLIREdslEntry
)

in your CMakeLists txt aren't actually defined/set to anything (so you're not actually linking your C extension to anything).

But overall I would advise you to not diverge from using upstream CMake targets/functions/helpers - I realize they're complex and verbose but it's damn near impossible to get things to build/work without them (and you'd probably end up re-rolling most of them even if you did do it "lone wolf" style).

[ravil-mobile]

Well for starters, dialect_libs and extension_libs in

target_link_libraries(edsl_cpp PUBLIC
  ${dialect_libs}
  ${extension_libs}
  MLIREdslEntry
)

in your CMakeLists txt aren't actually defined/set to anything (so you're not actually linking your C extension to anything).

But overall I would advise you to not diverge from using upstream CMake targets/functions/helpers - I realize they're complex and verbose but it's damn near impossible to get things to build/work without them (and you'd probably end up re-rolling most of them even if you did do it "lone wolf" style).

@makslevental, yes. You are correct.

Library, in this particular case, is handled here

https://github.com/ravil-mobile/edsl-mlir/blob/main/edsl/compiler/lib/CMakeLists.txt

https://github.com/ravil-mobile/edsl-mlir/blob/d8e8d57dd9131cb338677941fc045a92bfd678c8/edsl/compiler/CMakeLists.txt#L37

It doesn't seem to be a problem.

I did try to explicitly link all the stuff which I am using - e.g.,

target_link_libraries(edsl_cpp PUBLIC
  MLIRSupport
  MLIRAsmParser
  MLIRExecutionEngine
  MLIRDialect
  MLIRDialectUtils
  MLIRArithDialect
  MLIREdslEntry
)

But the problem remains.