Segmentation fault when initializing an in-memory database

[joedsilva]

• MonetDBLite-Python version: git clone from master (also happens with pip install)
• Python version: 3.6.2 also happens in 3.5.2
• Pip version:
• Operating System: Ubuntu 16.04.6 LTS

Description

We were originally trying to use monetdblite from within PostgreSQL python UDFs (long story) and it was crashing the postgres engine itself.

Upon closer inspection, we noticed that even if you just start monetdblite from a regular Python interpreter, do the init (seems to work) and then wait and count till 50 seconds, the Python interpreter will crash. This is reproducible. Although it does not seem to be an issue if you do not initialize it as an in-memory database.

What I Did

$ python3
Python 3.6.2 (default, Jul 17 2017, 23:14:31)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import monetdblite
>>> monetdblite.init(':memory:')
>>> Segmentation fault (core dumped)

[kutsurak]

Hi @joedsilva. First I would like to apologize for the late answer.

Support for in memory databases is problematic in MonetDBLite and we know it for some time now, but this project is not actively developed anymore, so this is not going to be fixed.

We are however preparing to release a new project in the space of embedded analytical databases, based on the official codebase of MonetDB. We are a few weeks away form the announcement of the beta release. One of the design goals is to be backwards compatible with MonetDBLite, and it has support for in-memory databases. If you are interested in a preview of this new project I can provide you contact information for someone who can help you install and test it. It is still a bit rough around the edges, but feedback at this stage would be very useful to us.

[joedsilva]

Hello @kutsurak. Thank you, for your response.

We would be very much interested in the new venture as our work is part of a research framework and we are open to experimenting with things. If it is DuckDB, we had similar problems there as well (although it crashes only within the PostgreSQL UDF - not when used standalone. I suspect the issues are pertaining to multi-threading of the database when used inside an embedded Python application like PostgreSQL that has its own multi-threading).

As presently, our only options (we need an embedded relational database in Python that is a column store) are pretty much MonetDBLite and DuckDB, we are willing to play around with any experimental prototypes even it has only limited functionality.

[kutsurak]

Hi @joedsilva

No it is not DuckDB. @njnes can help you with instructions about installing MonetDB Embedded. From the description though I suspect that you might encounter the same issue when you try to embed it in Postgres.

[joedsilva]

Thank you, @kutsurak and @njnes. We are willing to play around with anything at this point and will be happy to get our hands on this alternative.