Segmentation fault on database import

[Pan6ora]

I found this while working on a plugin system for AB. One of my test plugin was importing databases from bw2package files. While importing the databases, AB sometimes crashed with a segmentation fault error.

After some testing I found that this appends also when importing other databases such as Forwast or default Biosphere data.

The bug is not consistent between computers, and even on the same computer.

How to reproduce

• create a new project
• import default data
• add Forwast database
• repeat until AB crash with Segmentation fault (core dumped) (on my computer it appends usually between first and third try)

What I tried

• While searching on the internet I only found similar issues with Qt which make me think it may be a problem with the gui
  • I wasn't able to reproduce using only Brightway
  • I tried to upgrade PySide2 and Qt packages
• using GDB to trace the memory leak

Any idea ?

[Zoophobus]

Hi Pan6ora,
Thanks for the report! I wasn't looking at QT before this so that's been a big help.

So, since your post I have been looking at QT and memory leaks/pointer issues and have found some issues on other programs that point to issues with QThreads. I suppose this could be the same kind of problem here but the trouble I have is that this has never previously been an issue, which to me suggests something else. So if you don't mind could you provide some more details?

1. One thing I'm curious about, if you've been a long term AB user is if you had these problems before? If not then could you give some details on the versions, where this wasn't a problem?
2. You mentioned upgrading the PySide2 and Qt packages, if you upgraded them could you tell me from and to what?
3. Your output from gdb doesn't suggest to me an association with threads, could you confirm, or deny this?
4. I'm testing this problem by importing excel databases, this uses the same procedure with multiple QThreads executing the process, while I expect you to get similar results I would appreciate confirmation!

[Pan6ora]

Hi Zoophobus,
Thank you for your answer.

Were you able to reproduce ? In my team we had one computer on which the problem never appends.
Also about my config I'm running Arch Linux on a 16Go Ram Thinkpad X230.

About your questions:

1. No only since 6 months so I have no good answer to this question.
2. As I saw someone mentioning this could be a compilation error on a similar problem, I tried to update all dependencies from my conda env to latest version. I remember upgrading from PySide2 5.15.4 to 5.15.6 but this changed nothing.
3. I don't really know as python threading is not something I'm familiar with. But I see nothing that suggest it to me neither.
4. Maybe you could send me the database you're using so that I could test with it ?

[Pan6ora]

Some news !

After some testing I have found that the problem doesn't appear when the import is run from the main window code.
It looks like the segfault is due to a memory limitation of the wizard.

Doesn't solve the problem yet but should help.

[haasad]

Hey @Pan6ora and @Zoophobus,

I have been trying to track down this weird bug for quite some time now, basically since like 9 months ago when I first noticed that the automated tests for python 3.10 fail on all operating systems with a segfault: https://github.com/LCA-ActivityBrowser/activity-browser/actions?query=branch%3Atest-py310++

One consistent way to reproduce it that I found is creating a new project and adding the default biosphere data when using python 3.10. However as far as I can tell this is not limited to python 3.10, but just less frequent under other circumstances like @Pan6ora describes above.

What I found so far:

• It's definitely related to threading, i.e. the problem doesn't appear if the the import tasks aren't run in a separate QThread
• This is the most useful error message I got so far (usually it's just segmentation fault (core dumped)):

Fatal Python error: double free or corruption (out)PyThreadState_Get
: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized

Current thread 0x00007f8d8bb6e740 (most recent call first):
  File "/home/adrian/miniconda3/envs/abpytest/lib/python3.10/site-packages/peewee.py", line 2190 in _execute
  File "/home/adrian/miniconda3/envs/abpytest/lib/python3.10/site-packages/peewee.py[1]    485499 abort (core dumped)  python run-activity-browser.py

To me it looks like the problem (sometimes) happens when changing the sqlite database from a separate QThread, but I can't pinpoint exactly what the root cause is. Unfortunately writing to the sqlite database takes some time, so moving all the tasks to the main thread would be pretty ugly and freeze the GUI until it's finished. That's why these operations were moved to a separate worker thread in the first place.

[marc-vdm]

Closing as resolved in #1011