Skip to content
This repository has been archived by the owner on Aug 10, 2022. It is now read-only.

python3.PyImport_ImportModule(name) will emit a fatal error when called the second time. #29

Open
Zalberth opened this issue Jun 29, 2020 · 5 comments
Open

python3.PyImport_ImportModule(name) will emit a fatal error when called the second time. #29

Zalberth opened this issue Jun 29, 2020 · 5 comments

Comments

@Zalberth
Copy link

Describe what happened:
Environments:
On MacOS (Catalina Version 10.15.4)
Python3.7.6
Go1.13.8
I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/go-python3@v0.0.0-20191126174558-6ed25e33b3c4/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

Here is the sample code.
A wrapper of PyImport_ImportModule:

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
	fmt.Println("python3.PyImport_ImportModule before")
	sysModule := python3.PyImport_ImportModule("sys") // import sys
	fmt.Println("python3.PyImport_ImportModule success")
	path := sysModule.GetAttrString("path")           // path = sys.path
	ob := python3.PyList_GetItem(path, 1)
	fmt.Println("check:", python3.PyUnicode_Check(ob))
	fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
	fmt.Println("sysModule.GetAttrString success")
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
	fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
	fmt.Println("module name:", name)
	return python3.PyImport_ImportModule(name) 
}

Each time the algorithm is called in a goroutine.

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
	gstate = python3.PyGILState_Ensure()
	fmt.Println("Py_Initialize success")
	vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
	fmt.Println("ImportModule success")

	b := vbp.GetAttrString("estimate")
	fmt.Printf("[FUNC] b = %#v\n", b)
	bArgs := python3.PyTuple_New(1)
	python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
	re := b.Call(bArgs, python3.Py_None)
	sbp := python3.PyTuple_GetItem(re, 0)
	dbp := python3.PyTuple_GetItem(re, 1)
	defer func() {
		python3.Py_Finalize()
		fmt.Println("python3.Py_Finalize()")
	}()
	sbpI = python3.PyLong_AsLong(sbp)
	dbpI = python3.PyLong_AsLong(dbp)
	python3.PyGILState_Release(gstate)
	return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

Sample caller like this: go Calc("aaa.csv").

Describe what you expected:
We can call go Calc() many times without a fatal error.

Steps to reproduce the issue:
Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.
@Zalberth
Copy link
Author

Every thing is fine when go Calc("aaa.csv") called the first time. But the fatal error emits the second time when go Calc("aaa.csv") is called.

@Zalberth Zalberth changed the title python3.PyImport_ImportModule(name) will emmit an fatal error when called the second time. python3.PyImport_ImportModule(name) will emit a fatal error when called the second time. Jun 29, 2020
@christian-korneck
Copy link
Contributor

I haven't looked at your code in detail, but one thing I've noticed is the absence of any <PyObject>.DecRef() calls. Python objects referenced in Go should get decref'ed when they're no longer needed (i.e. bArgs.DecRef()).

@chinajuanbob
Copy link

It seems we could only PyImport_ImportModule some module for only one time. And even after that, calling functions in multiple goroutines still hit similar problems.

@javyxu
Copy link

javyxu commented Nov 6, 2020

Describe what happened:
Environments:
On MacOS (Catalina Version 10.15.4)
Python3.7.6
Go1.13.8
I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]

runtime stack:
runtime.throw(0x4967a75, 0x2a)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:378 +0x47c

goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
        _cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
        /Users/zhao/go/pkg/mod/github.com/!data!dog/go-python3@v0.0.0-20191126174558-6ed25e33b3c4/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
        /Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....

Here is the sample code.
A wrapper of PyImport_ImportModule:

// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
	fmt.Println("python3.PyImport_ImportModule before")
	sysModule := python3.PyImport_ImportModule("sys") // import sys
	fmt.Println("python3.PyImport_ImportModule success")
	path := sysModule.GetAttrString("path")           // path = sys.path
	ob := python3.PyList_GetItem(path, 1)
	fmt.Println("check:", python3.PyUnicode_Check(ob))
	fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
	fmt.Println("sysModule.GetAttrString success")
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
	python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
	fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
	fmt.Println("module name:", name)
	return python3.PyImport_ImportModule(name) 
}

Each time the algorithm is called in a goroutine.

func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
	python3.Py_Initialize()
	if !python3.Py_IsInitialized() {
		fmt.Println("Error initializing the python interpreter")
		os.Exit(1)
	}
	gstate = python3.PyGILState_Ensure()
	fmt.Println("Py_Initialize success")
	vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
	fmt.Println("ImportModule success")

	b := vbp.GetAttrString("estimate")
	fmt.Printf("[FUNC] b = %#v\n", b)
	bArgs := python3.PyTuple_New(1)
	python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
	re := b.Call(bArgs, python3.Py_None)
	sbp := python3.PyTuple_GetItem(re, 0)
	dbp := python3.PyTuple_GetItem(re, 1)
	defer func() {
		python3.Py_Finalize()
		fmt.Println("python3.Py_Finalize()")
	}()
	sbpI = python3.PyLong_AsLong(sbp)
	dbpI = python3.PyLong_AsLong(dbp)
	python3.PyGILState_Release(gstate)
	return
}

func Calc(dataFilePath string) {
    CalcPre(dataFilePath)
}

Sample caller like this: go Calc("aaa.csv").

Describe what you expected:
We can call go Calc() many times without a fatal error.

Steps to reproduce the issue:
Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.

Hello, do you have a solution to this problem?

@christian-korneck
Copy link
Contributor

maybe as a starting point:

  • here's an example for creating a Go function around a Python function that you can loop over (= it works a second time): code / explanation
  • if you add locking and releasing the GIL and locking the thread to it you should be able to use it in a go routine. Here's a example (it uses the sbinet/go-python instead of datadog/go-python3 but you get the idea). code / explanation

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@chinajuanbob @Zalberth @christian-korneck @javyxu