Python3 support as 2.7.x series finishes its life.

[olorin37]

Python 2.7.x ends its life in August of the 2020 see. So, Python 3 support is required, but support for 2.7.x can be just dropped.

[moritz]

I've created a python3 branch. In it, pyhelper.c compiles, but initialization hangs. Help would be very welcome!

[olorin37]

As I can see configure.pl6 is not updated. There are references to Python2.

[moritz]

Good catch @olorin37, updated it to python3.

(I've so far built it with zef build . which doesn't actually use configure.pl6).

[moritz]

Current state after a bit of hacking by @niner and me:

• Some basics works
• Four test files pass completely
• There seem to be some problems with
  • callbacks
  • Buf marshalling
  • exceptions

[olorin37]

Which specific version of python3 has you used? I compiled it with 3.8.1 and seems working, but some tests are failing (log in the attachment). Maybe older versions of py3 will work better (I am checking). Looks promising.

out.log

[moritz]

that's a python 3.6.9.

[moritz]

Current State

We've made progress. (Un)marshalling of strings and byte strings/buffers works.

Passing Tests

t/call.t ................ ok     
t/date.t ................ ok   
t/eval_return_values.t .. ok   
t/import.t .............. ok   
t/invoke.t .............. ok   
t/matplotlib.t .......... ok   
t/p6_to_py.t ............ ok     
t/py_to_p6.t ............ ok    

Tests that fail with name 'perl6' is not defined

• t/call_back.t
• t/callables.t
• t/inherit.t

Other failures

• t/exceptions.t: Segmentation fault
• t/eval.t: passes when called directly, doesn't produce test output under prove

[moritz]

From my reading of some python docs, it seems that Python looks for a perl6module.o when you do an import perl6. However even after doing a gcc -c (otherflags) -o perl6module.o pyhelper.c and setting PYTHONPATH to $PWD it doesn't find the module.

[JJ]

OK, so what's the state of this now?

[moritz]

@JJ This mostly[tm] works now, with the following limitations:

• There are some segfaults involved with exceptions that I have no idea how to trace down (the test suite triggers this). @niner maybe you could have a look at this?
• The parts that need the python module installed (for things like callbacks) needs some more work. Even after repeated headbanging, I haven't managed to get the dynamic lib into a path where python picks it up. Any and all help with this would be appreciated, but I'd even release it without this feature.

[adamsmd]

I did my best to debug the python3 branch. @moritz, I'm not sure if these are the same issues you are describing and some of my fixes address only part of the problem, but my finding are below. Since Inline::Python is useful in one of my projects, I up for continuing to help debug this.

Summary

The following changes make all tests pass. Some of them are hacks, but those hacks may help diagnose the causes of the issues.

• First Issue: Cannot locate native library
  • Status: Fixed
• Second Issue: 'perl6' not defined
  • Status: Fixed
• Third Issue: Perl6Object
  • Status: Fixed
• Fourth Issue: Segmentation fault
  • Status: Partially Diagnosed with a Hacky Workaround
• Fifth Issue: planned 2 tests but ran 0
  • Status: Diagnosed with a Fix but an Open Design Question

A version of the code with all these changes implemented in the python3 branch of my fork at https://github.com/adamsmd/fork-Inline-Python/tree/python3. However, for the Fourth and Fifth Issues the code is really for demonstration purposes as they are really workarounds.

Setup

• Ubuntu 20.04
• Python 3.8.2
• Rakudo version 2020.05.1 built on MoarVM version 2020.05 implementing Raku 6.d.

First Issue: Cannot locate native library

Initially make test fails almost all tests with errors similar to the following:

Cannot locate native library '(null)': /home/adamsmd/l/tex/bib-scrape-2/Inline-Python.git.2/resources/libraries/libpyhelper.so: undefined symbol: PyMethod_Type
  in method setup at /home/adamsmd/.rakubrew/versions/moar-2020.05.1/install/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 298
  in sub py_init_python at /home/adamsmd/.rakubrew/versions/moar-2020.05.1/install/share/perl6/core/sources/947BDAB9F96E0E5FCCB383124F923A6BF6F8D76B (NativeCall) line 594
  in method BUILD at /home/adamsmd/l/tex/bib-scrape-2/Inline-Python.git.2/lib/Inline/Python.pm6 (Inline::Python) line 535
  in block <unit> at t/call.t line 8

@moritz Is this the same issue you were talking about needing the python module installed? If not, is there maybe something wrong with my setup?

The problem is that libpython3.8.so is not being loaded. We can trigger its loading by calling it once directly from Raku instead of indirectly through libpyhelper.so. Once we do it through Raku, libpyhelper has no trouble finding it, so the following fixes this:

diff --git a/lib/Inline/Python.pm6 b/lib/Inline/Python.pm6
index 0000d8d..d7db711 100644
--- a/lib/Inline/Python.pm6
+++ b/lib/Inline/Python.pm6
@@ -7,6 +7,14 @@ has &!call_method;
 
 use NativeCall;
 
+sub Py_IsInitialized()
+    is native( 'python3.8' )
+    returns int { ... }
+
+BEGIN {
+    Py_IsInitialized();
+}
+
 my constant $pyhelper = %?RESOURCES<libraries/pyhelper>;
 
 class PythonObject { ... }

Note that any libpython function would work. I used Py_IsInitialized since it is a trivial and harmless function.

Second Issue: 'perl6' not defined

With the previous fix, the following tests pass:

t/call.t ................ ok     
t/date.t ................ ok   
t/eval_return_values.t .. ok   
t/import.t .............. ok   
t/invoke.t .............. ok   
t/matplotlib.t .......... ok   
t/p6_to_py.t ............ ok     
t/py_to_p6.t ............ ok    

Of the remaining tests, several fail with the following error:

name 'perl6' is not defined
  in method call at /home/adamsmd/l/tex/bib-scrape-2/Inline-Python.git.2/lib/Inline/Python.pm6 (Inline::Python) line 379
  in block <unit> at t/callables.t line 24

My fix to the second issue is to use PyImport_AppendInittab instead of calling PyInit_perl6 directly. This is implemented by the following change:

diff --git a/lib/Inline/Python.pm6 b/lib/Inline/Python.pm6
index d7db711..9fcb74e 100644
--- a/lib/Inline/Python.pm6
+++ b/lib/Inline/Python.pm6
@@ -544,6 +544,7 @@ method BUILD {
 
     self.run(q:heredoc/PYTHON/);
         import signal
+        import perl6
         from functools import partial
         signal.signal(signal.SIGINT, signal.SIG_DFL)
         class Perl6Object:
diff --git a/pyhelper.c b/pyhelper.c
index b58e2ca..2562f4d 100644
--- a/pyhelper.c
+++ b/pyhelper.c
@@ -2,7 +2,7 @@
 #include "datetime.h"
 
 
-void PyInit_perl6(void);
+PyObject* PyInit_perl6(void);
 
 PyObject *(*call_p6_object)(int, PyObject *, PyObject **);
 PyObject *(*call_p6_method)(int, char * , PyObject *, PyObject **);
@@ -17,10 +17,9 @@ void py_init_python(PyObject *(*call_object)(int, PyObject *, PyObject **), PyOb
     call_p6_method = call_method;
 
     Py_SetProgramName(L"python");
+    PyImport_AppendInittab("perl6", &PyInit_perl6);
     Py_Initialize();
 
-    PyInit_perl6();
-
     PySys_SetArgv(_python_argc, _python_argv);  /* Tk needs this */
     PyDateTime_IMPORT;
 }
@@ -372,7 +371,7 @@ static PyMethodDef perl_functions[] = {
     {NULL,              NULL}                /* sentinel */
 };
 
-void PyInit_perl6(void){
+PyObject* PyInit_perl6(void){
     /* Create the module and add the functions */
     static struct PyModuleDef perl_module = {
         PyModuleDef_HEAD_INIT,
@@ -385,5 +384,5 @@ void PyInit_perl6(void){
         NULL, /* m_clear */
         NULL /* m_free */
     };
-    (void) PyModule_Create(&perl_module);
+    return PyModule_Create(&perl_module);
 }

Third Issue: Perl6Object

Now t/call_back.t fails with the following error:

Perl6Object
  in method call at /home/adamsmd/l/tex/bib-scrape-2/Inline-Python.git.2/lib/Inline/Python.pm6 (Inline::Python) line 379
  in block <unit> at t/call_back.t line 38

"Perl6Object" in that error seems to be a red herring, and the real problem is memory corruption due to py_array_to_array decrementing reference counts on items that were just passed to py_to_p6, which also decrements their reference counts.

The following change fixes this:

diff --git a/lib/Inline/Python.pm6 b/lib/Inline/Python.pm6
index 9fcb74e..324901d 100644
--- a/lib/Inline/Python.pm6
+++ b/lib/Inline/Python.pm6
@@ -201,7 +201,6 @@ method py_array_to_array(Pointer $py_array) {
     for 0..^$len {
         my $item = py_sequence_get_item($py_array, $_);
         @array[$_] = self.py_to_p6($item);
-        py_dec_ref($item);
     }
     return @array;
 }

Fourth Issue: Segmentation fault

At this point, only t/eval.t, t/exceptions.t, and t/inherit.t fail. The latter two also seem to be due to too many reference-count decrements, as they are fixed by the following change:

diff --git a/pyhelper.c b/pyhelper.c
index 2562f4d..13ff477 100644
--- a/pyhelper.c
+++ b/pyhelper.c
@@ -184,7 +184,6 @@ PyObject *py_none(void) {
 }
 
 void py_dec_ref(PyObject *obj) {
-    Py_DECREF(obj);
 }
 
 void py_inc_ref(PyObject *obj) {

Obviously this is not a real solution but it does indicate that future debugging should look for too many reference-count decrements.

In addition, t/exceptions.t requires the following change to the test itself:

diff --git a/t/exceptions.t b/t/exceptions.t
index 2e2ed19..0f83633 100644
--- a/t/exceptions.t
+++ b/t/exceptions.t
@@ -60,7 +60,7 @@ my $py = Inline::Python.new();
         ok 1, 'survived Python missing method';
         when X::AdHoc {
             ok $_.isa('X::AdHoc'), 'got an exception for calling missing method';
-            is $_.Str(), "Foo instance has no attribute 'non_existing'", 'exception message found for calling missing method';
+            is $_.Str(), "instance has no attribute 'non_existing'", 'exception message found for calling missing method';
         }
     }
 }
@@ -77,8 +77,8 @@ $py.run(q:heredoc/PYTHON/);
     def test_foo(foo):
         try:
             foo.depart()
-        except Exception, e:
-            return e.message
+        except Exception as e:
+            return str(e)
     PYTHON
 
 is $py.call('__main__', 'test_foo', Foo.new), 'foo';

Fifth Issue: planned 2 tests but ran 0

Now the only failing test is t/eval.t. When run from make test or prove -e 'perl6 -Ilib' t/eval.t, it says "planned 2 tests but ran 0". However, if run directly via perl6 -Ilib t/eval.t it runs both tests successfully:

$ perl6 -Ilib t/eval.t 
1..2
ok 1 - EVAL eval
ok 2 - direct eval

But if run with output redirected the tests are not printed:

$ perl6 -Ilib t/eval.t | cat
1..2

This turns out to be due to I/O flushing. By default, flushes happen after newlines if stdout points to a terminal, but they do not happen if stdout goes to a pipe.

Manually calling flush in the test (as in the following change) fixes the issue, but you might instead want to have EVAL and $py.run call that automatically. This is a design question that I'm not sure how you want to do things.

diff --git a/t/eval.t b/t/eval.t
index a0445b3..cbddc65 100644
--- a/t/eval.t
+++ b/t/eval.t
@@ -5,9 +5,9 @@ use Inline::Python;
 
 say '1..2';
 
-EVAL 'print("ok 1 - EVAL eval")', :lang<Python>, :mode<file>;
+EVAL 'print("ok 1 - EVAL eval"); import sys; sys.stdout.flush()', :lang<Python>, :mode<file>;
 
 my $py = Inline::Python.new();
-$py.run('print("ok 2 - direct eval")', :file);
+$py.run('print("ok 2 - direct eval"); import sys; sys.stdout.flush()', :file);
 
 # vim: ft=perl6

[adamsmd]

Regarding the First Issue (Cannot locate native library), it turns out that adding the --embed flag to the calls to python-config in configure.pl6 (as shown below) fixes the issue without needing the change I previously listed (i.e., the call to Py_IsInitialized directly from Raku).

diff --git a/configure.pl6 b/configure.pl6
index 7feb9b6..83887b1 100755
--- a/configure.pl6
+++ b/configure.pl6
@@ -7,8 +7,8 @@ my %vars = get-vars('.');
 my $python-config = 'python3-config';
 
 %vars<pyhelper> = 'resources'.IO.child('libraries').child($*VM.platform-library-name('pyhelper'.IO)).Str;
-%vars<cflags> = chomp qqx/$python-config --cflags/;
-%vars<ldflags> = chomp qqx/$python-config --ldflags/;
+%vars<cflags> = chomp qqx/$python-config --cflags --embed/;
+%vars<ldflags> = chomp qqx/$python-config --ldflags --embed/;
 
 mkdir 'resources' unless 'resources'.IO.e;
 mkdir 'resources/libraries' unless 'resources/libraries'.IO.e;

[niner]

On Freitag, 24. Juli 2020 18:44:34 CEST Michael D. Adams wrote: Regarding the First Issue (Cannot locate native library), it turns out that adding the `--embed` flag to the calls to `python-config` in `configure.pl6` (as shown below) fixes the issue

Excellent find! Pushed that fix

[niner]

My fix to the second issue is to use `PyImport_AppendInittab` instead of calling `PyInit_perl6` directly.

Makes perfect sense once one knows about that function. Fix pushed

[niner]

"Perl6Object" in that error seems to be a red herring, and the real problem is memory corruption due to `py_array_to_array` decrementing reference counts on items that were just passed to `py_to_p6`, which also decrements their reference counts.

I think py_array_to_array is innocent and decrementing the reference count is actually correct. I can't find anything in Python's ChangeLog about PySequence_GetItem and the docs still say: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i) Return value: New reference. Since GetItem returns a new reference, we have to free it after converting the item to Raku. `py_to_p6` already increases the reference count for values that just get wrapped (i.e. objects). So it looks more like we're missing a py_inc_ref somewhere else.