Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

travis-ci: properly export PYTHON_VERSION and fix python 3 fallout #2124

Merged
merged 13 commits into from May 6, 2019

Conversation

@grondo
Copy link
Contributor

commented Apr 15, 2019

Be sure to export PYTHON_VERSION to the docker environment as discussed in #2123.

I haven't yet tried to fix any of the potential fallout from adding python 3.x checks under travis yet, so the checks may fail here. Also, there is the potential I was just confused and this will be a no-op.

@garlick

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

Not a no-op, good catch!

Looks like the Ubuntu / Python 3.6 builder is failing here, which probably accounts for the subsequent failures in tests that need jobspec:

 File "/usr/src/flux-core-0.11.0-556-g624deb6a/_build/sub/../../src/cmd/flux-jobspec", line 13, in <module>
    import yaml
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 284
    class YAMLObject(metaclass=YAMLObjectMetaclass):
                              ^
SyntaxError: invalid syntax
not ok 1 - sched-simple: generate jobspec for simple test job
@garlick

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

The centos7 py3.4 builder also fails, but it seems to get through most everything and lose it on one of the python test helpers I added to test the job-manager here:

expecting success: 
	${DRAIN_UNDRAIN} >drain_undrain.out &&
	grep re-enabled drain_undrain.out
Traceback (most recent call last):
  File "/usr/src/t/job-manager/drain-undrain.py", line 26, in <module>
    h = flux.Flux()
  File "/usr/src/src/bindings/python/flux/__init__.py", line 17, in Flux
    import flux.core.handle
  File "/usr/src/src/bindings/python/flux/core/handle.py", line 14, in <module>
    from flux.rpc import RPC
  File "/usr/src/src/bindings/python/flux/rpc.py", line 14, in <module>
    from flux.future import Future
  File "/usr/src/src/bindings/python/flux/future.py", line 13, in <module>
    from flux.util import check_future_error
  File "/usr/src/src/bindings/python/flux/util.py", line 17, in <module>
    from flux.core.inner import ffi, raw
  File "/usr/src/src/bindings/python/flux/core/inner.py", line 11, in <module>
    from _flux._core import ffi, lib
ImportError: dynamic module does not define init function (init_core)
not ok 45 - job-manager: flux job drain canceled by undrain

(This one's probably on me - will have a look)

@grondo grondo force-pushed the grondo:issue#2123 branch 2 times, most recently from b9cd5c5 to 8683464 May 1, 2019
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Rebased this branch on current master.

Also included some experimental fixes for fallout from the initial change, i.e. the flux-python wrapper script and a fix for getting the correct user site.config path in the sharness tests.

As discussed in #2125, these fixes lead to other errors in the testsuite, including:

  • segfault in t/job-manager/drain-undrain.py and drain-cancel.py during t2202-job-manager.t
  • t0021-flux-jobspec.t:
    • not ok 12 - an improperly formatted time string should produce an error
    • not ok 13 - requesting more nodes than tasks should produce an error
  • not ok 6 __main__.TestWrapper.test_null_handle_exception:
FAIL: python/t0002-wrapper.py 6 __main__.TestWrapper.test_null_handle_exception
 ---
   message: |
     Traceback (most recent call last):
       File "./python/t0002-wrapper.py", line 50, in test_null_handle_exception
         cm.exception.message,
     AttributeError: 'ValueError' object has no attribute 'message'
  ...
  • t2206-job-manager-bulk-state.t: bulk-state.py fails with:
Traceback (most recent call last):
 File "/usr/src/t/job-manager/bulk-state.py", line 83, in <module>
   if cmp(jobs[jobid], expected_states) != 0:
NameError: name 'cmp' is not defined

(I guess cmp was removed from Python 3)

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

(I guess cmp was removed from Python 3)

This fix was straightforward, so I went ahead and committed it here.

@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 2, 2019

For the t0002-wrapper.py error, the following should fix that (along with some other py2/py3 incompatibilities....not sure why they were working in py3 at all).

diff --git a/t/python/t0002-wrapper.py b/t/python/t0002-wrapper.py
index e232cfea8..20e1d3f07 100755
--- a/t/python/t0002-wrapper.py
+++ b/t/python/t0002-wrapper.py
@@ -12,6 +12,7 @@
 
 import unittest
 
+import six
 import flux
 from flux.core.inner import ffi, raw
 import flux.wrapper
@@ -44,18 +45,14 @@ class TestWrapper(unittest.TestCase):
         future = f.rpc("cmb.ping", payload)
         resp = future.get()
         future.pimpl.handle = None
-        with self.assertRaises(ValueError) as cm:
+        with six.assertRaisesRegex(self, ValueError, r"Attempting to call a cached, bound method.*NULL handle"):
             resp = future.get()
-        self.assertRegexpMatches(
-            cm.exception.message,
-            "Attempting to call a cached, " "bound method.*NULL handle",
-        )
 
     def test_automatic_unwrapping(self):
         flux.core.inner.raw.flux_log(flux.Flux("loop://"), 0, "stuff")
 
     def test_masked_function(self):
-        with self.assertRaisesRegexp(AttributeError, r".*masks function.*"):
+        with six.assertRaisesRegex(self, AttributeError, r".*masks function.*"):
             flux.Flux("loop://").rpc("topic").pimpl.flux_request_encode("request", 15)
 
     def test_set_pimpl_handle(self):
@@ -68,7 +65,7 @@ class TestWrapper(unittest.TestCase):
     def test_set_pimpl_handle_invalid(self):
         f = flux.Flux("loop://")
         r = f.rpc("topic")
-        with self.assertRaisesRegexp(TypeError, r".*expected a.*"):
+        with six.assertRaisesRegex(self, TypeError, r".*expected a.*"):
             r.handle = f.rpc("other topic")
 
     def test_read_basic_value(self):

Docs: https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaisesRegex

EDIT: there was a mistake in the original diff (six.assertRaisesRegexp should actually be six.assertRaisesRegex). This diff works now.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Thanks, @SteVwonder! For the flux-jobspec errors, the exception handling seems to have changed.

Strangely, with python3, the following exits with zero exit code and no output:

$ src/cmd/flux jobspec srun -t foo hostname
$ echo $?
0

even though with some debug, the "except" path is taken:

grondo@7f4047c1d037:/usr/src$ git diff
diff --git a/src/cmd/flux-jobspec b/src/cmd/flux-jobspec
index 206d1427..0ba92f97 100755
--- a/src/cmd/flux-jobspec
+++ b/src/cmd/flux-jobspec
@@ -126,6 +126,7 @@ def slurm_jobspec(args):
     try:
         validate_slurm_args(args)
     except ValueError as e:
+        print ("got ValueError\n")
         logger.error(e.message)
         sys.exit(1)
     t = slurm_walltime_to_duration(args.time)
grondo@7f4047c1d037:/usr/src$ src/cmd/flux jobspec srun -t foo hostname
got ValueError

If I comment out the logger.error(e.message), then we reach the sys.exit(1) at least:

grondo@7f4047c1d037:/usr/src$ git diff
diff --git a/src/cmd/flux-jobspec b/src/cmd/flux-jobspec
index 206d1427..72a1e6ed 100755
--- a/src/cmd/flux-jobspec
+++ b/src/cmd/flux-jobspec
@@ -126,7 +126,8 @@ def slurm_jobspec(args):
     try:
         validate_slurm_args(args)
     except ValueError as e:
-        logger.error(e.message)
+        print ("got ValueError\n")
+        #logger.error(e.message)
         sys.exit(1)
     t = slurm_walltime_to_duration(args.time)
     return create_slurm_style_jobspec(
grondo@7f4047c1d037:/usr/src$ src/cmd/flux jobspec srun -t foo hostname
got ValueError

grondo@7f4047c1d037:/usr/src$ echo $?
1

Something about trying to access e.message is causing the python interpreter to exit with 0 status and no message!? That seems very...wrong.

@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Something about trying to access e.message is causing the python interpreter to exit with 0 status and no message!? That seems very...wrong.

😱 Agreed! Apparently the .message attribute was deprecated in py2.6 and removed in py3.0+. It appears that the preferred/portable syntax is str(e). [1][2]

So that explains why things are breaking on usage of .message. As to why that causes an exit code of 0....uh, lemme dig into that some more 😆

@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Something about trying to access e.message is causing the python interpreter to exit with 0 status and no message!? That seems very...wrong.

The problem was with the try except block in the main function. It was also trying to access .message, which causing an exception, moving exceution to the finally block, which exits with exit_code. Since exit_code was being set to 1 after the .message access, the exit_code never changed from its default value of 0. This patch should fix it:

diff --git a/src/cmd/flux-jobspec b/src/cmd/flux-jobspec
index 206d14270..2c6760b3c 100755
--- a/src/cmd/flux-jobspec
+++ b/src/cmd/flux-jobspec
@@ -126,7 +126,7 @@ def slurm_jobspec(args):
     try:
         validate_slurm_args(args)
     except ValueError as e:
-        logger.error(e.message)
+        logger.error(str(e))
         sys.exit(1)
     t = slurm_walltime_to_duration(args.time)
     return create_slurm_style_jobspec(
@@ -192,8 +192,8 @@ if __name__ == "__main__":
     except SystemExit as e:  # don't intercept sys.exit calls
         exit_code = e
     except Exception as e:
-        logging.error(e.message)
         exit_code = 1
+        logging.error(str(e))
     finally:
         logging.shutdown()
         sys.exit(exit_code)

One of the flux-jobspec tests still fails due to the deprecationwarning that is being printed for our use of ABC in collections, which will not work in python 3.8+. This patch will fix that:

diff --git a/src/cmd/flux-jobspec b/src/cmd/flux-jobspec                
index 206d14270..b5f324439 100755           
--- a/src/cmd/flux-jobspec
+++ b/src/cmd/flux-jobspec           
@@ -8,13 +8,18 @@ import math
 import logging                                                                                
 import argparse
 import json
-from collections import Sequence
+import collections
+
+try:
+    collectionsAbc = collections.abc
+except AttributeError:
+    collectionsAbc = collections

 import yaml


 def create_resource(res_type, count, with_child=[]):
-    assert isinstance(with_child, Sequence), "child resource must be a sequence"
+    assert isinstance(with_child, collectionsAbc.Sequence), "child resource must be a sequence"
     assert not isinstance(with_child, str), "child resource must not be a string"
     assert count > 0, "resource count must be > 0"

Relevant StackOverflow post.

I realize the irony of commenting these patches to you after making fun of the linux kernel for doing the exact same thing over email

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

want to collect all your fixes into a branch and then I can cherry pick them here?

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

Also, I think there are some other uses of e.message extant in our code. We should fix those as well.

@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 2, 2019

want to collect all your fixes into a branch and then I can cherry pick them here?

Also, I think there are some other uses of e.message extant in our code. We should fix those as well.

👍 Here it is. You'll probably want to squash the first commit which is just running the black formatter on this PR as it is. https://github.com/SteVwonder/flux-core/tree/issue2123

For some reason, it works locally (with a simple make check) but the flux-python shebang is not working in travis for the PYTHON=3.6 distcheck builder. I'm guessing distcheck is the issue there?

EDIT: still investigating the potential double free in the drain/undrain

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

I'm guessing distcheck is the issue there?

Ah, yeah. I doubt $top_srcdir/src/cmd is in the default PATH for the sharness tests.

Now that flux-python is just for test scripts maybe we can do something better than putting it in src/cmd... I'll have to think about the least messy way to do this.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

I'll have to think about the least messy way to do this.

What if, for testing, we just put a link to the configured python in ${top_builddir}/src/cmd/python, which will be first in the PATH during make check since that is where we find the flux command itself.

Then for test scripts we can still use #!/usr/bin/env python. For installed python scripts, we would still use the approach of appending .py and teaching flux(1) to exec these scripts under the configured python.

The drawback is that all python scripts (run during testing) would use the configured python but perhaps that is ok.

@grondo grondo force-pushed the grondo:issue#2123 branch from db1805a to ae9ef4a May 3, 2019
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Ok, I've just pushed a revised branch (with @SteVwonder's fixes so far, thanks!) which uses the approach above. A nodist_SCRIPTS python wrapper script is created in src/cmd/python which is used during make check to ensure that the python used is the one used by configure.

The flux command itself also now checks for flux-<cmd>.py when searching for flux cmd, and if it exists and is executable, runs flux-cmd.py under the configured python interpreter. Finally, flux-jobspec is moved to flux-jobspec.py.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

In case this is helpful, the segfaults in python from t2202-job-manager.t go away if I remove the Future class destructor (obviously not the right fix, but I don't really understand the python bindings yet so just experimenting)

```index d4d94e95d..4dc471064 100644                                              
--- a/src/bindings/python/flux/future.py                                       
+++ b/src/bindings/python/flux/future.py                                       
@@ -42,7 +42,7 @@ class Future(WrapperPimpl):
             match=ffi.typeof("flux_future_t *"),
             filter_match=True,
             prefixes=None,
-            destructor=raw.flux_future_destroy,
+            #destructor=raw.flux_future_destroy,
         ):
             # avoid using a static list as a default argument
             # pylint error 'dangerous-default-value'
@@ -50,7 +50,7 @@ class Future(WrapperPimpl):
                 prefixes = ["flux_future_"]

             super(Future.InnerWrapper, self).__init__(
-                ffi, lib, handle, match, filter_match, prefixes, destructor

+                ffi, lib, handle, match, filter_match, prefixes,
             )

         def check_wrap(self, fun, name):

The final error is from pymod. The module causes a segfault at module remove:

grondo@986cb729824d:/usr/src$ flux module load pymod --verbose --path=$(pwd)/src
/modules/pymod echo                                                            
grondo@986cb729824d:/usr/src$ which python                                     
/usr/src/src/cmd/python                                                        
grondo@986cb729824d:/usr/src$ python -c 'import flux; print(flux.Flux().rpc ("echo.foo", {"data": "foo"}).get())'                                              
{'data': 'foo'}                                                                
grondo@986cb729824d:/usr/src$ flux module remove pymod                         
flux-module: cmb.rmmod[-1] pymod: Protocol error                               
Segmentation fault (core dumped)                 

Unfortunately, I'm not getting anything out of the corefile. I tried to run the broker under valgrind and reproduce, but once python is involved, there are many thousands of errors from valgrind.

Here's the subset of valgrind output after the module is removed:

grondo@986cb729824d:/usr/src$ flux module remove pymod
==18941== Invalid read of size 4
==18941==    at 0x4E545D0: flux_flags_get (handle.c:414)
==18941==    by 0x4E56DAF: dispatch_usecount_decr.part.5 (msg_handler.c:79)
==18941==    by 0x4E5742C: dispatch_usecount_decr (msg_handler.c:436)
==18941==    by 0x4E5742C: flux_msg_handler_destroy (msg_handler.c:432)
==18941==    by 0x164488B6: _cffi_f_flux_msg_handler_destroy (_core.c:14843)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926950: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA142D7: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB99FEEC: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Address 0xd3f70b4 is 20 bytes inside a block of size 128 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54468: flux_close (handle.c:315)
==18941==    by 0x16463C86: _cffi_f_flux_close (_core.c:4291)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926950: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA142D7: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB99FEEC: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54491: flux_handle_create (handle.c:321)
==18941==    by 0xDBAB000: connector_init (shmem.c:198)
==18941==    by 0x4E54773: flux_open (handle.c:280)
==18941==    by 0x11411B: module_thread (module.c:122)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Conditional jump or move depends on uninitialised value(s)
==18941==    at 0xB9B14CF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A62DD: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CD924: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A3614: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C55EA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CD9B3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CC6F7: PyDict_SetItemString (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908B3B: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==
==18941== Use of uninitialised value of size 8
==18941==    at 0xB9B14E3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A62DD: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CD924: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A3614: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C55EA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CD9B3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CC6F7: PyDict_SetItemString (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908B3B: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908DA7: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0x8355020 is 1,488 bytes inside a block of size 3,240 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9E4E9B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB904CB9: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BE9EA: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926D1D: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92BE64: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926EEA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31D2F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9E2BAA: PyList_Append (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB90384B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB902FD0: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB903BEE: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB904C97: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BE9EA: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926D1D: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92BE64: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C55EA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CD9B3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BCD76: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9E5B46: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8FCA89: _PyState_ClearModules (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908DF0: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd654020 is 16 bytes after a block of size 576 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9C7589: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C8FE2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB929645: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926A42: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13C77: PyObject_CallFunctionObjArgs (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92C0ED: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6F1E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C7262: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C8FE2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9080AE: _PyImport_FixupExtensionObject (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9088C3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926D1D: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==

hon3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd4cc020 is 1,440 bytes inside a block of size 1,608 free'd
==18941==    at 0x4C31D2F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9E2BAA: PyList_Append (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB90384B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB902FD0: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB903BEE: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB904C97: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BE9EA: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926D1D: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92BE64: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31D2F: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9E2BAA: PyList_Append (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB90384B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB902FD0: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB903BEE: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB902FD0: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB903BD8: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB904C97: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BE9EA: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Conditional jump or move depends on uninitialised value(s)
==18941==    at 0xB9B14CF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6DA8: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFEBF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908DF5: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Use of uninitialised value of size 8
==18941==    at 0xB9B14E3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6DA8: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFEBF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908DF5: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BD234: _PyModule_ClearDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908E8C: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd424020 is 416 bytes inside a block of size 576 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9CD9B3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9AB223: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A7664: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9321A9: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEA39: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9270FD: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB927FDA: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6F1E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C7262: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CE320: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9CE60C: PyDict_Copy (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9AAE96: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A7664: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9321A9: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEA39: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Conditional jump or move depends on uninitialised value(s)
==18941==    at 0xB9B14CF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9E7A86: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BD234: _PyModule_ClearDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908E8C: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Use of uninitialised value of size 8
==18941==    at 0xB9B14E3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9E7A86: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BD234: _PyModule_ClearDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908E8C: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BCD4E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BCB96: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C9166: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BD234: _PyModule_ClearDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908E8C: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd633020 is 16 bytes before a block of size 928 alloc'd
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E057D: _PyObject_GC_Malloc (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A6139: PyType_GenericAlloc (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9AA7A1: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A7664: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9321A9: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEA39: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A4B65: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFEBF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908F05: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd41c020 is 64 bytes inside a block of size 576 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9CD9B3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9320DB: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEA39: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9270FD: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB927FDA: PyEval_EvalCode (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB907418: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB90764E: PyImport_ImportFrozenModuleObject (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB907819: PyImport_ImportFrozenModule (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6F1E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C7262: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C8FE2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D89A: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9270FD: PyEval_EvalCodeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB93216A: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEA39: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A4B54: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFEBF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908F05: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd924020 is 48 bytes inside a block of size 96 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x52D8622: zframe_destroy (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x52E4177: zmsg_destroy (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x4E579AA: flux_msg_destroy (message.c:267)
==18941==    by 0x11489E: module_sendmsg (module.c:282)
==18941==    by 0x11789C: service_send (service.c:221)
==18941==    by 0x1133C0: broker_request_sendmsg (broker.c:1804)
==18941==    by 0x113A61: module_cb (broker.c:1649)
==18941==    by 0x4E77EE0: check_cb (ev_zmq.c:70)
==18941==    by 0x4E885A3: ev_invoke_pending (ev.c:3322)
==18941==    by 0x4E894AB: ev_run (ev.c:3726)
==18941==    by 0x4E55DE2: flux_reactor_run (reactor.c:126)
==18941==  Block was alloc'd at
==18941==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x52D869A: zframe_new (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x52E4DC6: zmsg_pushstr (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x4E58AEC: flux_msg_push_route (message.c:753)
==18941==    by 0x114929: module_sendmsg (module.c:260)
==18941==    by 0x11789C: service_send (service.c:221)
==18941==    by 0x1133C0: broker_request_sendmsg (broker.c:1804)
==18941==    by 0x113A61: module_cb (broker.c:1649)
==18941==    by 0x4E77EE0: check_cb (ev_zmq.c:70)
==18941==    by 0x4E885A3: ev_invoke_pending (ev.c:3322)
==18941==    by 0x4E894AB: ev_run (ev.c:3726)
==18941==    by 0x4E55DE2: flux_reactor_run (reactor.c:126)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A4F55: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFED2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908F05: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0x8257020 is 464 bytes inside a block of size 552 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9A384C: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9FDF56: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9E4F8B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254DE: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926EEA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926EEA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926EEA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E057D: _PyObject_GC_Malloc (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E06B9: _PyObject_GC_NewVar (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A8457: PyTuple_New (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB902FA7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB903BD8: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB904C97: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BE9EA: _PyCFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926E0B: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Conditional jump or move depends on uninitialised value(s)
==18941==    at 0xB9B14CF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C55EA: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9A4B65: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8DFEBF: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB8E0890: _PyGC_CollectNoFail (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB908F05: PyImport_Cleanup (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909127: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB877888: _PyFaulthandler_Fini (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB909145: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd654020 is 16 bytes after a block of size 576 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9C7589: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C8FE2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB929645: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926A42: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13C77: PyObject_CallFunctionObjArgs (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92C0ED: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9254A2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0xB9B54F7: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C6F1E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C7262: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9C8FE2: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9080AE: _PyImport_FixupExtensionObject (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9088C3: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926D1D: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92D091: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==
==18941== Invalid read of size 4
==18941==    at 0xB9B14C6: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA0596D: PyGrammar_RemoveAccelerators (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB9091E2: Py_FinalizeEx (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB574564: mod_main (py_mod.c:196)
==18941==    by 0x1141EC: module_thread (module.c:144)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0x81f9020 is 16 bytes before a block of size 56 alloc'd
==18941==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x400C3E7: _dl_new_object (dl-object.c:163)
==18941==    by 0x40069A4: _dl_map_object_from_fd (dl-load.c:943)
==18941==    by 0x4008FFB: _dl_map_object (dl-load.c:2389)
==18941==    by 0x4014EE3: dl_open_worker (dl-open.c:235)
==18941==    by 0x5F972DE: _dl_catch_exception (dl-error-skeleton.c:196)
==18941==    by 0x40147C9: _dl_open (dl-open.c:605)
==18941==    by 0x59DBF95: dlopen_doit (dlopen.c:66)
==18941==    by 0x5F972DE: _dl_catch_exception (dl-error-skeleton.c:196)
==18941==    by 0x5F9736E: _dl_catch_error (dl-error-skeleton.c:215)
==18941==    by 0x59DC734: _dlerror_run (dlerror.c:162)
==18941==    by 0x59DC050: dlopen@@GLIBC_2.2.5 (dlopen.c:87)
==18941==
==18941== Invalid read of size 4
==18941==    at 0x4E54ED5: lookup_clone_ancestor (handle.c:83)
==18941==    by 0x4E54ED5: flux_recv (handle.c:666)
==18941==    by 0x11422F: module_thread (module.c:153)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd3f70b4 is 20 bytes inside a block of size 128 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54468: flux_close (handle.c:315)
==18941==    by 0x16463C86: _cffi_f_flux_close (_core.c:4291)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926950: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA142D7: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB99FEEC: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54491: flux_handle_create (handle.c:321)
==18941==    by 0xDBAB000: connector_init (shmem.c:198)
==18941==    by 0x4E54773: flux_open (handle.c:280)
==18941==    by 0x11411B: module_thread (module.c:122)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 8
==18941==    at 0x4E54F56: flux_recv_any (handle.c:650)
==18941==    by 0x4E54F56: flux_recv (handle.c:673)
==18941==    by 0x11422F: module_thread (module.c:153)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd3f70d0 is 48 bytes inside a block of size 128 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54468: flux_close (handle.c:315)
==18941==    by 0x16463C86: _cffi_f_flux_close (_core.c:4291)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926950: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA142D7: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB99FEEC: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13A98: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E54491: flux_handle_create (handle.c:321)
==18941==    by 0xDBAB000: connector_init (shmem.c:198)
==18941==    by 0x4E54773: flux_open (handle.c:280)
==18941==    by 0x11411B: module_thread (module.c:122)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 8
==18941==    at 0x4E7A5E4: msglist_count (msglist.c:159)
==18941==    by 0x4E54F5E: flux_recv_any (handle.c:650)
==18941==    by 0x4E54F5E: flux_recv (handle.c:673)
==18941==    by 0x11422F: module_thread (module.c:153)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0xd3f7230 is 0 bytes inside a block of size 32 free'd
==18941==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E5438D: flux_handle_destroy (handle.c:389)
==18941==    by 0x4E54468: flux_close (handle.c:315)
==18941==    by 0x16463C86: _cffi_f_flux_close (_core.c:4291)
==18941==    by 0xB9BEC53: PyCFunction_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92FB0F: _PyEval_EvalFrameDefault (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB92663E: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB926950: _PyFunction_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA13B40: _PyObject_FastCallDict (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA1426C: _PyObject_Call_Prepend (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xBA142D7: PyObject_Call (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==    by 0xB99FEEC: ??? (in /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0)
==18941==  Block was alloc'd at
==18941==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==18941==    by 0x4E7A407: msglist_create (msglist.c:73)
==18941==    by 0x4E544D8: flux_handle_create (handle.c:332)
==18941==    by 0xDBAB000: connector_init (shmem.c:198)
==18941==    by 0x4E54773: flux_open (handle.c:280)
==18941==    by 0x11411B: module_thread (module.c:122)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==
==18941== Invalid read of size 8
==18941==    at 0x52DFC10: zlist_size (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x4E7A5EB: msglist_count (msglist.c:159)
==18941==    by 0x4E54F5E: flux_recv_any (handle.c:650)
==18941==    by 0x4E54F5E: flux_recv (handle.c:673)
==18941==    by 0x11422F: module_thread (module.c:153)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  Address 0x18 is not stack'd, malloc'd or (recently) free'd
==18941==
==18941==
==18941== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==18941==  Access not within mapped region at address 0x18
==18941==    at 0x52DFC10: zlist_size (in /usr/lib/x86_64-linux-gnu/libczmq.so.4.1.0)
==18941==    by 0x4E7A5EB: msglist_count (msglist.c:159)
==18941==    by 0x4E54F5E: flux_recv_any (handle.c:650)
==18941==    by 0x4E54F5E: flux_recv (handle.c:673)
==18941==    by 0x11422F: module_thread (module.c:153)
==18941==    by 0x57C36DA: start_thread (pthread_create.c:463)
==18941==    by 0x5F5188E: clone (clone.S:95)
==18941==  If you believe this happened as a result of a stack
==18941==  overflow in your program's main thread (unlikely but
==18941==  possible), you can try to increase the size of the
==18941==  main thread stack using the --main-stacksize= flag.
==18941==  The main thread stack size used in this run was 8388608.
==18941==
==18941== HEAP SUMMARY:
==18941==     in use at exit: 2,819,598 bytes in 3,724 blocks
==18941==   total heap usage: 45,094 allocs, 41,370 frees, 40,813,094 bytes allocated
==18941==
==18941== LEAK SUMMARY:
==18941==    definitely lost: 324 bytes in 10 blocks
==18941==    indirectly lost: 4,290 bytes in 7 blocks
==18941==      possibly lost: 17,628 bytes in 33 blocks
==18941==    still reachable: 2,797,356 bytes in 3,674 blocks
==18941==                       of which reachable via heuristic:
==18941==                         length64           : 112,800 bytes in 207 blocks
==18941==         suppressed: 0 bytes in 0 blocks
==18941== Rerun with --leak-check=full to see details of leaked memory
==18941==
==18941== For counts of detected and suppressed errors, rerun with: -v
==18941== Use --track-origins=yes to see where uninitialised values come from
==18941== ERROR SUMMARY: 9620 errors from 151 contexts (suppressed: 0 from 0)
flux-module: cmb.rmmod[-1] pymod: Protocol error
Segmentation fault
@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 3, 2019

In case this is helpful, the segfaults in python from t2202-job-manager.t go away if I remove the Future class destructor (obviously not the right fix, but I don't really understand the python bindings yet so just experimenting)

Sounds like the probability it's a GC'd double free is going up 😢 . I'll poke around at that between now and the Social Coding Hour linux bonaza 😄 .

I tried to run the broker under valgrind and reproduce, but once python is involved, there are many thousands of errors from valgrind.

It looks like for python 2.2-3.5, there are a bunch of hoops to jump through, including reconfiguring/recompiling python with various flags. With Python 3.6+, it looks like setting PYTHONMALLOC=malloc solves a lot of problems with running python under valgrind: https://stackoverflow.com/questions/20112989/how-to-use-valgrind-with-python

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

@SteVwonder, just fyi, make check passes for me with PYTHON_VERSION=3.6 and my kludge for the Future destructor above, plus the fix you suggested for py_mod.c (take a reference on flux_t before passing it to Python):

diff --git a/src/modules/pymod/py_mod.c b/src/modules/pymod/py_mod.c           
index 5a578d369..93fd2dfa1 100644                                              
--- a/src/modules/pymod/py_mod.c
+++ b/src/modules/pymod/py_mod.c
@@ -174,6 +174,8 @@ int mod_main (flux_t *h, int argc, char **argv)
         if (py_flux_handle == NULL)
           return -1;
         PyTuple_SetItem(py_args, 1, py_flux_handle);
+        // Take a reference for python                                        
+        flux_incref (h);

         //Convert zhash to native python dict, should preserve mods
         //through switch to argc-style arguments
============================================================================   
Testsuite summary for flux-core 0.11.0-639-gdb1805a99                          
============================================================================   
# TOTAL: 1590                                                                  
# PASS:  1572                                                                  
# SKIP:  15                                                                    
# XFAIL: 3                                                                     
# FAIL:  0                                                                     
# XPASS: 0                                                                     
# ERROR: 0                                                                     
============================================================================   
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 3, 2019

Another FYI - for python 3.4, I'm getting a different error from pymod:

Traceback (most recent call last):
  File "/usr/src/src/bindings/python/flux/core/trampoline.py", line 21, in mod_main_trampoline
    user_mod = importlib.import_module("flux.modules." + name, "flux.modules")
  File "/usr/lib64/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2249, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2199, in _sanity_check
SystemError: Parent module 'flux.modules' not loaded, cannot perform relative import

making this change removes the error, but again, I'm not sure of the real fix:

diff --git a/src/bindings/python/flux/core/trampoline.py b/src/bindings/python/f
index 8743491..0c2d8ec 100644                                                  
--- a/src/bindings/python/flux/core/trampoline.py                              
+++ b/src/bindings/python/flux/core/trampoline.py                              
@@ -18,7 +18,7 @@ def mod_main_trampoline(name, int_handle, args):             
     flux_instance = Flux(handle=lib.unpack_long(int_handle))                  
     user_mod = None                                                           
     try:                                                                      
-        user_mod = importlib.import_module("flux.modules." + name, "flux.module
+        user_mod = importlib.import_module("flux.modules." + name)            
     except ImportError:  # check user paths for the module                    
         user_mod = importlib.import_module(name)                              
                                                                               
@SteVwonder

This comment has been minimized.

Copy link
Member

commented May 4, 2019

@grondo: I just pushed a few changes to that same branch in my fork (after a rebase on your cherry-picked commits). It includes:

  • an explicit close method on the Flux object that raises a descriptive exception to prevent the double free on flux_future
  • removal of the close calls in the various python testing scripts
  • an incref in the mod_main trampoline (you were spot on about the source of this problem).

For the python3.4 importlib error, I believe that is the right fix. Per the docs, If the name is specified in relative terms, then the package argument must be set to the name of the package which is to act as the anchor for resolving the package name (e.g. import_module('..mod', 'pkg.subpkg') will import pkg.mod). Since that isn't used for relative imports and is instead for importing installed modules, that should be fine.

@grondo grondo force-pushed the grondo:issue#2123 branch from ae9ef4a to 5ff17dc May 4, 2019
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

@grondo: I just pushed a few changes to that same branch in my fork (after a rebase on your cherry-picked commits).

Great! 👍

With your latest changes this PR actually passes tests for Python 2.7, 3.4 and 3.6 AFAICT.

I rebased and force-pushed the PR branch in order to put commits in a logical order. (I also added commit description to some of your commits @SteVwonder, hope you don't mind)

The changes to flux(1) to search for flux-<cmd>.py should get another review (and I should probably think about adding a couple tests), as should the kludge to get testsuite python scripts to run the correct python by adding src/cmd/python wrapper script.

O/w, this "small" fix is ready I think.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

Hm, pylint doesn't like the new handle close() method:

$ pylint --rcfile=src/bindings/python/.pylintrc src/bindings/python/flux
************* Module flux.core.handle
src/bindings/python/flux/core/handle.py:45:4: R0201: Method could be a function (no-self-use)

I'll try adding:

# pylint: disable=no-self-use

For now to get past the error.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2019

Some problems with make distcheck still, looking into it...

grondo and others added 3 commits Apr 15, 2019
Be sure to export PYTHON_VERSION to the docker environment before
running configure. This is how we select a different python version
for the build and test.

Fixes #2123
Be sure to use `flux python` when obtaining user site path so that
we use the flux confingured python not just the first one in PATH.
Problem: cmp() was removed from Python 3, but is necessary for
the bulk job state test.

Define cmp() locally using the "official" Python 3 workaround:

 https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons

Also apply black formatter.

Co-authored-by: Stephen Herbein <herbein1@llnl.gov>
SteVwonder and others added 10 commits May 2, 2019
Use of collections.abc is deprecated and will not work in python 3.8+.

Use plain collections where necessary to quiet deprecation warnings,
and to support 3.8.
Exception errors no longer have `message` member. The suggested
way to print the exception error string is with str(e).
Add a "python" wrapper to src/cmd/python which execs the python
interpreter specified at build time. This ensures that python
scripts run during `make check` use the configured python and
not the python interpreter first in the default PATH.
Support invoking the configured PYTHON_INTERPRETER on Python
scripts in FLUX_EXEC_PATH directly by checking for `flux-<cmd>.py`
in each directory before trying to execute `flux-<cmd>`.

This allows flux(1) to run python scripts with the correct version
of the interpreter without jumping through wrapper scripts, etc.
In order to support flux(1) invoking flux-jobspec under the python
interpreter, rename the script with a .py extension.
if the user calls .close on a handle, then raise a RuntimeError.
When this method is not explicitly provided by the bindings, the Wrapper
__getattr__ method would search the auto-generated bindings for
`flux_close`, call it with the `self.handle`, and result in a still
existing `Flux` object instance with a closed/free'd inner flux handle
Remove all explicity handle close calls from Python, instead allowing
the GC to auto-destruct and close these handles when they go out of
scope.
The pymod module does not own the memory for the handle passed to it
by the broker, and during module unload the hanlde was being closed
twice: once by the Python GC and once by the broker.

To fix, take an explicit reference in python to the handle before
giving it to the loaded python module.
On Python 3.4, the following line in trampoline.py

  user_mod = importlib.import_module("flux.modules." + name, "flux.modules")

was generating the error:

   SystemError: Parent module 'flux.modules' not loaded,
                cannot perform relative import

Remove the package="flux.modules" argument to quiet the warning, since
import_module() is not being used here for a relative import.
@grondo grondo force-pushed the grondo:issue#2123 branch from a6c8b63 to 16c09c2 May 6, 2019
@codecov-io

This comment has been minimized.

Copy link

commented May 6, 2019

Codecov Report

Merging #2124 into master will increase coverage by 0.01%.
The diff coverage is 86.66%.

@@            Coverage Diff             @@
##           master    #2124      +/-   ##
==========================================
+ Coverage   80.41%   80.43%   +0.01%     
==========================================
  Files         200      200              
  Lines       31756    31794      +38     
==========================================
+ Hits        25537    25573      +36     
- Misses       6219     6221       +2
Impacted Files Coverage Δ
src/cmd/flux.c 84.07% <86.66%> (+0.03%) ⬆️
src/modules/connector-local/local.c 73.62% <0%> (-1.04%) ⬇️
src/common/libjob/job.c 76.42% <0%> (-0.17%) ⬇️
src/modules/job-ingest/job-ingest.c 74.42% <0%> (+0.09%) ⬆️
src/common/libflux/message.c 81.64% <0%> (+0.24%) ⬆️
src/cmd/flux-job.c 86.6% <0%> (+0.24%) ⬆️
src/modules/barrier/barrier.c 80.53% <0%> (+2.01%) ⬆️
src/modules/job-info/lookup.c 67.85% <0%> (+2.74%) ⬆️
src/modules/job-info/watch.c 73.01% <0%> (+3.98%) ⬆️
@garlick

This comment has been minimized.

Copy link
Member

commented May 6, 2019

This ready to go in?

@grondo grondo changed the title docker-run-checks.sh: export PYTHON_VERSION travis-ci: properly export PYTHON_VERSION and fix python 3 fallout May 6, 2019
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 6, 2019

Yes, probably ready to go in.

@garlick garlick merged commit e82cd50 into flux-framework:master May 6, 2019
2 checks passed
2 checks passed
Summary 1 potential rule
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@grondo grondo deleted the grondo:issue#2123 branch May 6, 2019
@grondo

This comment has been minimized.

Copy link
Contributor Author

commented May 6, 2019

Thanks @garlick and @SteVwonder!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.