fixing py3 tests

[AndreasAlbert]

No description provided.

[AndreasAlbert]

Issues addressed here:

• the latest ROOT container recently switched ubuntu versions, which necessitates a different treatment of python/python3. To avoid future surprises, I now fixed the container version to a specific one. If we want to update in the future, we should do it knowingly.
• jupyter needed extra commands to recognize the python3 kernel properly
• new pylint version enforces W0707 (raise-missing-from). We now follow this.

Still TODO:

• notebook execution fails with "dead kernel" error. I do not yet know what that even means. If others know, feel free to share.
• so far, I just fixed up one of the CI workflows. Once this one runs, can port knowledge to others.
• After everything works, should also consider reorganizing workflows for efficiency.

[emanueleusai]

I saw the kernel death too. It appears to be coming from a segfault when running cell 3 of:
https://github.com/HEPData/hepdata_lib/blob/master/examples/reading_histograms.ipynb

Oddly enough I cannot reproduce the segfault when running locally from my 6.22 installation through LCG98 (Edit: python 3.7).

EDIT: apparently rootproject/root:6.22.06-ubuntu20.04 ships python 3.8.5, that might potentially cause issues with some packages.

Here's the segfault output when running the notebook from the python cmd line (from the ubuntu 6.22 docker image):

from hepdata_lib import RootFileReader

reader = RootFileReader("example_inputs/mlfit_lm_1000.root")

RooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby
Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
All rights reserved, please read http://roofit.sourceforge.net/license.txt

*** Break *** segmentation violation
Generating stack trace...
0x00007fb5458e4c4a in from /opt/root/lib/libRIO.so
0x00007fb5458f190a in TStreamerInfo::CompareContent(TClass*, TVirtualStreamerInfo*, bool, bool, TFile*) + 0x2a7a from /opt/root/lib/libRIO.so
0x00007fb5458e8678 in TStreamerInfo::BuildCheck(TFile*, bool) + 0x898 from /opt/root/lib/libRIO.so
0x00007fb5458957bd in TFile::ReadStreamerInfo() + 0x20d from /opt/root/lib/libRIO.so
0x00007fb5458a126f in TFile::Init(bool) + 0x44f from /opt/root/lib/libRIO.so
0x00007fb5458a2b69 in TFile::TFile(char const*, char const*, char const*, int) + 0x839 from /opt/root/lib/libRIO.so
0x00007fb5348b30b6 in
0x00007fb546302898 in from /opt/root/lib/libcppyy_backend3_8.so.6.22
0x00007fb546303d20 in Cppyy::CallConstructor(long, unsigned long, unsigned long, void*) + 0x30 from /opt/root/lib/libcppyy_backend3_8.so.6.22
0x00007fb54638d676 in CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) + 0x36 from /opt/root/lib/libcppyy3_8.so
0x00007fb54638b455 in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) + 0x45 from /opt/root/lib/libcppyy3_8.so
0x00007fb54638611d in CPyCppyy::CPPConstructor::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*) + 0x22d from /opt/root/lib/libcppyy3_8.so
0x00007fb546390629 in from /opt/root/lib/libcppyy3_8.so
0x00000000005f3f3e in PyObject_Call + 0x25e from python3
0x000000000059cb04 in from python3
0x00000000005f463f in _PyObject_MakeTpCall + 0x1ff from python3
0x0000000000570936 in _PyEval_EvalFrameDefault + 0x5de6 from python3
0x00000000005f7146 in _PyFunction_Vectorcall + 0x1b6 from python3
0x00000000005f4ca1 in from python3
0x00000000005f4f65 in PyObject_CallFunctionObjArgs + 0x95 from python3
0x0000000000614eb1 in from python3
0x00000000005c19ab in PyObject_SetAttr + 0xd9b from python3
0x000000000056b9aa in _PyEval_EvalFrameDefault + 0xe5a from python3
0x00000000005f7146 in _PyFunction_Vectorcall + 0x1b6 from python3
0x000000000059c95d in from python3
0x00000000005f463f in _PyObject_MakeTpCall + 0x1ff from python3
0x00000000005704b9 in _PyEval_EvalFrameDefault + 0x5969 from python3
0x000000000056955a in _PyEval_EvalCodeWithName + 0x26a from python3
0x000000000068c4a7 in PyEval_EvalCode + 0x27 from python3
0x000000000067bc91 in from python3
0x000000000067bd0f in from python3
0x00000000004a3291 in from python3
0x00000000004a4305 in PyRun_InteractiveLoopFlags + 0xdb from python3
0x000000000067e059 in PyRun_AnyFileExFlags + 0x59 from python3
0x00000000004ee716 in from python3
0x00000000006b63bd in Py_BytesMain + 0x2d from python3
0x00007fb5489900b3 in __libc_start_main + 0xf3 from /lib/x86_64-linux-gnu/libc.so.6
0x00000000005fa4de in _start + 0x2e from python3
*** Break *** segmentation violation
Generating stack trace...
0x00007fb5458e4c4a in from /opt/root/lib/libRIO.so
0x00007fb5458f190a in TStreamerInfo::CompareContent(TClass*, TVirtualStreamerInfo*, bool, bool, TFile*) + 0x2a7a from /opt/root/lib/libRIO.so
0x00007fb5458e8678 in TStreamerInfo::BuildCheck(TFile*, bool) + 0x898 from /opt/root/lib/libRIO.so
0x00007fb5458957bd in TFile::ReadStreamerInfo() + 0x20d from /opt/root/lib/libRIO.so
0x00007fb5458a126f in TFile::Init(bool) + 0x44f from /opt/root/lib/libRIO.so
0x00007fb5458a2b69 in TFile::TFile(char const*, char const*, char const*, int) + 0x839 from /opt/root/lib/libRIO.so
0x00007fb5348b30b6 in
0x00007fb546302898 in from /opt/root/lib/libcppyy_backend3_8.so.6.22
0x00007fb546303d20 in Cppyy::CallConstructor(long, unsigned long, unsigned long, void*) + 0x30 from /opt/root/lib/libcppyy_backend3_8.so.6.22
0x00007fb54638d676 in CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) + 0x36 from /opt/root/lib/libcppyy3_8.so
0x00007fb54638b455 in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) + 0x45 from /opt/root/lib/libcppyy3_8.so
0x00007fb54638611d in CPyCppyy::CPPConstructor::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*) + 0x22d from /opt/root/lib/libcppyy3_8.so
0x00007fb546390629 in from /opt/root/lib/libcppyy3_8.so
0x00000000005f3f3e in PyObject_Call + 0x25e from python3
0x000000000059cb04 in from python3
0x00000000005f463f in _PyObject_MakeTpCall + 0x1ff from python3
0x0000000000570936 in _PyEval_EvalFrameDefault + 0x5de6 from python3
0x00000000005f7146 in _PyFunction_Vectorcall + 0x1b6 from python3
0x00000000005f4ca1 in from python3
0x00000000005f4f65 in PyObject_CallFunctionObjArgs + 0x95 from python3
0x0000000000614eb1 in from python3
0x00000000005c19ab in PyObject_SetAttr + 0xd9b from python3
0x000000000056b9aa in _PyEval_EvalFrameDefault + 0xe5a from python3
0x00000000005f7146 in _PyFunction_Vectorcall + 0x1b6 from python3
0x000000000059c95d in from python3
0x00000000005f463f in _PyObject_MakeTpCall + 0x1ff from python3
0x00000000005704b9 in _PyEval_EvalFrameDefault + 0x5969 from python3
0x000000000056955a in _PyEval_EvalCodeWithName + 0x26a from python3
0x000000000068c4a7 in PyEval_EvalCode + 0x27 from python3
0x000000000067bc91 in from python3
0x000000000067bd0f in from python3
0x00000000004a3291 in from python3
0x00000000004a4305 in PyRun_InteractiveLoopFlags + 0xdb from python3
0x000000000067e059 in PyRun_AnyFileExFlags + 0x59 from python3
0x00000000004ee716 in from python3
0x00000000006b63bd in Py_BytesMain + 0x2d from python3
0x00007fb5489900b3 in __libc_start_main + 0xf3 from /lib/x86_64-linux-gnu/libc.so.6
0x00000000005fa4de in _start + 0x2e from python3

[AndreasAlbert]

Summary of changes to testing:

1. Disabled macOS and py2.7 tests for now. These will require more attention to be fixed, but with this PR we at least have SOME working tests again.
2. Cleaned up setup for py3. I replaced the use of the ROOT container with a download of the precompiled binaries from the ROOT website. These support py3 now. Additional workarounds for ImageMagick setup.
3. Merged the coverage workflow into the testing workflow.
4. Smaller changes: Some pylint cleanup, and replacement of the mlfit file, which for unknown reasons is not readable in newer ROOT. I just rewrote the histograms to a new file and replaced it, which works.

[clelange]

Awesome, thanks a lot for this!
We could also think about deprecating Python 2.7, but we could also wait until CMSSW has finally moved...

[clelange]

We should track whatever we want to re-enable later on in issues.