Support Python 3.8

[mwichmann]

This is a work tracker issue to make sure scons works with Python 3.8.

The default warning filters have apparently changed, and now there are lots of warnings issued when running the test suite. As this issue is recorded, a run on Linux using Python 3.7.2 throws three warnings, all relating to the .sconsign.dblite file not being closed. With 3.8.0a1 the warning count goes up to about 13750. Since a large number of the tests capture stdout/stderr and judge success by whether they match expected text strings, if warnings appear in stderr those tests will fail. 871 test fails were recorded (up from two on my particular setup), they have not been examined for causes.

Updated: using 3.8.0a2, local setup, 193 passes, 874 fail, 132 skip.The problems that jump out the most are warnings that appear everywhere and spill into the stderr of the test; if the test is trying to exactly match stderr (including stderr is empty) will fail. That comes down to these three:

1. These two are from a code snippet used to remap module names for Py2/Py3 compatibility without sprinkling conditionals all over the place.

src/engine/SCons/compat/__init__.py:64: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp  # Use the "imp" module to protect imports from fixers.
src/engine/SCons/compat/__init__.py:84: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/mats/.pyenv/versions/3.8.0a2/lib/python3.8/cProfile.py' mode='r' encoding='utf-8'>
  sys.modules[new] = imp.load_module(old, *imp.find_module(old))

(cProfile is one of the modules that gets this special handling)
See also PR #3159, which proposes a solution to imp module usage.

2. .sconsign.dblite is apparently not closed:

src/engine/SCons/dblite.py:207: ResourceWarning: unclosed file <_io.BufferedReader name='.sconsign.dblite'>
  return dblite(file, flag, mode)
src/engine/SCons/dblite.py:118: ResourceWarning: unclosed file <_io.BufferedWriter name='.sconsign.dblite'>
  self._open(self._file_name, "wb", self._mode)

3. When the scons instance running a test opens and invokes SConstruct, it is apparently never closed.

src/engine/SCons/Script/Main.py:1015: ResourceWarning: unclosed file <_io.BufferedReader name='/tmp/testcmd.1949425.sh9zot0z/SConstruct'>
    SCons.Script._SConscript._SConscript(fs, script)

[thecraftman]

can this be done also using the jupyter notebook ?

[bdbaddog]

can this be done also using the jupyter notebook ?

How would that be useful?
(I've not used jupyter notebooks much)

[thecraftman]

@bdbaddog hey the installation guide for jupyter is on my profile. I created a native game in python called Tic-Tac-Toe and the intsllation guide for jupyter is there

[bdbaddog]

@bdbaddog hey the installation guide for jupyter is on my profile. I created a native game in python called Tic-Tac-Toe and the intsllation guide for jupyter is there

That's interesting. But again why would we use it for this?

[thecraftman]

In Jupyter you can investigate a new perspective with each new cell you create while the data and it's transformed copies remain ready, and the former investigations remain visible.

@bdbaddog hey the installation guide for jupyter is on my profile. I created a native game in python called Tic-Tac-Toe and the intsllation guide for jupyter is there

That's interesting. But again why would we use it for this?

In Jupyter you can investigate a new perspective with each new cell you create while the data and it's transformed copies remain ready, and the former investigations remain visible.

[bdbaddog]

@thecraftman That's interesting, but you can't run scons inside jupyter without significant work and we can spend that time doing more productive work. If you want to participate in any scons development I suggest you bring your discussion to the scons-users mailing list.

https://scons.org/lists.html

[thecraftman]

@bdbaddog alright thanks.

[mwichmann]

Proposed fixes for items 2 and 3 are contained in PR #3327

[bdbaddog]

@mwichmann - do we still need to resolve any of these issues for Python 3.8.*? Or has the remaining item been kicked down the road to Python 3.9?

[mwichmann]

Current scons works fine on 3.8. The imp/importlib thing is still just a warning. The only "problem" I'm seeing is that with 3.8, Python reports the line number differently in certain error situations, at the moment only one test fails because of this. It should be find to close (esp. if you merge #3552!)

Notes: ff a statement has multiple lines, Python now reports on the first line. From the test that fails:

12 env.Package( NAME           = 'foo',
13              VERSION        = '1.2.3',
14              PACKAGEVERSION = 0,
15              PACKAGETYPE    = 'rpm',
16              LICENSE        = 'gpl',
17              SUMMARY        = 'balalalalal',
18              X_RPM_GROUP    = 'Application/fu',
19              X_RPM_INSTALL  = r'%(_python_)s %(scons)s --tree=all ...
20              DESCRIPTION    = 'this should be really really long',
21              source         = [ prog ],
22              target         = "my_rpm_package.rpm",
23              SOURCE_URL     = 'http://foo.org/foo-1.2.3.tar.gz'
24         )

Test expectation is based on behavior of past Pythons, so we see this from running the test. Probably the test should be adjusted to not be looking for a specific line number.

STDERR =========================================================================
3c3
< File "/tmp/testcmd.498660.6hzmf6i6/SConstruct", line 23, in <module>
---
> File "/tmp/testcmd.498660.6hzmf6i6/SConstruct", line 12, in <module>

[mwichmann]

So barring the unresolved item from the previous comment, I don't see any reason to leave this open, scons is working fine with 3.8.

[bdbaddog]

Closing per @mwichmann