Python 2 and 3 compatibility

[behlendorf]

Motivation and Context

Update the build system to allow a specific version of Python to be targeted.

./configure --with-python=2|3|3.4

Since most of the Python code has been updated to be compatible with
Python 2.7 or Python 3.4 this allows us to install the correct version for the
distribution. This is necessary to support upcoming Fedora release which
will remove Python 2 entirely, and CentOS which will never provide Python 3
from the default repositories.

This change builds on pyzfs updates done by @aerusso in #7828.

Description

Almost all of the Python code in the respository has been updated
to be compatibile with Python 2.6, Python 3.4, or newer. The only
exceptions are arc_summery3.py which requires Python 3, and pyzfs
which requires at least Python 2.7. This allows us to maintain a
single version of the code and support most default versions of
python. This change does the following:

• Sets the default shebang for all Python scripts to python3. If
  only Python 2 is available, then at install time scripts which
  are compatible with Python 2 will have their shebangs replaced
  with /usr/bin/python. This is done for compatibility until
  Python 2 goes end of life. Since only the installed versions
  are changed this does mean Python 3 must be installed on the
  system when testing in-tree.

• Added --with-python=<2|3|3.4,etc> configure option which is used
  to set the PYTHON environment variable to target a specific
  python version. By default the newest installed version of
  Python we be used.

• Fixed --enable-pyzfs configure checks so they are run when
  --enable-pyzfs=check and --enable-pyzfs=yes.

• Enabled pyzfs for Python 3.4 and newer, which is now supported.

• Renamed pyzfs package to python-pyzfs and updated to
  install in the appropriate site location. For example, when
  building with --with-python=3.4 a python34-pyzfs will be
  created which installs in /usr/lib/python3.4/site-packages/.

How Has This Been Tested?

Packages build on CentOS 6 and CentOS 7 for Python 2.6, 2.7, and 3.4.
Submitted to the CI for additional build system testing.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the ZFS on Linux code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• All commit messages are properly formatted and contain Signed-off-by.

[Conan-Kudo]

Some initial feedback.

[behlendorf]

Refreshed based on @Conan-Kudo initial review feedback.

@loli10K @aerusso I'm going to need a hand finishing the conversion of pyzfs to be compatible with Python 2 and 3. I've updated the test infrastructure to run the unit tests using Python 3 and unfortunately many of the tests fail. I suspect someone actually familiar with Python could make short work of them.

15:59:33.22 ======================================================================
15:59:33.22 ERROR: test_boolean_array (libzfs_core.test.test_nvlist.TestNVList)
15:59:33.22 ----------------------------------------------------------------------
15:59:33.22 Traceback (most recent call last):
15:59:33.22   File "/usr/lib/python3.7/site-packages/libzfs_core/test/test_nvlist.py", line 374, in test_boolean_array
15:59:33.22     res = self._dict_to_nvlist_to_dict(props)
15:59:33.22   File "/usr/lib/python3.7/site-packages/libzfs_core/test/test_nvlist.py", line 39, in _dict_to_nvlist_to_dict
15:59:33.22     nv_in = nvlist_in(props)
15:59:33.22   File "/usr/lib/python3.7/site-packages/libzfs_core/_nvlist.py", line 76, in nvlist_in
15:59:33.22     _dict_to_nvlist(props, nvlist)
15:59:33.22   File "/usr/lib/python3.7/site-packages/libzfs_core/_nvlist.py", line 269, in _dict_to_nvlist
15:59:33.22     raise TypeError('Unsupported key type ' + type(k).__name__)
15:59:33.22 TypeError: Unsupported key type str

There may also need to be some more updates to test-runner needed.

Can someone also please confirm the minimum versions I set are correct:

Python 2.6+, or Python 3.4+: dbufstat, arcstat, arc_summary, test-runner.py, zts-report.py
Python 2.7+, or Python 3.4+: pyzfs

This means that by default pyzfs will only be available on CentOS 6 when Python 3.4 is installed from EPEL and Python 3.4 and pyzfs are requested,

$ ./configure --with-config=user --enable-pyzfs --with-python=3.4
$ make rpms
...
$ ls | grep python34
python34-pyzfs-0.8.0-rc1_99_g1cc98c0.el6.noarch.rpm

[rlaager]

For Python 3, should we install arc_summary3 as arc_summary?

[behlendorf]

For Python 3, should we install arc_summary3 as arc_summary?

Good idea, now would be the best time. The output is slightly different which may break some scripts, but not wildly so (whitespace, capitalization, extra rows).

[behlendorf]

Refreshed. The latest version addressed all the packaging issues I'm aware of related to this change. That should only leave updating test-runner and pyzfs for Python 3.

[loli10K]

This is troubling, buildbot does not detect these as failures

http://build.zfsonlinux.org/builders/Ubuntu%2017.04%20x86_64%20Coverage%20%28TEST%29/builds/4290/steps/shell_9/logs/tests

Test: /var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/zfs-tests/tests/functional/bootfs/bootfs_004_neg (run as root) [00:02] [PASS]
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 983, in emit
    stream.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 36-37: ordinal not in range(128)
Call stack:
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 922, in <module>
    main()
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 917, in main
    testrun.run(options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 728, in run
    self.testgroups[testgroup].run(self.logger, options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 483, in run
    test.log(logger, options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 287, in log
    logger.debug('%s %s' % (dt.strftime("%H:%M:%S.%f ")[:11], line))
Message: '00:41:24.97 SUCCESS: zpool create %s\xa3\xa3%^/testfs /mnt/testdir/bootfs_004.27578.dat exited 1'
Arguments: ()

The following patch seems to solve this kind of issue with the logging FileHandler:

diff --git a/tests/test-runner/bin/test-runner.py b/tests/test-runner/bin/test-runner.py
index a0fcd1c61..7f7db2dc0 100755
--- a/tests/test-runner/bin/test-runner.py
+++ b/tests/test-runner/bin/test-runner.py
@@ -689,7 +689,7 @@ class TestRun(object):
                 fail('%s' % e)
             filename = os.path.join(self.outputdir, 'log')
 
-            logfile = logging.FileHandler(filename)
+            logfile = logging.FileHandler(filename, encoding='utf-8')
             logfile.setLevel(logging.DEBUG)
             logfilefmt = logging.Formatter('%(message)s')
             logfile.setFormatter(logfilefmt)

[behlendorf]

@loli10K thanks, I added your test-runner change to this patch stack.

[loli10K]

The Coverage builder (Python 3.5) is now dying here (http://build.zfsonlinux.org/builders/Ubuntu%2017.04%20x86_64%20Coverage%20%28TEST%29/builds/4335/steps/shell_9/logs/tests):

Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 922, in <module>
    main()
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 917, in main
    testrun.run(options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 728, in run
    self.testgroups[testgroup].run(self.logger, options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 479, in run
    test.run(options)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 235, in run
    self.result.stdout, self.result.stderr = self.collect_output(proc)
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 201, in collect_output
    fd.read()
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 99, in read
    while self._read() is not None:
  File "/var/lib/buildbot/slaves/zfs/Ubuntu_17_04_x86_64_Coverage__TEST_/build/zfs/tests/test-runner/bin/test-runner.py", line 111, in _read
    buf = bytes.decode(os.read(fd, 4096))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 687: invalid continuation byte

It still troubles me buildbot does not mark the " zfstests" step as "FAILED" in the build summary: does it make sense to abort the build (and report) when such unhandled errors happens?

diff --git a/scripts/zfs-tests.sh b/scripts/zfs-tests.sh
index 6fdd658f7..996dca699 100755
--- a/scripts/zfs-tests.sh
+++ b/scripts/zfs-tests.sh
@@ -583,8 +583,11 @@ REPORT_FILE=$(mktemp -u -t zts-report.XXXX -p "$FILEDIR")
 #
 msg "${TEST_RUNNER} ${QUIET} -c ${RUNFILE} -T ${TAGS} -i ${STF_SUITE}" \
     "-I ${ITERATIONS}"
+set -o pipefail
 ${TEST_RUNNER} ${QUIET} -c "${RUNFILE}" -T "${TAGS}" -i "${STF_SUITE}" \
     -I "${ITERATIONS}" 2>&1 | tee "$RESULTS_FILE"
+RESULT=$? # <- fail with this != 0
+set +o pipefail
 
 #
 # Analyze the results.

[behlendorf]

Right, I definitely think this should fail. That last thing we want is to incorrectly reporting success here. Let me also pull in your pipefail change.

[loli10K]

We should be able to fix this by explicitly forcing UTF-8 encoding here:

diff --git a/tests/test-runner/bin/test-runner.py b/tests/test-runner/bin/test-runner.py
index 7f7db2d..d3a356a 100755
--- a/tests/test-runner/bin/test-runner.py
+++ b/tests/test-runner/bin/test-runner.py
@@ -108,7 +108,7 @@ class Output(object):
         for easy sorting/merging later.
         """
         fd = self.fileno()
-        buf = bytes.decode(os.read(fd, 4096))
+        buf = bytes.decode(os.read(fd, 4096), encoding='utf-8')
         if not buf:
             return None
         if '\n' not in buf:

We also need to convert tests/zfs-tests/tests/functional/channel_program/lua_core/tst.lib_table.lua from ISO-8859 to UTF-8 to fix the following issue:

Test: /usr/share/zfs/zfs-tests/tests/functional/channel_program/lua_core/tst.large_prog (run as root) [00:00] [PASS]
Traceback (most recent call last):
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 922, in <module>
    main()
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 917, in main
    testrun.run(options)
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 728, in run
    self.testgroups[testgroup].run(self.logger, options)
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 479, in run
    test.run(options)
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 235, in run
    self.result.stdout, self.result.stderr = self.collect_output(proc)
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 201, in collect_output
    fd.read()
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 99, in read
    while self._read() is not None:
  File "/usr/share/zfs/test-runner/bin/test-runner.py", line 111, in _read
    buf = bytes.decode(os.read(fd, 4096))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 687: invalid continuation byte

Test PASS from my Fedora27 builder:

[root@linux zfs]# sudo -u nobody -s ./scripts/zfs-tests.sh -d /var/tmp -t tests/functional/channel_program/lua_core/tst.libraries
Test: /usr/src/zfs/tests/zfs-tests/tests/functional/channel_program/lua_core/setup (run as root) [00:01] [PASS]
Test: /usr/src/zfs/tests/zfs-tests/tests/functional/channel_program/lua_core/tst.libraries (run as root) [00:00] [PASS]
Test: /usr/src/zfs/tests/zfs-tests/tests/functional/channel_program/lua_core/cleanup (run as root) [00:00] [PASS]

Results Summary
PASS	   3

Running Time:	00:00:03
Percent passed:	100.0%
Log directory:	/var/tmp/test_results/20181113T212434

Tests with results other than PASS that are expected:

Tests with result of PASS that are unexpected:

Tests with results other than PASS that are unexpected:
[root@linux zfs]# file tests/zfs-tests/tests/functional/channel_program/lua_core/tst.lib_table.lua* 
tests/zfs-tests/tests/functional/channel_program/lua_core/tst.lib_table.lua:     UTF-8 Unicode text
tests/zfs-tests/tests/functional/channel_program/lua_core/tst.lib_table.lua.old: ISO-8859 text
[root@linux zfs]#  

[Conan-Kudo]

This needs to be marked to block the 0.8.0 release, and #7828 should probably be closed since this supersedes it.

[codecov]

Codecov Report

Merging #8096 into master will decrease coverage by 0.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8096      +/-   ##
==========================================
- Coverage   78.57%   78.34%   -0.23%     
==========================================
  Files         379      379              
  Lines      114924   114924              
==========================================
- Hits        90299    90042     -257     
- Misses      24625    24882     +257

Flag	Coverage Δ
#kernel	78.79% <ø> (-0.19%)	⬇️
#user	67.14% <ø> (-0.25%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0b8e441...b75e1c5. Read the comment docs.

[behlendorf]

@loli10K @Conan-Kudo @aerusso @GregorKopka @johnramsden the updated patch stack is ready for review. The the updated pyzfs library and unit tests have been tested on Python 2.7, 3.4, 3.5, 3.6, and 3.7 and the build system was updated so a particular version can be requested.

[Conan-Kudo]

Pending the final run of the tests, it looks good to me.