Permalink
Browse files

pyzfs: python3 support (build system)

Almost all of the Python code in the respository has been updated
to be compatibile with Python 2.6, Python 3.4, or newer.  The only
exceptions are arc_summery3.py which requires Python 3, and pyzfs
which requires at least Python 2.7.  This allows us to maintain a
single version of the code and support most default versions of
python.  This change does the following:

* Sets the default shebang for all Python scripts to python3.  If
  only Python 2 is available, then at install time scripts which
  are compatible with Python 2 will have their shebangs replaced
  with /usr/bin/python.  This is done for compatibility until
  Python 2 goes end of life.  Since only the installed versions
  are changed this means Python 3 must be installed on the system
  for test-runner when testing in-tree.

* Added --with-python=<2|3|3.4,etc> configure option which sets
  the PYTHON environment variable to target a specific python
  version.  By default the newest installed version of Python
  will be used or the preferred distribution version when
  creating pacakges.

* Fixed --enable-pyzfs configure checks so they are run when
  --enable-pyzfs=check and --enable-pyzfs=yes.

* Enabled pyzfs for Python 3.4 and newer, which is now supported.

* Renamed pyzfs package to python<VERSION>-pyzfs and updated to
  install in the appropriate site location.  For example, when
  building with --with-python=3.4 a python34-pyzfs will be
  created which installs in /usr/lib/python3.4/site-packages/.

* Renamed the following python scripts according to the Fedora
  guidance for packaging utilities in /bin

  - dbufstat.py     -> dbufstat
  - arcstat.py      -> arcstat
  - arc_summary.py  -> arc_summary
  - arc_summary3.py -> arc_summary3

* Updated python-cffi package name.  On CentOS 6, CentOS 7, and
  Amazon Linux it's called python-cffi, not python2-cffi.  For
  Python3 it's called python3-cffi or python3x-cffi.

* Install one version of arc_summary.  Depending on the version
  of Python available install either arc_summary2 or arc_summary3
  as arc_summary.  The user output is only slightly different.

Reviewed-by: John Ramsden <johnramsden@riseup.net>
Reviewed-by: Neal Gompa <ngompa@datto.com>
Reviewed-by: loli10K <ezomori.nozomu@gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #8096
  • Loading branch information...
behlendorf authored and madwizard committed Oct 31, 2018
1 parent 2d1d005 commit 7d6bee1e7cc8fbd0bb04a954a939e4b2e0ec90e1
Showing with 339 additions and 162 deletions.
  1. +2 −0 .gitignore
  2. +13 −1 cmd/arc_summary/Makefile.am
  3. +8 −6 cmd/arc_summary/{arc_summary.py → arc_summary2}
  4. +3 −3 cmd/arc_summary/{arc_summary3.py → arc_summary3}
  5. +13 −1 cmd/arcstat/Makefile.am
  6. +8 −7 cmd/arcstat/{arcstat.py → arcstat}
  7. +13 −1 cmd/dbufstat/Makefile.am
  8. +9 −7 cmd/dbufstat/{dbufstat.py → dbufstat}
  9. +102 −0 config/always-python.m4
  10. +28 −68 config/always-pyzfs.m4
  11. +1 −1 config/deb.am
  12. +7 −3 config/zfs-build.m4
  13. +1 −1 contrib/pyzfs/Makefile.am
  14. +7 −2 contrib/pyzfs/setup.py
  15. +56 −23 rpm/generic/zfs.spec.in
  16. +11 −0 tests/test-runner/bin/Makefile.am
  17. +2 −0 tests/test-runner/bin/test-runner.py
  18. +3 −1 tests/test-runner/bin/zts-report.py
  19. +4 −4 tests/zfs-tests/include/commands.cfg
  20. +2 −2 tests/zfs-tests/tests/functional/arc/dbufstats_001_pos.ksh
  21. +6 −6 tests/zfs-tests/tests/functional/arc/dbufstats_002_pos.ksh
  22. +3 −3 tests/zfs-tests/tests/functional/cli_user/misc/arc_summary3_001_pos.ksh
  23. +5 −5 tests/zfs-tests/tests/functional/cli_user/misc/arc_summary_001_pos.ksh
  24. +3 −3 tests/zfs-tests/tests/functional/cli_user/misc/arc_summary_002_neg.ksh
  25. +3 −3 tests/zfs-tests/tests/functional/cli_user/misc/arcstat_001_pos.ksh
  26. +5 −5 tests/zfs-tests/tests/functional/cli_user/misc/dbufstat_001_pos.ksh
  27. +1 −0 tests/zfs-tests/tests/functional/pyzfs/.gitignore
  28. +17 −3 tests/zfs-tests/tests/functional/pyzfs/Makefile.am
  29. +3 −3 tests/zfs-tests/tests/functional/pyzfs/{pyzfs_unittest.ksh → pyzfs_unittest.ksh.in}
@@ -22,6 +22,8 @@
*.swp
*.gcno
*.gcda
*.pyc
*.pyo
.deps
.libs
.dirstamp
@@ -1 +1,13 @@
dist_bin_SCRIPTS = arc_summary.py arc_summary3.py
EXTRA_DIST = arc_summary2 arc_summary3

if USING_PYTHON_2
dist_bin_SCRIPTS = arc_summary2
install-exec-hook:
mv $(DESTDIR)$(bindir)/arc_summary2 $(DESTDIR)$(bindir)/arc_summary
endif

if USING_PYTHON_3
dist_bin_SCRIPTS = arc_summary3
install-exec-hook:
mv $(DESTDIR)$(bindir)/arc_summary3 $(DESTDIR)$(bindir)/arc_summary
endif
@@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/python2
#
# $Id: arc_summary.pl,v 388:e27800740aa2 2011-07-08 02:53:29Z jhell $
#
@@ -35,6 +35,8 @@
# Note some of this code uses older code (eg getopt instead of argparse,
# subprocess.Popen() instead of subprocess.run()) because we need to support
# some very old versions of Python.
#

"""Print statistics on the ZFS Adjustable Replacement Cache (ARC)
Provides basic information on the ARC, its efficiency, the L2ARC (if present),
@@ -1005,7 +1007,7 @@ def zfs_header():
def usage():
"""Print usage information"""

sys.stdout.write("Usage: arc_summary.py [-h] [-a] [-d] [-p PAGE]\n\n")
sys.stdout.write("Usage: arc_summary [-h] [-a] [-d] [-p PAGE]\n\n")
sys.stdout.write("\t -h, --help : "
"Print this help message and exit\n")
sys.stdout.write("\t -a, --alternate : "
@@ -1018,10 +1020,10 @@ def usage():
"should be an integer between 1 and " +
str(len(unSub)) + "\n\n")
sys.stdout.write("Examples:\n")
sys.stdout.write("\tarc_summary.py -a\n")
sys.stdout.write("\tarc_summary.py -p 4\n")
sys.stdout.write("\tarc_summary.py -ad\n")
sys.stdout.write("\tarc_summary.py --page=2\n")
sys.stdout.write("\tarc_summary -a\n")
sys.stdout.write("\tarc_summary -p 4\n")
sys.stdout.write("\tarc_summary -ad\n")
sys.stdout.write("\tarc_summary --page=2\n")


def main():
@@ -346,7 +346,7 @@ def get_version(request):
error_msg = '(ERROR: "{0}" requested)'.format(request)
return error_msg

# The original arc_summary.py called /sbin/modinfo/{spl,zfs} to get
# The original arc_summary called /sbin/modinfo/{spl,zfs} to get
# the version information. We switch to /sys/module/{spl,zfs}/version
# to make sure we get what is really loaded in the kernel
command = ["cat", "/sys/module/{0}/version".format(request)]
@@ -374,7 +374,7 @@ def print_header():
"""

# datetime is now recommended over time but we keep the exact formatting
# from the older version of arc_summary.py in case there are scripts
# from the older version of arc_summary in case there are scripts
# that expect it in this way
daydate = time.strftime(DATE_FORMAT)
spc_date = LINE_LENGTH-len(daydate)
@@ -586,7 +586,7 @@ def section_archits(kstats_dict):

# For some reason, anon_hits can turn negative, which is weird. Until we
# have figured out why this happens, we just hide the problem, following
# the behavior of the original arc_summary.py
# the behavior of the original arc_summary.
if anon_hits >= 0:
prt_i2('Anonymously used:',
f_perc(anon_hits, arc_stats['hits']), f_hits(anon_hits))
@@ -1 +1,13 @@
dist_bin_SCRIPTS = arcstat.py
dist_bin_SCRIPTS = arcstat

#
# The arcstat script is compatibile with both Python 2.6 and 3.4.
# As such the python 3 shebang can be replaced at install time when
# targeting a python 2 system. This allows us to maintain a single
# version of the source.
#
if USING_PYTHON_2
install-exec-hook:
sed --in-place 's|^#!/usr/bin/python3|#!/usr/bin/python2|' \
$(DESTDIR)$(bindir)/arcstat
endif
@@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/python3
#
# Print out ZFS ARC Statistics exported via kstat(1)
# For a definition of fields, or usage, use arctstat.pl -v
@@ -42,7 +42,8 @@
# @hdr is the array of fields that needs to be printed, so we
# just iterate over this array and print the values using our pretty printer.
#

# This script must remain compatible with Python 2.6+ and Python 3.4+.
#

import sys
import time
@@ -109,7 +110,7 @@
sep = " " # Default separator is 2 spaces
version = "0.4"
l2exist = False
cmd = ("Usage: arcstat.py [-hvx] [-f fields] [-o file] [-s string] [interval "
cmd = ("Usage: arcstat [-hvx] [-f fields] [-o file] [-s string] [interval "
"[count]]\n")
cur = {}
d = {}
@@ -138,10 +139,10 @@ def usage():
sys.stderr.write("\t -s : Override default field separator with custom "
"character or string\n")
sys.stderr.write("\nExamples:\n")
sys.stderr.write("\tarcstat.py -o /tmp/a.log 2 10\n")
sys.stderr.write("\tarcstat.py -s \",\" -o /tmp/a.log 2 10\n")
sys.stderr.write("\tarcstat.py -v\n")
sys.stderr.write("\tarcstat.py -f time,hit%,dh%,ph%,mh% 1\n")
sys.stderr.write("\tarcstat -o /tmp/a.log 2 10\n")
sys.stderr.write("\tarcstat -s \",\" -o /tmp/a.log 2 10\n")
sys.stderr.write("\tarcstat -v\n")
sys.stderr.write("\tarcstat -f time,hit%,dh%,ph%,mh% 1\n")
sys.stderr.write("\n")

sys.exit(1)
@@ -1 +1,13 @@
dist_bin_SCRIPTS = dbufstat.py
dist_bin_SCRIPTS = dbufstat

#
# The dbufstat script is compatibile with both Python 2.6 and 3.4.
# As such the python 3 shebang can be replaced at install time when
# targeting a python 2 system. This allows us to maintain a single
# version of the source.
#
if USING_PYTHON_2
install-exec-hook:
sed --in-place 's|^#!/usr/bin/python3|#!/usr/bin/python2|' \
$(DESTDIR)$(bindir)/dbufstat
endif
@@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/python3
#
# Print out statistics for all cached dmu buffers. This information
# is available through the dbufs kstat and may be post-processed as
@@ -27,6 +27,8 @@
# Copyright (C) 2013 Lawrence Livermore National Security, LLC.
# Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
#
# This script must remain compatible with Python 2.6+ and Python 3.4+.
#

import sys
import getopt
@@ -106,7 +108,7 @@
hdr = None
xhdr = None
sep = " " # Default separator is 2 spaces
cmd = ("Usage: dbufstat.py [-bdhnrtvx] [-i file] [-f fields] [-o file] "
cmd = ("Usage: dbufstat [-bdhnrtvx] [-i file] [-f fields] [-o file] "
"[-s string] [-F filter]\n")
raw = 0

@@ -167,11 +169,11 @@ def usage():
"character or string\n")
sys.stderr.write("\t -F : Filter output by value or regex\n")
sys.stderr.write("\nExamples:\n")
sys.stderr.write("\tdbufstat.py -d -o /tmp/d.log\n")
sys.stderr.write("\tdbufstat.py -t -s \",\" -o /tmp/t.log\n")
sys.stderr.write("\tdbufstat.py -v\n")
sys.stderr.write("\tdbufstat.py -d -f pool,object,objset,dsize,cached\n")
sys.stderr.write("\tdbufstat.py -bx -F dbc=1,objset=54,pool=testpool\n")
sys.stderr.write("\tdbufstat -d -o /tmp/d.log\n")
sys.stderr.write("\tdbufstat -t -s \",\" -o /tmp/t.log\n")
sys.stderr.write("\tdbufstat -v\n")
sys.stderr.write("\tdbufstat -d -f pool,object,objset,dsize,cached\n")
sys.stderr.write("\tdbufstat -bx -F dbc=1,objset=54,pool=testpool\n")
sys.stderr.write("\n")

sys.exit(1)
@@ -0,0 +1,102 @@
dnl #
dnl # ZFS_AC_PYTHON_VERSION(version, [action-if-true], [action-if-false])
dnl #
dnl # Verify Python version
dnl #
AC_DEFUN([ZFS_AC_PYTHON_VERSION], [
ver_check=`$PYTHON -c "import sys; print (sys.version.split()[[0]] $1)"`
AS_IF([test "$ver_check" = "True"], [
m4_ifvaln([$2], [$2])
], [
m4_ifvaln([$3], [$3])
])
])

dnl #
dnl # ZFS_AC_PYTHON_MODULE(module_name, [action-if-true], [action-if-false])
dnl #
dnl # Checks for Python module. Freely inspired by AX_PYTHON_MODULE
dnl # https://www.gnu.org/software/autoconf-archive/ax_python_module.html
dnl # Required by ZFS_AC_CONFIG_ALWAYS_PYZFS.
dnl #
AC_DEFUN([ZFS_AC_PYTHON_MODULE], [
PYTHON_NAME=`basename $PYTHON`
AC_MSG_CHECKING([for $PYTHON_NAME module: $1])
AS_IF([$PYTHON -c "import $1" 2>/dev/null], [
AC_MSG_RESULT(yes)
m4_ifvaln([$2], [$2])
], [
AC_MSG_RESULT(no)
m4_ifvaln([$3], [$3])
])
])

dnl #
dnl # The majority of the python scripts are written to be compatible
dnl # with Python 2.6 and Python 3.4. Therefore, they may be installed
dnl # and used with either interpreter. This option is intended to
dnl # to provide a method to specify the default system version, and
dnl # set the PYTHON environment variable accordingly.
dnl #
AC_DEFUN([ZFS_AC_CONFIG_ALWAYS_PYTHON], [
AC_ARG_WITH([python],
AC_HELP_STRING([--with-python[=VERSION]],
[default system python version @<:@default=check@:>@]),
[with_python=$withval],
[with_python=check])

AS_CASE([$with_python],
[check],
[AS_IF([test -x /usr/bin/python3],
[PYTHON="python3"],
[AS_IF([test -x /usr/bin/python2],
[PYTHON="python2"],
[PYTHON=""]
)]
)],
[2*], [PYTHON="python${with_python}"],
[*python2*], [PYTHON="${with_python}"],
[3*], [PYTHON="python${with_python}"],
[*python3*], [PYTHON="${with_python}"],
[no], [PYTHON=""],
[AC_MSG_ERROR([Unknown --with-python value '$with_python'])]
)

AS_IF([$PYTHON --version >/dev/null 2>&1], [ /bin/true ], [
AC_MSG_ERROR([Cannot find $PYTHON in your system path])
])

AM_PATH_PYTHON([2.6], [], [:])
AM_CONDITIONAL([USING_PYTHON], [test "$PYTHON" != :])
AM_CONDITIONAL([USING_PYTHON_2], [test "${PYTHON_VERSION:0:2}" = "2."])
AM_CONDITIONAL([USING_PYTHON_3], [test "${PYTHON_VERSION:0:2}" = "3."])

dnl #
dnl # Minimum supported Python versions for utilities:
dnl # Python 2.6.x, or Python 3.4.x
dnl #
AS_IF([test "${PYTHON_VERSION:0:2}" = "2."], [
ZFS_AC_PYTHON_VERSION([>= '2.6'], [ /bin/true ],
[AC_MSG_ERROR("Python >= 2.6.x is not available")])
])

AS_IF([test "${PYTHON_VERSION:0:2}" = "3."], [
ZFS_AC_PYTHON_VERSION([>= '3.4'], [ /bin/true ],
[AC_MSG_ERROR("Python >= 3.4.x is not available")])
])

dnl #
dnl # Request that packages be built for a specific Python version.
dnl #
AS_IF([test $with_python != check], [
PYTHON_PKG_VERSION=`echo ${PYTHON} | tr -d 'a-zA-Z.'`
DEFINE_PYTHON_PKG_VERSION='--define "__use_python_pkg_version '${PYTHON_PKG_VERSION}'"'
DEFINE_PYTHON_VERSION='--define "__use_python '${PYTHON}'"'
], [
DEFINE_PYTHON_VERSION=''
DEFINE_PYTHON_PKG_VERSION=''
])

AC_SUBST(DEFINE_PYTHON_VERSION)
AC_SUBST(DEFINE_PYTHON_PKG_VERSION)
])
Oops, something went wrong.

0 comments on commit 7d6bee1

Please sign in to comment.