Skip to content

Commit

Permalink
Corrected install of xmllint and of lxml prereqs
Browse files Browse the repository at this point in the history 
Background:

- Since its version 4.0.0, the lxml package on Pypi is a binary wheel,
  which contains all of its prerequisite OS-level packages. The pywbem
  development setup still (incorrectly) assumed that the OS-level
  packages `libxml2`, `libxslt`, and for Windows, `iconv` and `zlib`
  would be needed for lxml. Using the binary wheel for lxml eliminates
  these dependencies.

- However, the `xmllint` utility that is used by the pywbem testsuite,
  on Linux and OS-X is part of the `libxml2` / `libxml2-utils` package.
  So that package is still needed.
  On Windows, it comes with the Unix-like environments such as CygWin.

Details:

- Removed the manual download and installation of the OS-level
  packages `libxml2`, `libxslt`, `iconv` and `zlib` in the Appveyor CI
  environment (Windows).

- Removed the OS-level package `libxslt` from the `pywbem_os_setup.sh`
  script that is used on Linux and OS-X.

- Updated the development section of the documentation accordingly.

- Enabled the 64-bit environment on Python 2.7 in the Appveyor CI
  and removed the corresponding documented restriction about testing
  and the resulting uncertainly of whether it would work in that
  environment.

- Added a recommendation to the development section of the documentation
  to use a Unix-like environment on Windows for development.

- Added display of location and version of `xmllint` command in the
  Appveyor CI.

Signed-off-by: Andreas Maier <maiera@de.ibm.com>
  • Loading branch information
@andy-maier
andy-maier committed Jan 29, 2018
1 parent 4438f53 commit de1c4be
Show file tree
Hide file tree
Showing 4 changed files with 33 additions and 307 deletions.
151 changes: 7 additions & 144 deletions appveyor.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,6 @@ environment:
# To support Python 2.6, M2Crypto would need to be built upon install from
# the M2Crypto package. See https://github.com/dsoprea/M2CryptoWindows for
# details on how to build it.
# * The use of libxslt devel packages from ftp.zlatkovic.com causes issues
# on some 64-bit environments. The 64-bit version of this package contains
# the libxslt library in .a format, but the lxml build process wants it
# in .lib format, so the linker step fails with:
# LINK : fatal error LNK1181: cannot open input file 'libxslt.lib'
# For reasons not completely understood, this causes the 64-bit build only
# to fail on Python 2.6.
#------------------------
# - PYTHON_VERSION: 2.6
# PYTHON_ARCH: 32
Expand All @@ -29,9 +22,9 @@ environment:
- PYTHON_VERSION: 2.7
PYTHON_ARCH: 32
PYTHON_HOME: C:\Python27
# - PYTHON_VERSION: 2.7
# PYTHON_ARCH: 64
# PYTHON_HOME: C:\Python27-x64
- PYTHON_VERSION: 2.7
PYTHON_ARCH: 64
PYTHON_HOME: C:\Python27-x64
# - PYTHON_VERSION: 3.4
# PYTHON_ARCH: 32
# PYTHON_HOME: C:\Python34
Expand Down Expand Up @@ -125,139 +118,6 @@ install:
#- ps: (new-object System.Net.WebClient).Downloadfile('https://bootstrap.pypa.io/get-pip.py', 'C:\Users\appveyor\get-pip.py')
#- ps: Start-Process -FilePath "C:\Python27\python.exe" -ArgumentList "C:\Users\appveyor\get-pip.py" -Wait -Passthru

## Disabled: The following is an attempt to install the lxml installation
## prereqs with choco. The lxml installation currently fails with unresolved
## symbols at the link step. Reported the issue to the Appveyor Problems forum:
## http://help.appveyor.com/discussions/problems/5330-pip-install-lxml-fails-with-missing-symbols
##
## Install OS-level prereqs for lxml installation: libxml2, libxslt, zlib,
## libiconv (needs iconv.lib).
#- choco source add -n=nuget -s="https://www.nuget.org/api/v2/"
#- set PATH=C:\Users\appveyor\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin;%PATH%
#- choco install -y -v libxml2
#- set INCLUDE=C:\ProgramData\chocolatey\lib\libxml2\build\native\include;%INCLUDE%
#- set _LIB_XML2=C:\ProgramData\chocolatey\lib\libxml2\build\native\lib\v110\Win32\Release\static\cdecl
#- set LIB=%_LIB_XML2%;%LIB%
#- dir %_LIB_XML2%
#- dumpbin /symbols %_LIB_XML2%\libxml2.lib
#- dumpbin /exports %_LIB_XML2%\libxml2.lib
#- choco install -y -v libxslt
#- set INCLUDE=C:\ProgramData\chocolatey\lib\libxslt\build\native\include;%INCLUDE%
#- set _LIB_XSLT=C:\ProgramData\chocolatey\lib\libxslt\build\native\lib\v110\Win32\Release\static
#- set LIB=%_LIB_XSLT%;%LIB%
#- dir %_LIB_XSLT%
#- choco install -y -v zlib
#- set INCLUDE=C:\ProgramData\chocolatey\lib\zlib.v120.windesktop.msvcstl.dyn.rt-dyn\build\native\include;%INCLUDE%
#- set _LIB_ZLIB=C:\ProgramData\chocolatey\lib\zlib.v120.windesktop.msvcstl.dyn.rt-dyn\lib\native\v120\windesktop\msvcstl\dyn\rt-dyn\Win32\Release
#- set LIB=%_LIB_ZLIB%;%LIB%
#- dir %_LIB_ZLIB%
#- choco install -y -v libiconv
#- set INCLUDE=C:\ProgramData\chocolatey\lib\libiconv\build\native\include;%INCLUDE%
#- set _LIB_ICONV=C:\ProgramData\chocolatey\lib\libiconv\build\native\lib\v110\Win32\Release\static\cdecl
#- set LIB=%_LIB_ICONV%;%LIB%
#- copy %_LIB_ICONV%\libiconv.lib %_LIB_ICONV%\iconv.lib
#- dir %_LIB_ICONV%
#- find c:/ProgramData/chocolatey -name "*.h"
#- find c:/ProgramData/chocolatey -name "*.lib"
#- find c:/ProgramData/chocolatey -name "*.dll"

# Install OS-level prereqs for lxml installation: libxml2, libxslt, zlib,
# libiconv (needs iconv.lib). This approach uses the binary libraries
# that are linked from the lxml site.
- echo set _PWD=%%%%~dp0>tmp_prereq_dir.bat
- call tmp_prereq_dir.bat
- rm tmp_prereq_dir.bat
- set _PREREQ_DIR=prereqs
- set _PREREQ_ABSDIR=%_PWD%%_PREREQ_DIR%
- echo Installing lxml prereqs into %_PREREQ_ABSDIR%
- mkdir %_PREREQ_DIR%

# Add OS-level prereq: libxml2
- if %PYTHON_ARCH%.==32. (
set _PKGFILE=libxml2-2.7.8.win32.zip&
set _PKGZIPDIR=libxml2-2.7.8.win32&
set _PKGLOCALDIR=.)
else (
set _PKGFILE=libxml2-2.9.3-win32-x86_64.7z&
set _PKGZIPDIR=.&
set _PKGLOCALDIR=libxml2-2.9.3.win32-x86_64)
- if %PYTHON_ARCH%.==32. (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/%_PKGFILE%&
unzip -q -d %_PREREQ_DIR% %_PREREQ_DIR%/%_PKGFILE%)
else (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/64bit/%_PKGFILE%&
mkdir %_PREREQ_DIR%\%_PKGLOCALDIR%&
7z x -o%_PREREQ_DIR%/%_PKGLOCALDIR% %_PREREQ_DIR%/%_PKGFILE%)
- if %PYTHON_ARCH%.==32. (
set INCLUDE=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\include;%INCLUDE%)
else (
set INCLUDE=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\include\libxml2;%INCLUDE%)
- set LIB=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\lib;%LIB%
- set PATH=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\bin;%PATH%

# Add OS-level prereq: libxslt
- if %PYTHON_ARCH%.==32. (
set _PKGFILE=libxslt-1.1.26.win32.zip&
set _PKGZIPDIR=libxslt-1.1.26.win32&
set _PKGLOCALDIR=.)
else (
set _PKGFILE=libxslt-1.1.28-win32-x86_64.7z&
set _PKGZIPDIR=.&
set _PKGLOCALDIR=libxslt-1.1.28.win32-x86_64)
- if %PYTHON_ARCH%.==32. (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/%_PKGFILE%&
unzip -q -d %_PREREQ_DIR% %_PREREQ_DIR%/%_PKGFILE%)
else (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/64bit/%_PKGFILE%&
echo mkdir %_PREREQ_DIR%\%_PKGLOCALDIR%&
7z x -o%_PREREQ_DIR%/%_PKGLOCALDIR% %_PREREQ_DIR%/%_PKGFILE%)
- set INCLUDE=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\include;%INCLUDE%
- set LIB=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\lib;%LIB%
- set PATH=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\bin;%PATH%

# Add OS-level prereq: zlib
- if %PYTHON_ARCH%.==32. (
set _PKGFILE=zlib-1.2.5.win32.zip&
set _PKGZIPDIR=zlib-1.2.5&
set _PKGLOCALDIR=.)
else (
set _PKGFILE=zlib-1.2.8-win32-x86_64.7z&
set _PKGZIPDIR=.&
set _PKGLOCALDIR=zlib-1.2.8.win32-x86_64)
- if %PYTHON_ARCH%.==32. (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/%_PKGFILE%&
unzip -q -d %_PREREQ_DIR% %_PREREQ_DIR%/%_PKGFILE%)
else (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/64bit/%_PKGFILE%&
echo mkdir %_PREREQ_DIR%\%_PKGLOCALDIR%&
7z x -o%_PREREQ_DIR%/%_PKGLOCALDIR% %_PREREQ_DIR%/%_PKGFILE%)
- set INCLUDE=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\include;%INCLUDE%
- set LIB=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\lib;%LIB%
- set PATH=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\bin;%PATH%

# Add OS-level prereq: iconv
- if %PYTHON_ARCH%.==32. (
set _PKGFILE=iconv-1.9.2.win32.zip&
set _PKGZIPDIR=iconv-1.9.2.win32&
set _PKGLOCALDIR=.)
else (
set _PKGFILE=iconv-1.14-win32-x86_64.7z&
set _PKGZIPDIR=.&
set _PKGLOCALDIR=iconv-1.14.win32-x86_64)
- if %PYTHON_ARCH%.==32. (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/%_PKGFILE%&
unzip -q -d %_PREREQ_DIR% %_PREREQ_DIR%/%_PKGFILE%)
else (
wget -q -P %_PREREQ_DIR% ftp://ftp.zlatkovic.com/libxml/64bit/%_PKGFILE%&
echo mkdir %_PREREQ_DIR%\%_PKGLOCALDIR%&
7z x -o%_PREREQ_DIR%/%_PKGLOCALDIR% %_PREREQ_DIR%/%_PKGFILE%)
- set INCLUDE=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\include;%INCLUDE%
- set LIB=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\lib;%LIB%
- set PATH=%_PREREQ_ABSDIR%\%_PKGLOCALDIR%\%_PKGZIPDIR%\bin;%PATH%

# Examine files in OS-level prereq subtree
- find %_PREREQ_DIR% -type f

# Examine the final environment
- 'echo "%PATH%"'
- 'echo "%PYTHONPATH%"'
Expand Down Expand Up @@ -299,11 +159,14 @@ install:
- where find
- find --version

# Verify that the commands used in the pywbem testsuite are available
- where xmllint
- xmllint --version

# Not a C# project, build stuff at the test step instead.
build: false

before_test:

test_script:
- tox -e pywin

167 changes: 21 additions & 146 deletions docs/development.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,125 +73,22 @@ OS-X should work as well, but Windows requires a number of manual setup steps.
Manual setup on Windows
^^^^^^^^^^^^^^^^^^^^^^^

The development environment needs the ``xmllint`` utility and the ``lxml``
Python package. To get ``xmllint`` and to prepare the environment for
installation of the ``lxml`` package, some packages need to be installed.

Some of the steps described below depend on the bit size of the active Python
environment. You can detect that bit size as follows:

.. code-block:: text
> python -c "import ctypes; print(ctypes.sizeof(ctypes.c_void_p)*8)"
64
Perform these steps (in the plain Windows Command Prompt):

* Set an environment variable to the absolute (!) path of an installation
directory for these packages:

.. code-block:: text
> set pkgdir=C:\....\packages
* Install the libxml2 package:

For a 32-bit Python environment:

.. code-block:: text
> set pkg=libxml2-2.7.8.win32
> wget ftp://ftp.zlatkovic.com/libxml/%pkg%.zip
> unzip -d %pkgdir% %pkg%.zip
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
For a 64-bit Python environment:

.. code-block:: text
> set pkg=libxml2-2.7.8.win32
> wget ftp://ftp.zlatkovic.com/libxml/64bit/%pkg%.7z
> 7z x -o%pkgdir%\%pkg% %pkg%.7z
> set INCLUDE=%pkgdir%\%pkg%\include\libxml2;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
* Install the libxslt package:

For a 32-bit Python environment:

.. code-block:: text
For development of pywbem, it is recommended to use one of the Unix-like
environments on Windows (such as CygWin, MinGW, Babun, or Gow). The pywbem
tests that run on the Appveyor CI use CygWin.

> set pkg=libxslt-1.1.26.win32
> wget ftp://ftp.zlatkovic.com/libxml/%pkg%.zip
> unzip -d %pkgdir% %pkg%.zip
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
Note that Unix-like environments on Windows bring their own Python, so double
check that the active Python environment is the one you want to install to.

For a 64-bit Python environment:

.. code-block:: text
> set pkg=libxslt-1.1.28-win32-x86_64
> wget ftp://ftp.zlatkovic.com/libxml/64bit/%pkg%.7z
> 7z x -o%pkgdir%\%pkg% %pkg%.7z
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
* Install the zlib package:

For a 32-bit Python environment:

.. code-block:: text
> set pkg=zlib-1.2.5.win32
> wget ftp://ftp.zlatkovic.com/libxml/%pkg%.zip
> unzip -d %pkgdir% %pkg%.zip
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
For a 64-bit Python environment:

.. code-block:: text
> set pkg=zlib-1.2.8-win32-x86_64
> wget ftp://ftp.zlatkovic.com/libxml/64bit/%pkg%.7z
> 7z x -o%pkgdir%\%pkg% %pkg%.7z
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
* Install the iconv package:

For a 32-bit Python environment:

.. code-block:: text
> set pkg=iconv-1.9.2.win32
> wget ftp://ftp.zlatkovic.com/libxml/%pkg%.zip
> unzip -d %pkgdir% %pkg%.zip
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
For a 64-bit Python environment:

.. code-block:: text
The development environment needs the ``xmllint`` utility and the ``lxml``
Python package.

> set pkg=iconv-1.14-win32-x86_64
> wget ftp://ftp.zlatkovic.com/libxml/64bit/%pkg%.7z
> 7z x -o%pkg% %pkg%.7z
> set INCLUDE=%pkgdir%\%pkg%\include;%INCLUDE%
> set LIB=%pkgdir%\%pkg%\lib;%LIB%
> set PATH=%pkgdir%\%pkg%\bin;%PATH%
The ``lxml`` Python package for Windows meanwhile comes as a binary wheel
archive that includes all of its dependencies, so it has no additional
dependencies you would need to take care about.

Collect all the ``set`` commands shown above in a batch script for use when
you re-open the command prompt in the future.
The ``xmllint`` utility is part of CygWin and likely also part of the other
Unix-like environments.


.. _`Prerequisite operating system packages for development`:
Expand All @@ -213,43 +110,21 @@ of the steps described in :ref:`Setting up the development environment`.
+--------------------------+--------------------+----------------------+-------+
| Op.system / Distribution | Package name | Version requirements | Notes |
+==========================+====================+======================+=======+
| Linux RedHat family | libxml2 | >=2.7.0, !=2.7.4, | |
| (RHEL, CentOS, Fedora) | | !=2.7.5, !=2.7.6 | |
| +--------------------+----------------------+-------+
| | libxslt | >=1.1.23, !=1.1.25 | |
| Linux RedHat family | | | |
| (RHEL, CentOS, Fedora) | | | |
+--------------------------+--------------------+----------------------+-------+
| Linux Debian family | libxml2-utils | >=2.7.0, !=2.7.4, | |
| (Ubuntu, Debian, | | !=2.7.5, !=2.7.6 | |
| LinuxMint) +--------------------+----------------------+-------+
| | libxml2-dev | >=2.7.0, !=2.7.4, | |
| | | !=2.7.5, !=2.7.6 | |
| +--------------------+----------------------+-------+
| | libxslt-dev | >=1.1.23, !=1.1.25 | |
| Linux Debian family | | | |
| (Ubuntu, Debian, | | | |
| LinuxMint) | | | |
+--------------------------+--------------------+----------------------+-------+
| Linux SUSE family | libxml2 | >=2.7.0, !=2.7.4, | |
| (SLES, openSUSE) | | !=2.7.5, !=2.7.6 | |
| +--------------------+----------------------+-------+
| | libxslt | >=1.1.23, !=1.1.25 | |
| Linux SUSE family | | | |
| (SLES, openSUSE) | | | |
+--------------------------+--------------------+----------------------+-------+
| OS-X | libxml2 | >=2.7.0, !=2.7.4, | |
| | | !=2.7.5, !=2.7.6 | |
| +--------------------+----------------------+-------+
| | libxslt | >=1.1.23, !=1.1.25 | |
| OS-X | | | |
+--------------------------+--------------------+----------------------+-------+
| Windows | libxml2 | >=2.7.0, !=2.7.4, | |
| | | !=2.7.5, !=2.7.6 | |
| +--------------------+----------------------+-------+
| | libxslt | >=1.1.23, !=1.1.25 | |
| +--------------------+----------------------+-------+
| | iconv | >=1.9.2 | win |
| +--------------------+----------------------+-------+
| | zlib | >=1.2.8 | win |
| Windows | xmllint | | |
+--------------------------+--------------------+----------------------+-------+

Notes:

* win: Only needed for Windows. On Linux and OS-X, the functionality provided
by these packages is part of glibc.

.. _`Building the documentation`:

Expand Down

0 comments on commit de1c4be

Please sign in to comment.