Skip to content

Latest commit

 

History

History
88 lines (71 loc) · 4.58 KB

ply-handling.md

File metadata and controls

88 lines (71 loc) · 4.58 KB
Raw

Problem with LEX/YACC table modules generated by ply

The LEX/YACC table modules are generated by the ply package and contain the version of the ply package that generated them. These modules are packaged into the pywbem distribution archive and are up to date w.r.t. the ply package on the build system.

However, the target Python environment may have a different version of ply installed. Any use of ply of these table modules will re-generate them automatically in that case. There are also more subtle reasons for re-generation that are not fully understood at this point.

The automatic re-generation of the table modules creates two potential issues:

  • The target Python environment may be a system Python, and the user that uses the package may not have permission to update the table modules.

  • The re-generation produces some log messages on stdout, and depending on the circumstances this may be a problem.

Our strategy is therefore to ensure that the table modules are up to date in the target Python environment, so that they will not need to be re-generated. That is a solution for many but not for all situations, because a user with permissions could update the ply package and lateron a user of pywbem without permissions would trigger a re-generation that fails.

The following pywbem installation tools need to be supported:

  • pip install, with any package source (local dist archive, or PyPI).
  • python setup.py install, from the repo work directory.

At this point, we did not find a way to build the table modules automatically in both types of installation. Part of the problem is that setuptools and pip do the installation quite differently.

Generally, the approach can be to update the table modules in the Python source files that are used as input to the installation process, or to update them after the installation in the place where they have been installed to.

Updating them in the input files is the proper approach considering that there are post-processing steps such as pre-compiling the files into Python byte code. However, at such an early point in the installation, the dependencies of the pywbem package (e.g. ply itself) have not been installed (e.g. if the environment is fresh), and they are needed to build the table modules.

The dependency problem is the reason to choose the approach of updating the installed modules. That is possible with python setup.py install in a number of ways, but so far we did not find a way to do that for pip install, because every hook that is possible to use is before pip finally copies the files into the target place where they would be found in the Python module search path.

Just for the record, here is what was tried:

  • Use the setup_requires keyword of setup() in order to establish the prerequisites for building the table modules, and hook up in the install process (e.g. custom install command, or somewhere else). This runs into the problem that the dependencies specified with setup_requires get installed into a local .eggs directory, and even when they are again specified in install_requires their 'real' installation is skipped because they are already found loaded. See setuptools issue 391. Conclusion is that setup_requires cannot be used at all, at the moment.

  • Have a custom install command and invoke this script after calling the original function. The install command is invoked by both setup.py and pip. This runs into the problem that the original function (install.run() in setuptools.command.install) looks at the call stack and because of the presence of the custom install hook, takes a different path that leads to skipping the installation of dependencies specified in install_requires. See setuptools issue 456 Conclusion is that a custom install command cannot be used at all, at the moment.

  • Invoke this script in the setup.py script after returning from setup(). This script must be run in a child Python in order to pick up the possibly changed module search path resulting from installing eggs. For setup.py this works, but has the drawback that it is not clear how the setup.py command can be figured out, so it is done for every command. For pip, this does not work because even after returning from setup(), the pywbem module files are not yet in their final destination.

  • Some more variations of custom commands, but they were either too early, or not used by both setup.py and pip.