From eabd27df02c7c5c25f6e7f64c7a170c6dda178ed Mon Sep 17 00:00:00 2001 From: KOLANICH Date: Fri, 31 Jan 2020 02:55:52 +0300 Subject: [PATCH] Python packaging-related fixes: * improved the docs on installing python runtime * fixed the links to kaitaiStructCompile.setuptools * added more examples of usage of kaitaiStructCompile.setuptools --- lang_python.adoc | 97 +++++++++++++++++++++++++++++------------------- 1 file changed, 58 insertions(+), 39 deletions(-) diff --git a/lang_python.adoc b/lang_python.adoc index 0c48b37..c23592d 100644 --- a/lang_python.adoc +++ b/lang_python.adoc @@ -17,9 +17,15 @@ To generate Python modules, Kaitai Struct compiler should be invoked with kaitai-struct-compiler -t python my_spec.ksy This normally generates `my_spec.py` parser module, which is compatible -with both Python 2 and 3. +with both Python 2 and 3. It is strongly recommended to use python 3 +because python 2 is EOL since 2020-01-01. -TODO: document `--python-package` +`--python-package` allows you to select a parent package name used by +one generated file to import symbols from another one. If you want to +embed KSC-generated files into an own package, it is recommended to +set it to `.`, which means "current package" rather than to the real name +of the subpackage, because setting it to a real name will cause breakage +when you want to reorganize or rename your package. [[add-runtime]] === Adding runtime library @@ -32,6 +38,8 @@ GitHub]). Runtime library API version must match compiler version that you're using (i.e. if you use ksc v0.8, then you must use compatible v0.8 runtime). +If you are using non-release KSC, you should also use non-release runtime. + [[add-runtime-release]] ==== Using release version @@ -40,72 +48,80 @@ methods: * On Debian / Ubuntu, one can install it from apt repositories: -** For Python 2.x: +** For Python 3.x: + [source,shell] -sudo apt-get install python-kaitaistruct +sudo apt-get install -y python3-kaitaistruct + -Reference: https://packages.debian.org/buster/python-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/testing/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_testing/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/eoan/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_19_10/python:python-kaitaistruct.svg[package info for Ubuntu]] -** For Python 3.x: +** For Python 2.x: + [source,shell] -sudo apt-get install python3-kaitaistruct +sudo apt-get install -y python-kaitaistruct + -Reference: https://packages.debian.org/buster/python3-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python3-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/stable/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_stable/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/eoan/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_19_10/python:python-kaitaistruct.svg[package info for Ubuntu]] + -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: +* Add the following dependency specifier into the package configuration +of the packaging tool you use: + ---- kaitaistruct ---- + -and then run `pip install -r requirements.txt` to update all your -dependencies. +and then run `python3 -m pip install --upgrade -e ./` in the dir with your project +in order to install your package and all its dependencies in +an editable way. If you don't want to package your software, +but only installbits dependencies, add `kaitaistruct` into `requirements.txt`. -* Otherwise, you can just install the package manually using *pip* +* Otherwise, you can just install the package manually using `pip` (Python package manager): + [source,shell] -pip install kaitaistruct +python3 -m pip install --upgrade kaitaistruct [[add-runtime-snapshot]] ==== Using latest (snapshot) version -If you're using latest (unreleased AKA "unstable" AKA "snapshot") build +If you're using latest (unreleased AKA "unstable" AKA "snapshot" AKA "nightly") build of Kaitai Struct compiler, that will require latest runtime libraries as well: -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: +* You can a make `pip` to use a bleeding edge version of the runtime by + adding a https://www.python.org/dev/peps/pep-0508/[PEP 508] + dependency specifier into the config of your favorite build tool: + ---- -kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git + kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git ---- + -and then run `pip install -r requirements.txt` to update all your -dependencies. +. * Otherwise, you can just install the package manually using *pip* (Python package manager): + [source,shell] -pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git +python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git + [[add-dependencies]] -=== Adding dependencies +=== Adding dependencies manually -<> implementation in Python requires a enum -support as described in +<> implementation in Python requires +a enum support as described in https://www.python.org/dev/peps/pep-0435/[PEP-0435]. This support is available out of the box in standard libraries since Python v3.4, or since v2.4 using https://pypi.python.org/pypi/enum34[enum34] PyPi compatibility layer. -* On Debian / Ubuntu, this library can be installed with `sudo apt-get - install python-enum34` -* Otherwise, one can use just `sudo pip install enum34` +* One usually doesn't need to install this lib manually, `pip` will install it + automatically if it is needed. +* One can also install it manually: +** One can use just `sudo python -m pip install enum34`. Don't do it on + later pythons or the apps using just `enum` will be broken! +** On Debian / Ubuntu, this library can be installed with + `sudo apt-get install -y python-enum34`. [[automation]] === Automation of compilation process @@ -114,30 +130,33 @@ https://setuptools.readthedocs.io/en/latest/[Setuptools] is the de-facto standard way of building and installing python packages. This library allows extension with https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins]. -https://github.com/KOLANICH have developed -https://github.com/KOLANICH/kaitaiStructCompile.py[an extension to +https://gitlab.com/KOLANICH has developed +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools[an extension to setuptools], allowing you to automate compilation of `*.ksy` files as the part of python module build process. The library provides a mostly declarative syntax to facilitate installation: you add a parameter `kaitai` to your `setup.py` where you enumerate files that you want compiled, output directory, compilation flags, and source code post-compilation transformations. Here are some examples: -https://github.com/KOLANICH/NTMDTRead/blob/master/setup.py[1], -https://github.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2]. +https://gitlab.com/KOLANICH/NTMDTRead/blob/master/setup.py[1], +https://gitlab.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2], +https://gitlab.com/KOLANICH/FrozenTable.py/blob/master/setup.py[3], +https://gitlab.com/KOLANICH/MFCTool.py/blob/master/setup.py[4], +https://gitlab.com/KOLANICH/RDataParser.py/blob/master/setup.py[5]. Here are -https://github.com/KOLANICH/kaitaiStructCompile.py/blob/master/kaitaiStructCompile/config.schema.json[the -docs] in JSON Schema format. +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.schemas/blob/master/kaitaiStructCompile/schemas/schemas/config.schema.json[the +docs] in https://json-schema.org/specification.html[JSON Schema] format. Please remember that this lib is VERY UNSTABLE and very likely to be improved in a way that changes API (but due to API simplicity it'd be easy to fix your code), but I guess it's much better to use this than to reinvent the wheel. -https://github.com/KOLANICH/kaitaiStructCompile.py/issues[Contributions +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.py/issues[Contributions are welcome!] -Please note that this lib is NOT available on -https://pypi.python.org[pypi] and if you want to install it -automatically, you should add -`git+https://github.com/KOLANICH/kaitaiStructCompile.py` into -`dependency_links` in the config passed to `setuptools.setup`. +Please note that this lib is NOT available on https://pypi.python.org[pypi] +and if you want to install it automatically, you should add +`kaitaiStructCompile.setuptools @ https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools` +into `setup_requires` into `options` section of your `setup.cfg` and into +`requires` of `build_system` section of your `pyproject.toml`.