Skip to content

Commit

Permalink
Python packaging-related fixes:
Browse files Browse the repository at this point in the history
* improved the docs on installing python runtime
* fixed the links to kaitaiStructCompile.setuptools
* added more examples of usage of kaitaiStructCompile.setuptools
  • Loading branch information
KOLANICH committed Feb 2, 2020
1 parent 925fdab commit eabd27d
Showing 1 changed file with 58 additions and 39 deletions.
97 changes: 58 additions & 39 deletions lang_python.adoc
Expand Up @@ -17,9 +17,15 @@ To generate Python modules, Kaitai Struct compiler should be invoked with
kaitai-struct-compiler -t python my_spec.ksy

This normally generates `my_spec.py` parser module, which is compatible
with both Python 2 and 3.
with both Python 2 and 3. It is strongly recommended to use python 3
because python 2 is EOL since 2020-01-01.

TODO: document `--python-package`
`--python-package` allows you to select a parent package name used by
one generated file to import symbols from another one. If you want to
embed KSC-generated files into an own package, it is recommended to
set it to `.`, which means "current package" rather than to the real name
of the subpackage, because setting it to a real name will cause breakage
when you want to reorganize or rename your package.

[[add-runtime]]
=== Adding runtime library
Expand All @@ -32,6 +38,8 @@ GitHub]).
Runtime library API version must match compiler version that you're using
(i.e. if you use ksc v0.8, then you must use compatible v0.8 runtime).

If you are using non-release KSC, you should also use non-release runtime.

[[add-runtime-release]]
==== Using release version

Expand All @@ -40,72 +48,80 @@ methods:

* On Debian / Ubuntu, one can install it from apt repositories:

** For Python 2.x:
** For Python 3.x:
+
[source,shell]
sudo apt-get install python-kaitaistruct
sudo apt-get install -y python3-kaitaistruct
+
Reference: https://packages.debian.org/buster/python-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python-kaitaistruct[package info for Ubuntu]
Reference: https://packages.debian.org/testing/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_testing/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/eoan/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_19_10/python:python-kaitaistruct.svg[package info for Ubuntu]]

** For Python 3.x:
** For Python 2.x:
+
[source,shell]
sudo apt-get install python3-kaitaistruct
sudo apt-get install -y python-kaitaistruct
+
Reference: https://packages.debian.org/buster/python3-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python3-kaitaistruct[package info for Ubuntu]
Reference: https://packages.debian.org/stable/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_stable/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/eoan/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_19_10/python:python-kaitaistruct.svg[package info for Ubuntu]]


* If your project is using *requirements.txt* to manage your
dependencies, add the following line to it:
* Add the following dependency specifier into the package configuration
of the packaging tool you use:
+
----
kaitaistruct
----
+
and then run `pip install -r requirements.txt` to update all your
dependencies.
and then run `python3 -m pip install --upgrade -e ./` in the dir with your project
in order to install your package and all its dependencies in
an editable way. If you don't want to package your software,
but only installbits dependencies, add `kaitaistruct` into `requirements.txt`.

* Otherwise, you can just install the package manually using *pip*
* Otherwise, you can just install the package manually using `pip`
(Python package manager):
+
[source,shell]
pip install kaitaistruct
python3 -m pip install --upgrade kaitaistruct

[[add-runtime-snapshot]]
==== Using latest (snapshot) version

If you're using latest (unreleased AKA "unstable" AKA "snapshot") build
If you're using latest (unreleased AKA "unstable" AKA "snapshot" AKA "nightly") build
of Kaitai Struct compiler, that will require latest runtime libraries as
well:

* If your project is using *requirements.txt* to manage your
dependencies, add the following line to it:
* You can a make `pip` to use a bleeding edge version of the runtime by
adding a https://www.python.org/dev/peps/pep-0508/[PEP 508]
dependency specifier into the config of your favorite build tool:
+
----
kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
----
+
and then run `pip install -r requirements.txt` to update all your
dependencies.
.

* Otherwise, you can just install the package manually using *pip*
(Python package manager):
+
[source,shell]
pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git


[[add-dependencies]]
=== Adding dependencies
=== Adding dependencies manually

<<ksy_reference#enums,Enums>> implementation in Python requires a enum
support as described in
<<ksy_reference#enums,Enums>> implementation in Python requires
a enum support as described in
https://www.python.org/dev/peps/pep-0435/[PEP-0435]. This support is
available out of the box in standard libraries since Python v3.4, or
since v2.4 using https://pypi.python.org/pypi/enum34[enum34] PyPi
compatibility layer.

* On Debian / Ubuntu, this library can be installed with `sudo apt-get
install python-enum34`
* Otherwise, one can use just `sudo pip install enum34`
* One usually doesn't need to install this lib manually, `pip` will install it
automatically if it is needed.
* One can also install it manually:
** One can use just `sudo python -m pip install enum34`. Don't do it on
later pythons or the apps using just `enum` will be broken!
** On Debian / Ubuntu, this library can be installed with
`sudo apt-get install -y python-enum34`.

[[automation]]
=== Automation of compilation process
Expand All @@ -114,30 +130,33 @@ https://setuptools.readthedocs.io/en/latest/[Setuptools] is the de-facto
standard way of building and installing python packages. This library
allows extension with
https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins].
https://github.com/KOLANICH have developed
https://github.com/KOLANICH/kaitaiStructCompile.py[an extension to
https://gitlab.com/KOLANICH has developed
https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools[an extension to
setuptools], allowing you to automate compilation of `*.ksy` files as the
part of python module build process. The library provides a mostly
declarative syntax to facilitate installation: you add a parameter
`kaitai` to your `setup.py` where you enumerate files that you want
compiled, output directory, compilation flags, and source code
post-compilation transformations. Here are some examples:
https://github.com/KOLANICH/NTMDTRead/blob/master/setup.py[1],
https://github.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2].
https://gitlab.com/KOLANICH/NTMDTRead/blob/master/setup.py[1],
https://gitlab.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2],
https://gitlab.com/KOLANICH/FrozenTable.py/blob/master/setup.py[3],
https://gitlab.com/KOLANICH/MFCTool.py/blob/master/setup.py[4],
https://gitlab.com/KOLANICH/RDataParser.py/blob/master/setup.py[5].

Here are
https://github.com/KOLANICH/kaitaiStructCompile.py/blob/master/kaitaiStructCompile/config.schema.json[the
docs] in JSON Schema format.
https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.schemas/blob/master/kaitaiStructCompile/schemas/schemas/config.schema.json[the
docs] in https://json-schema.org/specification.html[JSON Schema] format.

Please remember that this lib is VERY UNSTABLE and very likely to be
improved in a way that changes API (but due to API simplicity it'd be
easy to fix your code), but I guess it's much better to use this than to
reinvent the wheel.
https://github.com/KOLANICH/kaitaiStructCompile.py/issues[Contributions
https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.py/issues[Contributions
are welcome!]

Please note that this lib is NOT available on
https://pypi.python.org[pypi] and if you want to install it
automatically, you should add
`git+https://github.com/KOLANICH/kaitaiStructCompile.py` into
`dependency_links` in the config passed to `setuptools.setup`.
Please note that this lib is NOT available on https://pypi.python.org[pypi]
and if you want to install it automatically, you should add
`kaitaiStructCompile.setuptools @ https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools`
into `setup_requires` into `options` section of your `setup.cfg` and into
`requires` of `build_system` section of your `pyproject.toml`.

0 comments on commit eabd27d

Please sign in to comment.