Python runtime version check is slow

[dgelessus]

The code generated by ksc to check the version of the Python runtime is quite slow. To measure this, I compiled the spec common/bytes_with_io.ksy, which parses the entire input as a single byte array, so there's practically no overhead from type definitions. The current ksc 0.9 snapshot (ce18dc1, kaitai-io/kaitai_struct_compiler@f725faf) generates the following Python code:

# This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

from pkg_resources import parse_version
import kaitaistruct
from kaitaistruct import KaitaiStruct, KaitaiStream, BytesIO


if parse_version(kaitaistruct.__version__) < parse_version('0.9'):
    raise Exception("Incompatible Kaitai Struct Python API: 0.9 or later is required, but you have %s" % (kaitaistruct.__version__))

class BytesWithIo(KaitaiStruct):
    """Helper type to work around Kaitai Struct not providing an `_io` member for plain byte arrays.
    """
    def __init__(self, _io, _parent=None, _root=None):
        self._io = _io
        self._parent = _parent
        self._root = _root if _root else self
        self._read()

    def _read(self):
        self.data = self._io.read_bytes_full()

With this code, running python3 -c 'import bytes_with_io' takes about 260 ms (on average, on my machine). For comparison, if I remove the version check and related import:

--- bytes_with_io.py
+++ bytes_with_io.py
@@ -1,13 +1,9 @@
 # This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild
 
-from pkg_resources import parse_version
 import kaitaistruct
 from kaitaistruct import KaitaiStruct, KaitaiStream, BytesIO
 
 
-if parse_version(kaitaistruct.__version__) < parse_version('0.9'):
-    raise Exception("Incompatible Kaitai Struct Python API: 0.9 or later is required, but you have %s" % (kaitaistruct.__version__))
-
 class BytesWithIo(KaitaiStruct):
     """Helper type to work around Kaitai Struct not providing an `_io` member for plain byte arrays.
     """

I get times of around 40 ms. Starting Python and running nothing takes 38 ms, which means that the KS code (runtime and generated code) only takes 2 ms, and import pkg_resources and the version check takes 220 ms! (Practically all of this time comes from import pkg_resources, not from the actual version check code.) This is a noticeable delay, especially for small scripts that don't process a lot of data, which often take less than 200 ms to execute (without the version check). So I experimented with some possible solutions for reducing the time needed for the version check:

• Removing the version check entirely. This is obviously not a good solution, but it is very fast. 😛
• Parsing __version__ using packaging.version.Version. The behavior is identical to pkg_resources.parse_version, because parse_version actually uses packaging.version internally. The disadvantage is that it requires a dependency on the packaging library. (pkg_resources has its own internal copy of packaging, so even if pkg_resources is available, import packaging might not work.) This version check takes about 20 ms (the entire command takes 60 ms). Just noticed that this isn't actually an option for us. Even if we make the next version of kaitaistruct depend on packaging, older versions won't have that dependency, so packaging might not be installed when using an older runtime and the check wouldn't work.
• Parsing __version__ by hand into a tuple of ints and comparing that (something like tuple(int(part) for part in kaitaistruct.__version__.split(".")) < (0, 9)). This requires no external dependencies, but can only handle simple version strings like 0.9.1 - something like 0.9.1.dev, which is allowed in Python version strings, won't work. This version check has no noticeable performance impact (the entire command takes 40 ms).
• Adding a new kaitaistruct.__version_info__ attribute that stores the version number as a tuple of ints, which can be compared. This is similar to the previous solution, except that it doesn't require any parsing in the generated code, and doesn't need to handle non-numeric version parts (those parts can be left out of __version_info__). The disadvantage is that it requires a change to the kaitaistruct runtime. This is not a big issue though - as long as __version__ stays, older version checks will continue to work, and new version checks need to handle old runtimes that don't have __version_info__ (something like getattr(kaitaistruct, "__version_info__", (0, 8)) < (0, 9)). This version check has no noticeable performance impact (the entire command takes 40 ms).

I personally prefer the last solution, because it's the simplest, has the least runtime overhead, and doesn't require any external dependencies. But if anyone has arguments for the other solutions, or a better idea that I haven't thought of, let me know.

(based on these Gitter messages by myself: https://gitter.im/kaitai_struct/Lobby?at=5f372b993e6ff00c28939f29)

[KOLANICH]

pkg_resources

pkg_resources is slow because ... it walks all the dirs with metadata in site-packages.

A common way to speed-up is to use version.py. We can move kaitaistruct.py to kaitaistruct/__init__.py and put version into kaitaistruct/version.py. Then content of version.py can be generated using setuptools_scm.

[webbnh]

In the event that KOLANICH's observation bears no fruit....

Parsing version by hand into a tuple of ints and comparing that (something like tuple(int(part) for part in kaitaistruct.version.split(".")) < (0, 9)). This requires no external dependencies, but can only handle simple version strings like 0.9.1 - something like 0.9.1.dev, which is allowed in Python version strings, won't work.

This seems like a very good approach to me -- all that's missing is a custom comparison function which compares the numbers as numbers and does something intelligent if it encounters string components or suffixes.

This saves you from having add an attribute and having to change the runtime to support both the "current" approach and the "legacy" approach.

[dgelessus]

pkg_resources is slow because ... it walks all the dirs with metadata in site-packages.

... does it do that at import time?

To clarify, the ksc-generated version check uses pkg_resources only for the parse_version function. It doesn't read the version information from the package metadata (using get_distribution). The version string is obtained through a direct attribute access (kaitaistruct.__version__), which is going to be fast no matter where you put the version attribute or how you generate it. Parsing the version is also very fast:

$ python3 -m timeit --setup='import pkg_resources' -- 'pkg_resources.parse_version("0.9")'
20000 loops, best of 5: 15.2 usec per loop

The performance problem here is importing pkg_resources:

$ for i in 1 2 3 4 5; do python3 -m timeit --number=1 --repeat=1 -- 'import pkg_resources'; done
1 loop, best of 1: 179 msec per loop
1 loop, best of 1: 209 msec per loop
1 loop, best of 1: 227 msec per loop
1 loop, best of 1: 185 msec per loop
1 loop, best of 1: 199 msec per loop

(imports get cached by the Python interpreter, so to see the actual import time I have to start a new Python interpreter each time and can't use timeit's built in looping)

[dgelessus]

all that's missing is a custom comparison function which compares the numbers as numbers and does something intelligent if it encounters string components or suffixes.

The problem is that the version check/comparison code is placed directly in the source code of each generated Python file, so it should be kept short/compact - there certainly shouldn't be any long/complex function definitions. Also, if there's a bug in the generated version comparison code, the only way to fix it is to update the compiler and ask users to regenerate all their parsers (or patch the generated code by hand).

This is of course why normally you don't implement this kind of logic by hand and instead use a library. The problem is that we can't put the relevant code into the kaitaistruct module (because we can't assume that it has the code until we've checked the version) and the standard library solution is quite slow.

[webbnh]

@dgelessus, you're quite right. (Although, I'm not sure that expecting people to update the compiler and runtime together is unreasonable.) And, I concur that having the comparison function, if it is non-trivial, placed in the runtime would be better. But, this basic approach has enough advantages to make it worth considering.

I imagine that, if we wanted to, we could come up with some clever ways for the generated code to interrogate the features of the runtime. For instance, I suspect that it could check if the runtime has a "check version function", and, if it does not, it could fall back to the present mechanism.

However, if you're willing to permit a dependency on Python regular expressions, this might be sufficient:

tuple((int(part) if re.match("[0-9]+", part) else part) for part in kaitaistruct.__version__.split(".")) < (0, 9)

I think that covers all but the most obscure of cases. (I.e., it doesn't cover "post release" numbers, and it may get "release candidate" vs. "development" releases wrong, but otherwise I think it covers the first nine "alpha", "beta", and "release candidate" versions correctly, in addition to all the normal cases.)

[KOLANICH]

... does it do that at import time?

It seems that yes, though from the code it is not obvious.

_initialize_master_working_set -> _build_master -> require -> Environment -> scan -> find_distributions -> _find_adapter returns find_on_path -> find_on_path -> safe_listdir -> os.listdir

[KOLANICH]

tuple((int(part) if re.match("[0-9]+", part) else part) for part in kaitaistruct.__version__.split(".")) < (0, 9)

IMHO an overkill to use re. instead string.digits can be used.

And BTW: https://github.com/KOLANICH/AnyVer.py

[webbnh]

Yes! Much better:

tuple((int(part) if str.isdigit(part) else part) for part in ver.split(".")) < (0, 9, 'debug')

[dgelessus]

(Although, I'm not sure that expecting people to update the compiler and runtime together is unreasonable.)

The problem is that the version check can't assume that the runtime is up-to-date - its entire purpose is to check for outdated runtime versions and give a user-friendly error message in that case.

I imagine that, if we wanted to, we could come up with some clever ways for the generated code to interrogate the features of the runtime. For instance, I suspect that it could check if the runtime has a "check version function", and, if it does not, it could fall back to the present mechanism.

This is more or less what I was suggesting with my fourth solution :) If kaitaistruct.__version_info__ exists, the version check can compare against it. If it doesn't exist, we don't even need to fall back to a different version checking mechanism - we know that the runtime is older than the version that added __version_info__, and we know that the compiler is at least the version that added __version_info__ (otherwise it would be using the old version check), so we know for certain that the runtime is not compatible with the compiler.

On that topic - if we do use that solution, we should perhaps use a different name than __version_info__. I've seen some other libraries use __version_info__, and they include all parts of the version, even for pre-release versions - for example if the version is 1.2.3b4, the __version_info__ tuple would be (1, 2, 3, "beta", 4), not just (1, 2, 3) as I suggested. Maybe we should call it something like COMPATIBILITY_CHECK_VERSION to make the purpose clear.

(I.e., it doesn't cover "post release" numbers, and it may get "release candidate" vs. "development" releases wrong, but otherwise I think it covers the first nine "alpha", "beta", and "release candidate" versions correctly, in addition to all the normal cases.)

I don't think that's a good idea... IMHO either we should support pre/post-releases correctly, or we shouldn't support them at all and hard-error out if they are encountered in the version string. I don't think it's very useful to only handle some pre/post-release combinations correctly and return wrong results for other combinations.

Also, the suggested parsing code (both the re and isdigit versions) doesn't handle alpha, beta and release candidate versions correctly - their format is 1.2a3, 1.2b3, 1.2rc3 and not 1.2.a3, 1.2.b3, 1.2.rc3. So for example comparing 0.10b1 against 0.10 wouldn't work.

Alternatively we could decide that the Python runtime version should only ever use numbers and dots, but I think it's a bad idea to assume that - especially if at some point we decide to use a tool like setuptools_scm, which will generate pre/post-release version numbers in many cases.

(This is another reason why I suggested adding a new attribute for the compatibility check. That way we only need to keep that attribute in a format that is compatible with the version check, and the __version__ string can use all the fancy features allowed by the Python version format without breaking anything.)

And BTW: https://github.com/KOLANICH/AnyVer.py

The version check can't depend on any external libraries (see my correction in the top comment), and that module is definitely too long to be included directly in the generated code.

[KOLANICH]

The version check can't depend on any external libraries

I know that it is inacceptible to use a dependency (it can be vendored, but it is still too large for just a version check) for just a version check. In fact I have developed it for a package management solution. It was just mentioned to make sure at least someone heard about it :)

Also, the suggested parsing code (both the re and isdigit versions) doesn't handle alpha, beta and release candidate versions correctly - their format is 1.2a3, 1.2b3, 1.2rc3 and not 1.2.a3, 1.2.b3, 1.2.rc3. So for example comparing 0.10b1 against 0.10 wouldn't work.

hmm, good catch.

How about just cutting the part before the first non-digit non-dots, and then splitting it by dots?

[GreyCat]

As an off-though — any chance switching to something like what Perl does will help the performance in this case? I.e. instead of having weird version formats and non-trivial comparison algorithms, we can just formalize everything to a.b.c, serialize that with leading zeroes and compare it as number — plain and simple?

[generalmimon]

@GreyCat:

As an off-though — any chance switching to something like what Perl does will help the performance in this case? I.e. instead of having weird version formats and non-trivial comparison algorithms, we can just formalize everything to a.b.c, serialize that with leading zeroes and compare it as number — plain and simple?

I have a similar opinion - as @dgelessus pointed out, one can also compare tuples in Python, which seems to be a better alternative to an integer. AFAIK all languages other than Python store the runtime version in a single integer, so why would we need to introduce an algorithm for parsing and comparing versions stored in a string? What's the benefit of doing that?

I vote for this option:

• Adding a new kaitaistruct.__version_info__ attribute that stores the version number as a tuple of ints, which can be compared.

However, I'm quite stunned by what @dgelessus said about it:

• This version check has no noticeable performance impact (the entire command takes 40 ms).

How is it possible that a simple comparison of two tuples takes 40 ms? Can you please expand it and show the benchmark, @dgelessus?

If the problem is in tuples, does the substitution with plain integers help?

My common sense says that parsing version from a string must necessarily take much longer than comparing any two primitives in Python.

[dgelessus]

How is it possible that a simple comparison of two tuples takes 40 ms? Can you please expand it and show the benchmark, @dgelessus?

It doesn't, don't worry 🙂 I should have worded that more clearly - 40 ms isn't the execution time of the version check, but the execution time of the entire command (python3 -c 'import bytes_with_io'), which includes the Python interpreter startup (about 38 ms) and the rest of the kaitaistruct and bytes_with_io code. If I remove the version check, the same command still takes 40 ms, which is why I said that the tuple-based comparison has no noticeable performance impact.

PS: And yes, if Python didn't have built-in support for comparing tuples, I would have probably suggested a format based on a single number instead. But Python has proper tuple comparison that is also very fast, and with it you don't need to worry about adding leading/trailing zeroes in the right places.

[generalmimon]

which is why I said that the tuple-based comparison has no noticeable performance impact.

@dgelessus Oh, I'm sorry, I missed the little no there 😆