Skip to content

Incorrect insertion of align_to_byte() / alignToByte() calls #1070

Closed

Description

@generalmimon

I think it's safe to say that the biggest obstacle to using bit-sized integers is the incorrect insertion of align_to_byte() calls by the compiler. Instances of this problem have been reported dozens of times before, but none of the existing issues felt general enough to make it canonical, which is why I'm opening a new one. The following issues with checkboxes should be closed as duplicates of this issue:


  • Alignment options #12

    This one shouldn't be closed because it's in fact a broader issue, but people have also reported instances of the improper align_to_byte() insertion several times there, so I'm including it in this list.

  • un-aligned, bit-based data stream #576

    This is a broader issue as well - although the issue with align_to_byte() insertion is also mentioned there, it even proposes to make operations that are currently byte-aligned by design (u{X} and s{X} integers, floats, byte arrays) non-byte aligned, so technically it's not a pure duplicate.

The problem is that the align_to_byte() calls are inserted by the compiler, and its logic for deciding when to do this is very crude. Mainly:

  • it treats type switches and user-defined types as byte-aligned (which is certainly undesirable - I don't see any reason why you shouldn't be able to use a type switching or cross a type boundary at any position where the format you're describing requires it)
  • doesn't consider ifs, repeats etc.

Activity

  1. generalmimon commented on Oct 13, 2023

    @generalmimon
    MemberAuthor

    On the one hand, you could say that the compiler implementation is very incomplete and you'd be right. On the other hand, I've realized over time that it would in fact be far from easy to make the compiler behave properly in all cases, because there are a lot of situations that you have to handle specifically, for example (#743 (comment)):

    seq:
      - id: a
        type:
          switch-on: _index
          cases:
            0: b2
            1: u1
        repeat: expr
        repeat-expr: 2

    This is effectively a b2 - u1 sequence at runtime, so you have to insert an align_to_byte() call between the two types. But from the perspective of implementing the alignment in the compiler, this case is quite unusual - usually you need 2 consecutive attributes to require the alignment, but here it's within the same attribute due to repetition. Of course, the above situation can be equivalently implemented via a user type with if-guarded fields, for example:

    seq:
      - id: a
        type: item(_index)
        repeat: expr
        repeat-expr: 2
    types:
      item:
        params:
          - id: i
            type: s4
        seq:
          - id: byte_val
            type: u1
            if: _index == 1
          - id: bit_val
            type: b2
            if: _index == 0

    (I've intentionally defined byte_val before bit_val, because it doesn't matter given that only one is active at a time.)

    This has the same effect in practice, but for the compiler this is a very different case, and this situation is just one of many. Somehow the compiler would have to see through all of this, which definitely sounds like an awful lot of work.


    Instead, the approach I've already taken in Java and Python (Java: kaitai-io/kaitai_struct_java_runtime@1bc75aa, Python: kaitai-io/kaitai_struct_python_runtime@1cb84b8) is to handle all bit-byte alignment in the runtime library. It turned out that doing this at runtime is simply at least 20x easier than at compile time - and more reliable too. Consequently, it also makes the runtime library more suitable for standalone use, because now a call sequence like read_bits_int_be(3) - read_bytes(1) - read_bits_int_be(3) is not an undefined behavior anymore (due to the missing align_to_byte() call between the first and the second call), but will actually work as expected, issuing an alignment between the first two operations automatically.

    The only potential disadvantage of handling this at runtime is, obviously, some runtime performance penalty (because essentially all byte-aligned reads call align_to_byte() now). I haven't done any benchmarks, but I don't think it will be significant in most cases because the align_to_byte() is very simple and I believe it can be well optimized (inlined etc.) by the compilers. And we in fact need it for correctness, so I think it's worth it.

    Now, all that's left to do is to port the existing scheme from Java and Python runtimes to all the other languages and all align_to_byte() problems will hopefully be gone for good.

  2. changed the title [-]Incorrect insertion of `align_to_byte()` calls[/-] [+]Incorrect insertion of `align_to_byte()` / `alignToByte()` calls[/+] on Apr 4, 2024
  3. sakethsusarla commented on Dec 3, 2024

    @sakethsusarla

    Has anyone found out a solution or workaround to this issue?

  4. added this to the v0.11 milestone on Mar 6, 2025
  5. modified the milestones: v0.11, v0.12 on Jul 30, 2025
  6. generalmimon commented on Dec 23, 2025

    @generalmimon
    MemberAuthor

    To be clear: this issue is fixed in 0.11 for Java and Python, and now also in the master branches for the rest of the languages. So if you're using KS 0.11 and targeting some other language than Java or Python, you're still affected by this. Since the Web IDE runs on JavaScript, the stable version at https://ide.kaitai.io/ that matches KS 0.11 is affected by this bug, but the development version at https://ide.kaitai.io/devel/ is not, as it follows the master branches.

    If you are affected, it means that a boundary between a primitive bit-sized integer type (e.g. b4) and a user-defined type that also starts with a bit-sized integer type, like between foo and bar in the following example:

    seq:
      - id: foo
        type: b4
      - id: bar
        type: b4_wrapper
    types:
      b4_wrapper:
        seq:
          - id: value
            type: b4 

    ... is incorrectly treated by KSC 0.11 as the place where to insert an alignToByte() call, so b4_wrapper will be aligned to the next byte boundary. As a result (assuming the top-level seq starts at the beginning of the stream), foo parses the first 4 bits of the first byte, then 4 bits are skipped (this is what alignToByte() does) and bar.value parses the first 4 bits of the second byte.

    As hinted in #1070 (comment), entering a user-defined type is not the only kind of boundary which the compiler recognizes as requiring byte alignment. Any attribute with type switching is also considered byte-aligned (regardless of whether that makes any sense), so you get a call to alignToByte() between foo and bar even in the following case:

    seq:
      - id: foo
        type: b4
      - id: bar
        type:
          switch-on: foo
          cases:
            _: b4

    There is a universal workaround: avoid these pairs of a primitive bit-sized integer type type: bX immediately followed by something considered byte aligned (i.e. a user type or type switching) back-to-back in the same seq, because that is where the compiler will insert an alignment operation. You can do this by wrapping primitive bit integers into a user type as well. In the first example, it would be:

     seq:
       - id: foo
    -    type: b4
    +    type: b4_wrapper
       - id: bar
         type: b4_wrapper
     types:
       b4_wrapper:
         seq:
           - id: value
             type: b4 

    It will work in the second example too:

     seq:
       - id: foo
    -    type: b4
    +    type: b4_wrapper
       - id: bar
         type:
           switch-on: foo
           cases:
             _: b4

    If you don't like the fact that the bit integers are wrapped in a user-defined type when using this workaround (so for example for the b4_wrapper type, you have to access the actual value via .value), you can "wrap" the bit-sized integer type into a type switching instead:

      - id: bar
        type:
          switch-on: 0
          cases:
            _: b4

    Of course, as explained at the beginning of this comment, these are really just workarounds for something that has already been fixed, but these fixes did not make it into the latest version 0.11 in all languages. So it's not really recommended to use these things unless you have to. If you just want your .ksy specs to work in the Web IDE, I would say it's easier to use https://ide.kaitai.io/devel/ instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions