Check endianness of headers

[gendx]

Check whether the dictionary size and decompressed length headers use little-endian or big-endian encoding. The decompressed length is often replaced by a placeholder 0xFFFF_FFFF_FFFF_FFFF (in which case an end-of-stream marker is required instead), which makes it more difficult to test.