Some tests fail on PowerPC: 75% tests passed, 4 tests failed out of 16

[barracuda156]

--->  Testing carquet
Executing:  cd "/opt/local/var/macports/build/carquet-1c42e9c8/work/build" && /usr/bin/make test 
Running tests...
/opt/local/bin/ctest --force-new-ctest-process 
Test project /opt/local/var/macports/build/carquet-1c42e9c8/work/build
      Start  1: test_core
 1/16 Test  #1: test_core ........................   Passed    0.01 sec
      Start  2: test_thrift
 2/16 Test  #2: test_thrift ......................   Passed    0.01 sec
      Start  3: test_encodings
 3/16 Test  #3: test_encodings ...................   Passed    0.01 sec
      Start  4: test_reader
 4/16 Test  #4: test_reader ......................   Passed    0.01 sec
      Start  5: test_encodings_extended
 5/16 Test  #5: test_encodings_extended ..........***Failed    0.01 sec
      Start  6: test_compression
 6/16 Test  #6: test_compression .................   Passed    0.01 sec
      Start  7: test_utils
 7/16 Test  #7: test_utils .......................   Passed    0.01 sec
      Start  8: test_production
 8/16 Test  #8: test_production ..................   Passed    0.02 sec
      Start  9: test_maturity
 9/16 Test  #9: test_maturity ....................   Passed    0.71 sec
      Start 10: test_large_schema
10/16 Test #10: test_large_schema ................   Passed    0.41 sec
      Start 11: test_edge_encodings
11/16 Test #11: test_edge_encodings ..............***Failed    0.01 sec
      Start 12: test_edge_compression
12/16 Test #12: test_edge_compression ............   Passed    0.02 sec
      Start 13: test_edge_malformed
13/16 Test #13: test_edge_malformed ..............   Passed    0.01 sec
      Start 14: test_edge_boundaries
14/16 Test #14: test_edge_boundaries .............***Failed    0.18 sec
      Start 15: test_edge_io
15/16 Test #15: test_edge_io .....................   Passed    0.01 sec
      Start 16: test_mmap
16/16 Test #16: test_mmap ........................***Failed    0.02 sec

75% tests passed, 4 tests failed out of 16

Total Test time (real) =   1.48 sec

The following tests FAILED:
      5 - test_encodings_extended (Failed)
     11 - test_edge_encodings (Failed)
     14 - test_edge_boundaries (Failed)
     16 - test_mmap (Failed)
Errors while running CTest

Tests log:

carquet_test_log.txt

[Vitruves]

Thanks for the detailed test logs! These were classic big-endian issues. I've pushed fixes for all 4 failing tests:

1. Dictionary encoding failures (test_encodings_extended, test_edge_encodings)

• Issue: Dictionary decode functions were casting raw bytes directly to int32_t*/int64_t*/float*/double* and reading without byte-swapping
• Fix: Updated dictionary.c to use carquet_read_i32_le(), carquet_read_i64_le(), carquet_read_f32_le(), carquet_read_f64_le() for all dictionary lookups

2. Arena alignment failure (test_edge_boundaries)

• Issue: On 32-bit PPC, the flexible array member data[] in carquet_arena_block_t started at offset 12 (not 8-byte aligned)
• Fix: Changed arena.h to use a union with alignment-forcing members (double, long long, void*) ensuring proper alignment on all platforms

3. Large compression roundtrip failure (test_edge_boundaries)

• Issue: lz4_count() used trailing-zero counting that assumed little-endian byte order
• Fix: Rewrote lz4_count() to use an endian-independent byte-by-byte comparison when a mismatch is found within an 8-byte block

4. Memory-mapped read failures (test_mmap)

• Issue: Zero-copy reads returned little-endian data directly on big-endian systems
• Fix: carquet_page_is_zero_copy_eligible() now returns false on big-endian systems, forcing the standard decode path with proper byte-swapping

Could you test again and confirm these resolve all the PPC issues?

[barracuda156]

Thank you for the update.
Using 444bad9 commit, three tests still fail:

carquet_2.txt

[Vitruves]

Thanks for the detailed test logs! I've pushed fixes for all three remaining issues:

1. Dictionary encoding endianness (src/encoding/dictionary.c)
   The encode functions were storing values in native byte order instead of little-endian. Fixed by using carquet_write_*_le() functions when adding values to the dictionary buffer.

2. Arena alignment on 32-bit (src/core/arena.c)
   The alignment calculation was relative to the buffer start rather than absolute addresses. On 32-bit systems where malloc() may only guarantee 4-byte alignment, this caused misaligned pointers. Fixed by calculating alignment based on the absolute address of each allocation.

3. Bitpack endianness (src/core/bitpack.c)
   The bit unpacking functions used memcpy() to load multi-byte integers, which interprets bytes differently on big-endian systems. Fixed by adding explicit little-endian read helpers (read_le16, read_le24, etc.) for all bitunpack functions.

All tests pass on x86. Could you verify these fixes on your PowerPC setup when you get a chance?

[barracuda156]

Looks like all passes now. Thank you!

carquet3.txt

[Vitruves]

Awesome, thanks for confirming! Appreciate you testing on PowerPC.