ORDER BY works incorrectly on OCTEON (mips64 big-endian) platform

[bgli100]

What happens?

400.csv

D select id,create_time from '/disk/400.csv'  order by id limit 200;
┌───────┬─────────────────────┐
│  id   │     create_time     │
│ int32 │      timestamp      │
├───────┼─────────────────────┤
│ 50048 │ 2022-11-22 12:00:08 │
│ 50304 │ 2022-11-22 12:00:09 │
│ 50049 │ 2022-11-22 12:00:08 │
│ 50305 │ 2022-11-22 12:00:09 │
│ 50050 │ 2022-11-22 12:00:08 │
│ 50306 │ 2022-11-22 12:00:09 │
│ 50051 │ 2022-11-22 12:00:08 │
│ 50307 │ 2022-11-22 12:00:09 │
│ 50052 │ 2022-11-22 12:00:08 │
│ 50308 │ 2022-11-22 12:00:09 │
│   ·   │          ·          │
│   ·   │          ·          │
│   ·   │          ·          │
│ 50143 │ 2022-11-22 12:00:08 │
│ 50399 │ 2022-11-22 12:00:09 │
│ 50144 │ 2022-11-22 12:00:08 │
│ 50400 │ 2022-11-22 12:00:09 │
│ 50145 │ 2022-11-22 12:00:08 │
│ 50401 │ 2022-11-22 12:00:09 │
│ 50146 │ 2022-11-22 12:00:08 │
│ 50402 │ 2022-11-22 12:00:09 │
│ 50147 │ 2022-11-22 12:00:08 │
│ 50148 │ 2022-11-22 12:00:08 │
├───────┴─────────────────────┤
│     200 rows (20 shown)     │
└─────────────────────────────┘

To Reproduce

Run the following query:
select id,create_time from '/disk/400.csv' order by id limit 200;

OS:

Linux on OCTEON (mips64, big-endian)

DuckDB Version:

0.5 and 0.6 (0.4 works OK)

DuckDB Client:

DuckDB shell

Full Name:

Alvin Li

Affiliation:

No Affiliation

Have you tried this on the latest master branch?

• I agree

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• I agree

[lnkuiper]

DuckDB detects needs to detect endianness correctly for sorting / index creation. I would expect this to be the problem here

[Mytherin]

Thanks for the report!

DuckDB assumes little-endian architecture in several places - specifically the ORDER BY and the index implementations. In addition storage files are not portable between little endian and big endian machines.

Big endian machines are incredibly rare nowadays - I would not even know where to find a machine to debug endianness issues. As such, supporting big endian architectures is not a high priority for us right now.

We will fix this issue by adding a not implemented exception for operations that do not work correctly on big endian systems.

If you want to discuss better support of DuckDB for big endian systems we do offer commercial support.

[bgli100]

Thanks for the report!

DuckDB assumes little-endian architecture in several places - specifically the ORDER BY and the index implementations. In addition storage files are not portable between little endian and big endian machines.

Big endian machines are incredibly rare nowadays - I would not even know where to find a machine to debug endianness issues. As such, supporting big endian architectures is not a high priority for us right now.

We will fix this issue by adding a not implemented exception for operations that do not work correctly on big endian systems.

If you want to discuss better support of DuckDB for big endian systems we do offer commercial support.

Thanks for your fast response!

Sadly commercial support is not a possible option for me. Would you mind to point out some source file name / function name / code line that assumes little-endian architecture which could relate to ORDER BY? So that I could try debug this myself.

Your help is highly appreciated.

[Mytherin]

The code in question is related to the bit decomposition used for the radix sort. I would start looking at the PhysicalOrder operator.

May I ask why you are using a (presumably very old) big endian machine?

[lnkuiper]

It used to be in src/common/radix.hpp. We still have the method to check endianness there, but it is no longer used. The EncodeData functions used to take an extra argument

[bgli100]

Some embedded system manufacturer here are still producing big-endian new machines (I have no idea it is a big-endian one when I purchased it).

I am trying to reuse what I have on my intrest project. I have x86 PC but the power consumption is an order of magitude higher so I don't like to keep it on. If it could be easily solved then no need to change device.

[bgli100]

Thanks for you both. Very helpful!

[Mytherin]

If it is a more recent chip there is a good chance it is bi-endian, rather than only big endian, and you should be able to use it in little endian mode as well. That might save you some headaches.