support different byte orders in serializer #1295

Open
vtjnash opened this Issue Sep 20, 2012 · 5 comments

Comments

Projects

Decision Needed in Julia

3 participants
Member

vtjnash commented Sep 20, 2012

In the serializer (and elsewhere?), array reads and writes probably should be done in network byte order.

Owner

StefanKarpinski commented Sep 20, 2012

God no. Network byte order needs to die. There's nothing better about it. If anything, the serializer should have a BOM in the header of the data stream and conditionally translate from the serialized order, which ought to be native to the sender, to the native order of the receiver. Alternatively, the receiver can just die if the BOM doesn't match its local endianness. Who the hell runs a cluster with heterogenous endianness? That's a silly thing to do.

Member

vtjnash commented Sep 27, 2012

It's better only in that it is consistant. Isn't Serialization used for writing files too? Adding a header with endianness makes sense too (since it reduces work for the common case of little-endian to little-endian). I was talking with Jeff about making a few changes to the structure of the serialization protocol, to make it more robust against other sorts of data corruption issues.

Owner

StefanKarpinski commented Sep 27, 2012

My general opinion on endianness and serialization formats is that byte swapping should be done lazily even for writing files because by far the most common case is that you are communicating between machines with the same endianness. Moreover, these days it is almost certainly little-endian, especially since Julia doesn't even run on any big-endian architectures. Are there any common big-endian architectures that people even use these days?

Member

vtjnash commented Sep 27, 2012

PowerPC (the only other architecture currently supported by the LLVM JIT) is big/bi-endian. It shows up in FPGAs (e.g. Xilinx Virtex 4/5), all modern game consoles (e.g. XBox360, Wii and PS3 are big-endian), high-performance computers[citation needed], and old apple computers (which are stuck at Mac OS 10.5).

ARM is little/bi-endian. So it can be used in big-endian mode (although it appears that Android generally uses little-endian everywhere).

Owner

StefanKarpinski commented Sep 27, 2012

Ok, well, it sounds like something we ought to plan to accommodate, but not at the expense of making the vastly most common case of little-endian-to-little-endian do lots of pointless byte swapping.

ViralBShah removed the feature label Feb 14, 2015

StefanKarpinski added this to the 1.0 milestone Sep 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment