LZFSE doesn't work on big endian

[nemequ]

I'm setting up CI for Squash on POWER8, and LZFSE is one of the codecs which fails. To make sure the issue isn't with Squash, I tried a simple test of just LZFSE:

[fedora@squash-be lzfse]$ ./build/bin/lzfse -encode < README.md | ./build/bin/lzfse -decode
malloc: Cannot allocate memory
[fedora@squash-be lzfse]$ uname -a
Linux squash-be.novalocal 4.5.7-202.fc23.ppc64 #1 SMP Thu Jun 30 10:45:03 UTC 2016 ppc64 ppc64 ppc64 GNU/Linux

There are lots of other errors for LZFSE in the Squash test suite (which is pretty evil) on ppc64be (sample log), so it could be helpful for testing.

If it would help, I can provide access to a VM for testing. Just e-mail me, or drop by #squash on freenode, and we can figure something out.

[nemequ]

I started playing with this, see https://github.com/nemequ/lzfse/tree/endian. I think I have the encoder done (I haven't tested many files, but the ones I have tested provide the same output on LE and BE). Decoder still crashes.

Not sure when I'll have time to finish this up, so if anyone else wants to pick it up feel free.

[juanandresgs]

Any fix for this?

[Artoria2e5]

The tree still does not have the endian functions here, so probably not in here yet. I will see if I can do a forward port while adding a test case from a LE amd64-gcc/clang machine, since another pet peeve of mine is about loss of packing information in msvc v2 header.