msgpack unit test failures on ppc64 arch #524
Comments
|
Thanks for the report. Could this be an endianess issue? |
|
Yeah, that would be my guess too. Consider: which looks like |
|
ppc64 is big endian |
|
if you decide to do the swaps don't forget about bit_cast<>(). |
|
I currently do not have the means to test this. Is there a way to have a big endian system with Vagrant? |
|
Apparently the users, test this for you, you don't need to test yourself.
Otherwise, try qemu, install a big endian distro and start testing.
2017-04-01 1:08 GMT+02:00 Niels Lohmann <notifications@github.com>:
… I currently do not have the means to test this. Is there a way to have a
big endian system with Vagrant?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVM6a_CIFxnG3AQFUuhJPrTh_kJCBks5rrYdigaJpZM4Mh1xK>
.
|
|
In the end, I need a test setup to at least verify a fix. I could get qemu to work, but I failed to get a recent Linux with a recent compiler to work, let alone properly integrate it into my network. Hints greatly appreciated! |
|
Its very involved, I don't even remember how I did it myself, but I managed
at last. You should use Debian and follow some online tuts.
…On Apr 3, 2017 22:31, "Niels Lohmann" ***@***.***> wrote:
In the end, I need a test setup to at least verify a fix.
I could get qemu to work, but I failed to get a recent Linux with a recent
compiler to work, let alone properly integrate it into my network. Hints
greatly appreciated!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVNy1ltorQmhawYxdQLn7SktGNza1ks5rsVckgaJpZM4Mh1xK>
.
|
|
Hey @dkopecek, could you please try https://github.com/nlohmann/json/blob/feature/manual_lexer/src/json.hpp ? I could not test it with a big-endian system myself. It is a reimplementation of the binary readers/writers which should respect the byte order. The main functions are: // from http://stackoverflow.com/a/1001328/266378
static bool little_endianess()
{
int num = 1;
return (*reinterpret_cast<char*>(&num) == 1);
}// writes a number of type T to vector v
template<typename T>
void write_number(T n)
{
std::array<uint8_t, sizeof(T)> vec;
std::memcpy(vec.data(), &n, sizeof(T));
for (size_t i = 0; i < sizeof(T); ++i)
{
// reverse byte order prior to conversion if necessary
if (is_little_endian)
{
v.push_back(vec[sizeof(T) - i - 1]);
}
else
{
v.push_back(vec[i]);
}
}
}// reads a number of type T from input
template<typename T>
T get_number()
{
std::array<uint8_t, sizeof(T)> vec;
for (size_t i = 0; i < sizeof(T); ++i)
{
get();
check_eof();
// reverse byte order prior to conversion if necessary
if (is_little_endian)
{
vec[sizeof(T) - i - 1] = static_cast<uint8_t>(current);
}
else
{
vec[i] = static_cast<uint8_t>(current);
}
}
T result;
std::memcpy(&result, vec.data(), sizeof(T));
return result;
} |
nlohmann
added
the
solution: proposed fix
|
This code is inefficient considering that endianess is a compile-time
constant. You could do away with the if.
…On Apr 5, 2017 06:58, "Niels Lohmann" ***@***.***> wrote:
Hey @dkopecek <https://github.com/dkopecek>, could you please try
https://github.com/nlohmann/json/blob/feature/manual_lexer/src/json.hpp ?
I could not test it with a big-endian system myself.
It is a reimplementation of the binary readers/writers which should
respect the byte order.
The main functions are:
// from http://stackoverflow.com/a/1001328/266378static bool little_endianess()
{
int num = 1;
return (*reinterpret_cast<char*>(&num) == 1);
}
// writes a number of type T to vector vtemplate<typename T>void write_number(T n)
{
std::array<uint8_t, sizeof(T)> vec;
std::memcpy(vec.data(), &n, sizeof(T));
for (size_t i = 0; i < sizeof(T); ++i)
{
// reverse byte order prior to conversion if necessary
if (is_little_endian)
{
v.push_back(vec[sizeof(T) - i - 1]);
}
else
{
v.push_back(vec[i]);
}
}
}
// reads a number of type T from inputtemplate<typename T>
T get_number()
{
std::array<uint8_t, sizeof(T)> vec;
for (size_t i = 0; i < sizeof(T); ++i)
{
get();
check_eof();
// reverse byte order prior to conversion if necessary
if (is_little_endian)
{
vec[sizeof(T) - i - 1] = static_cast<uint8_t>(current);
}
else
{
vec[i] = static_cast<uint8_t>(current);
}
}
T result;
std::memcpy(&result, vec.data(), sizeof(T));
return result;
}
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVK0SirO4aXnunKTw7iKvxEGUVaMCks5rsx9agaJpZM4Mh1xK>
.
|
|
I could not find such a constant in the standard, and I would like to avoid dozens of |
|
you can still have a constexpr variable, if you wish, but the real bonus
would be to avoid the if statement.
Example:
#if defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ || \
defined(__BIG_ENDIAN__)
constexpr auto big_endian = true;
#endif
2017-04-05 7:37 GMT+02:00 Niels Lohmann <notifications@github.com>:
… I could not find such a constant in the standard, and I would like to
avoid dozens of #ifdefs to cover all compilers. I'm curious whether this
fix works in the first place.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVHN1biNPfYQQ403e-hqpH0GsO3z7ks5rsyiDgaJpZM4Mh1xK>
.
|
|
Is that macro portable? |
|
checks boost's endian.hpp and adapt the code there, it's more complete than
my example.
2017-04-05 8:51 GMT+02:00 Niels Lohmann <notifications@github.com>:
… Is that macro portable?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVKDeqG1WBhabe7_LRr2IWgsRUBQuks5rszn9gaJpZM4Mh1xK>
.
|
|
This was exactly what I wanted to avoid. |
|
Avoid boost, yes, but you can still learn from their code.
2017-04-05 8:55 GMT+02:00 Niels Lohmann <notifications@github.com>:
… This was exactly what I wanted to avoid.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVI_xjEyzutaV-6xgTYEerhT3Ukgrks5rszrxgaJpZM4Mh1xK>
.
|
|
I wanted to avoid "dozens of |
|
@nlohmann Cannot confirm that the fix works, because there are some compilation errors. See the attached |
|
Sorry, I forgot I also changed the test suite. You need to check out the whole branch. |
|
@nlohmann Unit tests fail on ppc64 and ppc64le: ppc64le: https://kojipkgs.fedoraproject.org//work/tasks/1585/18791585/build.log |
|
Thanks for the feedback. I really need a test system with ppc64. |
|
@nlohmann Check out virt-builder: There are templates for Fedora ppc64, ppc64le, armv7, ... etc. (use |
|
This has nothing to do with the actual error, but the build script is odd: Instead of calling doxygen directly, one should call |
nlohmann
removed
the
solution: proposed fix
|
I managed to set up a Debian 3.16.39 with powerpc architecture using qemu. With GCC 4.9.2 I could compile all tests. CBOR/MessagePack are still failing, but at least I can now debug myself ;) |
|
Are you sure that it is a big-endian distro? Last time I checked I only saw
little endian ones. Maybe you need a MIPS distro?
2017-04-08 23:06 GMT+02:00 Niels Lohmann <notifications@github.com>:
… I managed to set up a Debian 3.16.39 with powerpc architecture using qemu.
With GCC 4.9.2 I could compile all tests. CBOR/MessagePack are still
failing, but at least I can now debug myself ;)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVHiD9MWvLThmaC7BCniJKDHDFhcKks5rt_a6gaJpZM4Mh1xK>
.
|
|
It seems so: |
|
With the manual_lexer branch, there is only one test failing for CBOR and MessagePack, respectively:
It seems that the test itself assumes little endianess. I shall add a fix. |
|
With the last commit (6f99d5b) all tests succeed. |
nlohmann
added
the
solution: proposed fix
|
@dkopecek Could you please check whether the fix works for you? |
|
Hey, don't forget MIPS, now that you're at it.
2017-04-10 21:13 GMT+02:00 Niels Lohmann <notifications@github.com>:
… @dkopecek <https://github.com/dkopecek> Could you please check whether
the fix works for you?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVMVUgEfhxzwmzwGiXQxZAtm5xxQFks5run9PgaJpZM4Mh1xK>
.
|
|
@nlohmann Oops, sorry, I was planning to but totally forgot :-) I'll check it out tomorrow (CEST timezone) and let you know. |
|
What's wrong with MIPS? |
|
It's big-endian, just like PPC.
2017-04-10 21:37 GMT+02:00 Niels Lohmann <notifications@github.com>:
… What's wrong with MIPS?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVPKZA6JyINcdK88_0wASBsCASFLKks5ruoUNgaJpZM4Mh1xK>
.
|
|
Yes, and I proposed a fix for big endianess. I assume it also works for MIPS, but I already had troubles getting PPC to work. |
|
Try it if you have time, on MIPS.
2017-04-10 21:40 GMT+02:00 Niels Lohmann <notifications@github.com>:
… Yes, and I proposed a fix for big endianess. I assume it also works for
MIPS, but I already had troubles getting PPC to work.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#524 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AH6jVKrRqkQwPt97McsCXqCQbNCFjqVoks5ruoWTgaJpZM4Mh1xK>
.
|
|
Sorry, i don't, and would appreciate help on this issue. |
|
@nlohmann results: x86_64: ok but this time it's a different fail. see the attached ppc64-build.log.txt |
|
Thanks for reporting. Could you rerun the tests with This runs all tests and does not abort after the first failure. |
|
Interesting. With that testing procedure, the tests don't fail on ppc64 and do fail on armv7hl. So there seems to be something non-deterministic in the tests. |
|
That is indeed strange. The tests should be (of course) deterministic, but since the ARM error is a SIGSEGV, maybe just a different call could have triggered it. Without debugging, I cannot tell much about the issue... Could I ask you to try the following? This should give us more information on when the test crashes. |
|
(On the bright side, it seems as if the fix for the endianess issue seems to work) |
Doesn't fail at all. Tried several times. |
|
Strange... Could you try to execute the tests with Valgrind? |
|
Here are the logs. x86_64 is ok but I've attached it anyway. armv7hl-build.3.log.txt |
|
I think I need to close this issue. I cannot reproduce the problem, and I can do little with the provided logs. |
Hi, I was trying to build[1] the newest release on multiple architectures (Fedora build architectures). There seems to be some msgpack related unit test failure on ppc64. See the attached build log:
json-ppc64-build.log.txt
[1] https://koji.fedoraproject.org/koji/taskinfo?taskID=18476755
The text was updated successfully, but these errors were encountered: