Test suite fails on big-endian machine

[jasontibbitts]

I got to the point of doing a package build in Fedora's infrastructure, and in our development branch that builds for seven architectures. Two of them (ppc64 and s390x/IBM System z) are big-endian. The build succeeds, but make check doesn't. Both fail with the same error:

Running unit tests
     CUnit - A unit testing framework for C - Version 2.1-3
     http://cunit.sourceforge.net/
Suite: annotate
  Test: getset_server_shared ...timeout: Unknown command '�'

That unprintable character is three bytes ("ef bf bd") but it could have been mangled in any number of ways by the time I gets to me. You can see the complete build logs at https://koji.fedoraproject.org/koji/taskinfo?taskID=19726766 (which will be live for two weeks before the build is garbage collected). Just click through the red links for the two failed builds and look for the "build.log" links. "hw_info.log" gives useful information about the hardware.

I could just disable the test suite on those architectures, I guess, but while I suspect this is purely a test suite problem, it could also be a problem with the main code and so I'd like to make sure.

If you need access to either a ppc64 or s390x machine, please let me know. Fedora does consider it important that everything supports LE and BE architectures and I can probably scrounge up someone to help if needed. I've done some preliminary looking into this but haven't managed to figure it out.

[elliefm]

So it looks like it's barfing out on the very first test in the very first test suite :/

You can run the cunit test suites individually by making a target like check-foo -- that is, to run just the annotate suite, run make check-annotate, or to run just the imapurl suite, make check-imapurl and so on. Each of the files "cunit/foo.testc" constitutes a suite named foo

I don't have any particular leads at the moment, but I'm curious whether this issue presents for all, some, or just the annotate suite?

Also, the fact that you can get most of the Cassandane tests working is kind of implying to me at the moment that maybe the endianness issue is somewhere in our cunit infrastructure, rather than in Cyrus proper. It'll be interesting to see if any of the cunit suites succeed, and which!

[elliefm]

For whatever it's worth:

cunit/timeout.c
144:            fprintf(stderr, "timeout: Unknown command '%c'\n", c);

[elliefm]

I don't suppose this teeny little patch fixes the issue?

diff --git a/cunit/timeout.c b/cunit/timeout.c
index f02c9ac..a444ac8 100644
--- a/cunit/timeout.c
+++ b/cunit/timeout.c
@@ -197,7 +197,7 @@ int timeout_init(void (*cb)(void))
 
 int timeout_begin(int millisec)
 {
-    int c;
+    char c;
     int r;
 
 // fprintf(stderr, "timeout_begin\n");
@@ -220,7 +220,7 @@ int timeout_begin(int millisec)
 
 int timeout_end(void)
 {
-    int c;
+    char c;
     int r;
 
 // fprintf(stderr, "timeout_end\n");

[jasontibbitts]

Well, it's better but it still doesn't pass. Here's a build: https://koji.fedoraproject.org/koji/taskinfo?taskID=19734442

Just look at the ppc64 build and ignore the fact that some other architectures fail as well; they're failing in cassandane and not make check (ugh). Maybe s390x will fail as well; we have fewer of those and the build hasn't even started yet.

To save you the clicking, here's the failed test output from ppc64. Everything else it runs appears to succeed, though I don't know if it runs all of the tests or stops at the first failure. The segfault in there is kind of troubling, too.

Suite: conversations
  Test: open ...passed
  Test: getset ...FAILED
    1. ./cunit/conversations.testc:72  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID=81985533797785328)
  Test: abort ...passed
  Test: prune ...FAILED
    1. ./cunit/conversations.testc:183  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID1=73278643237678883)
    2. ./cunit/conversations.testc:189  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID2=73579982724461108)
    3. ./cunit/conversations.testc:195  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID3=73897814885606213)
    4. ./cunit/conversations.testc:203  - nseen >= 3
    5. ./cunit/conversations.testc:204  - ndeleted >= 2
    6. ./cunit/conversations.testc:205  - nseen - ndeleted >= 1
    7. ./cunit/conversations.testc:215  - CU_ASSERT_EQUAL(arrayu64_size(&cids)=1,0=0)
    8. ./cunit/conversations.testc:227  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID3=73897814885606213)
  Test: two ...FAILED
    1. ./cunit/conversations.testc:312  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID2=74779769797035367)
  Test: cid_encode ...passed
  Test: cid_decode ...passed
  Test: folder_rename ...FAILED
    1. ./cunit/conversations.testc:427  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:439  - CU_ASSERT_PTR_NOT_NULL_FATAL(conv)
  Test: folders .../bin/sh: line 10: 27303 Segmentation fault      (core dumped) $vg ./unit $f
make[3]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make[3]: *** [Makefile:6559: check-local] Error 139
make[2]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make[2]: *** [Makefile:6325: check-am] Error 2
make[1]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make[1]: *** [Makefile:6033: check-recursive] Error 1
RPM build errors:
make: *** [Makefile:6327: check] Error 2

[elliefm]

That looks useful, cheers, will see if I can sort that one out too. One step at a time!

[elliefm]

This doesn't bode well, the first failure (line 72) is from this:

    /* get should still succeed after the db is closed & reopened */
    memset(&cid, 0x45, sizeof(cid));
    arrayu64_truncate(&cids, 0);
    r = conversations_get_msgid(state, C_MSGID, &cids);
    CU_ASSERT_EQUAL(r, 0);
    CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0), C_CID);

There doesn't seem to be anything endianness-dependent in the test itself, which suggests that the conversations module itself is busted -- doh!

[elliefm]

Okay, I don't expect this to fix this current problem, but I'm hoping the cunit conversations tests will report more precise errors with the following patch applied, which in turn might help narrow down where the issue lies. Can you give it a shot please? Cheers

diff --git a/imap/conversations.c b/imap/conversations.c
index 504c5db..2b151dc 100644
--- a/imap/conversations.c
+++ b/imap/conversations.c
@@ -560,11 +560,14 @@ EXPORTED int conversations_get_msgid(struct conversations_state *state,
                       &data, &datalen,
                       &state->txn);
 
+    if (r == CYRUSDB_NOTFOUND)
+        return 0; /* not an error, but nothing more to do */
+
     if (!r) r = _conversations_parse(data, datalen, cids, NULL);
 
     if (r) arrayu64_truncate(cids, 0);
 
-    return 0;
+    return r;
 }
 
 /*

[jasontibbitts]

Sorry for the late response; Monday was a holiday in the US. A fresh build is at https://koji.fedoraproject.org/koji/taskinfo?taskID=19785301

Both ppc64 and s390x fail pretty much identically; here's the relevant portion of the log. I'm not sure there's actually any more useful information in there, though.

Suite: conversations
  Test: open ...passed
  Test: getset ...FAILED
    1. ./cunit/conversations.testc:71  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:72  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID=81985533797785328)
  Test: abort ...passed
  Test: prune ...FAILED
    1. ./cunit/conversations.testc:182  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:183  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID1=73278643237678883)
    3. ./cunit/conversations.testc:188  - CU_ASSERT_EQUAL(r=-1,0=0)
    4. ./cunit/conversations.testc:189  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID2=73579982724461108)
    5. ./cunit/conversations.testc:194  - CU_ASSERT_EQUAL(r=-1,0=0)
    6. ./cunit/conversations.testc:195  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID3=73897814885606213)
    7. ./cunit/conversations.testc:203  - nseen >= 3
    8. ./cunit/conversations.testc:204  - ndeleted >= 2
    9. ./cunit/conversations.testc:205  - nseen - ndeleted >= 1
    10. ./cunit/conversations.testc:215  - CU_ASSERT_EQUAL(arrayu64_size(&cids)=1,0=0)
    11. ./cunit/conversations.testc:220  - CU_ASSERT_EQUAL(r=-1,0=0)
    12. ./cunit/conversations.testc:226  - CU_ASSERT_EQUAL(r=-1,0=0)
    13. ./cunit/conversations.testc:227  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID3=73897814885606213)
  Test: two ...FAILED
    1. ./cunit/conversations.testc:311  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:312  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID2=74779769797035367)
  Test: cid_encode ...passed
  Test: cid_decode ...passed
  Test: folder_rename ...FAILED
    1. ./cunit/conversations.testc:427  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:439  - CU_ASSERT_PTR_NOT_NULL_FATAL(conv)
  Test: folders .../bin/sh: line 10: 23048 Segmentation fault      (core dumped) $vg ./unit $f
make[3]: *** [Makefile:6559: check-local] Error 139
make[3]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make[2]: *** [Makefile:6325: check-am] Error 2
make[2]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make[1]: *** [Makefile:6033: check-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/cyrus-imapd-3.0.1'
make: *** [Makefile:6327: check] Error 2

s390x fails identically.

[elliefm]

Nah that's helpful! Notice this:

    1. ./cunit/conversations.testc:71  - CU_ASSERT_EQUAL(r=-1,0=0)
    2. ./cunit/conversations.testc:72  - CU_ASSERT_EQUAL(arrayu64_nth(&cids, 0)=0,C_CID=81985533797785328)

We can now see that the failure is actually occurring one line earlier, and the arrayu64_nth(...) failures are a natural consequence of some problem in the prior call :)

I think line 71, r=-1 means it has thrown CYRUSDB_IOERROR = -1.

Will keep digging and see what next step presents itself.

[elliefm]

Okay I've dug around in the conversations API and it really does look like that error being handed down from cyrusdb. But we don't get to see the results of the cyrusdb tests because they're alphabetically sorted such that they run after the conversations tests, doh!

I've changed the order that the tests run so that the database tests are run first -- e3ee582 is the relevant commit for the 3.0 branch. Are you able to put through another test run with this commit applied? With it, I expect something in the (newly renamed) aaa-db suite to fail -- which will hopefully point us towards the next step!

Thanks so much for working through this with me, it's really great to be able to exercise these code corners on platforms we don't usually have access to :)

[elliefm]

I've also added 443e546 to the 3.0 branch which corrects the conversations suite to treat failures to open the conversations database as fatal errors. I think this should prevent it segfaulting out later on, so hopefully we get to see some additional tests/suites being run!