Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

neomutt crashes with hcache #188

Closed
hurricanehrndz opened this issue Oct 7, 2016 · 15 comments
Closed

neomutt crashes with hcache #188

hurricanehrndz opened this issue Oct 7, 2016 · 15 comments
Labels

Comments

@hurricanehrndz
Copy link

hurricanehrndz commented Oct 7, 2016

When setting hcache back end to kyotocabinet and setting header_cache and message_cachedir,
I receive an error stating illegal operation.

NeoMutt 20161003 (1.7.0)
Copyright (C) 1996-2016 Michael R. Elkins and others.
Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
Mutt is free software, and you are welcome to redistribute it
under certain conditions; type `mutt -vv' for details.

System: Linux 4.7.2-2-default (x86_64)
slang: 20300
libidn: 1.33 (compiled with 1.33)
hcache backend: kyotocabinet 1.2.76

Compiler:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/6/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada,go --enable-offload-targets=hsa --enable-checking=release --with-gxx-include-dir=/usr/include/c++/6 --enable-ssp --disable-libssp --disable-libvtv --disable-libcc1 --enable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --enable-linux-futex --enable-gnu-indirect-function --program-suffix=-6 --without-system-libunwind --enable-multilib --with-arch-32=x86-64 --with-tune=generic --build=x86_64-suse-linux --host=x86_64-suse-linux
Thread model: posix
gcc version 6.2.1 20160830 [gcc-6-branch revision 239856] (SUSE Linux) 

Configure options: '--host=x86_64-suse-linux-gnu' '--build=x86_64-suse-linux-gnu' '--program-prefix=' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/lib' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--infodir=/usr/share/info' '--prefix=/usr' '--sysconfdir=/etc/mutt' '--mandir=/usr/share/man' '--with-docdir=/usr/share/doc/packages/neomutt' '--with-mailpath=/var/mail' '--disable-dependency-tracking' '--with-slang' '--with-gnutls' '--with-sasl' '--with-gss' '--with-idn' '--with-mixmaster' '--with-kyotocabinet' '--without-tokyocabinet' '--without-gdbm' '--without-bdb' '--without-qdbm' '--enable-fcntl' '--enable-hcache' '--enable-gpgme' '--enable-pop' '--enable-imap' '--enable-smtp' '--enable-sidebar' '--enable-notmuch' '--enable-nntp' '--enable-compressed' 'build_alias=x86_64-suse-linux-gnu' 'host_alias=x86_64-suse-linux-gnu' 'CFLAGS=-fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables'

Compilation CFLAGS: -Wall -pedantic -Wno-long-long -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fno-delete-null-pointer-checks

Compile options:
+CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME +CRYPT_BACKEND_GPGME 
-DEBUG -DL_STANDALONE +ENABLE_NLS -EXACT_ADDRESS -HOMESPOOL -LOCALES_HACK 
-SUN_ATTACHMENT -HAVE_BKGDSET +HAVE_COLOR -HAVE_CURS_SET +HAVE_GETADDRINFO 
+HAVE_GETSID +HAVE_ICONV +HAVE_LANGINFO_CODESET +HAVE_LANGINFO_YESEXPR 
+HAVE_LIBIDN -HAVE_META +HAVE_REGCOMP -HAVE_RESIZETERM -HAVE_START_COLOR 
-HAVE_TYPEAHEAD +HAVE_WC_FUNCS +ICONV_NONTRANS +USE_COMPRESSED -USE_DOTLOCK 
+USE_FCNTL -USE_FLOCK -USE_FMEMOPEN -USE_GNU_REGEX +USE_GSS +USE_HCACHE 
+USE_IMAP +USE_NOTMUCH +USE_NNTP +USE_POP +USE_SASL -USE_SETGID +USE_SIDEBAR 
+USE_SMTP +USE_SSL_GNUTLS -USE_SSL_OPENSSL 
-DOMAIN
MIXMASTER="mixmaster"
-ISPELL
SENDMAIL="/usr/sbin/sendmail"
MAILPATH="/var/mail"
PKGDATADIR="/usr/share/mutt"
SYSCONFDIR="/etc/mutt"
EXECSHELL="/bin/sh"

patch-attach-headers-color-neomutt
patch-compose-to-sender-neomutt
patch-compress-neomutt
patch-cond-date-neomutt
patch-encrypt-to-self-neomutt
patch-fmemopen-neomutt
patch-forgotten-attachments-neomutt
patch-ifdef-neomutt
patch-index-color-neomutt
patch-initials-neomutt
patch-keywords-neomutt
patch-kyoto-neomutt
patch-limit-current-thread-neomutt
patch-lmdb-neomutt
patch-multiple-fcc-neomutt
patch-nested-if-neomutt
patch-new-mail-neomutt
patch-nntp-neomutt
patch-notmuch-neomutt
patch-progress-neomutt
patch-quasi-delete-neomutt
patch-reply-with-xorig-neomutt
patch-sensible-browser-neomutt
patch-sidebar-neomutt
patch-skip-quoted-neomutt
patch-smime-encrypt-self-neomutt
patch-status-color-neomutt
patch-timeout-neomutt
patch-tls-sni-neomutt

To learn more about NeoMutt, visit: http://www.neomutt.org/
If you find a bug in NeoMutt, please raise an issue at:
    https://github.com/neomutt/neomutt/issues
or send an email to: <neomutt-devel@neomutt.org>
@flatcap flatcap added the bug label Oct 8, 2016
@flatcap
Copy link
Member

flatcap commented Oct 8, 2016

From @hurricanehrndz's debugging:

Program received signal SIGILL, Illegal instruction.
0x00007ffff70234e6 in ?? () from /usr/lib64/libkyotocabinet.so.16
(gdb) bt
#0  0x00007ffff70234e6 in ?? () from /usr/lib64/libkyotocabinet.so.16
#1  0x00007ffff7068ba4 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::save_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*) ()
   from /usr/lib64/libkyotocabinet.so.16
#2  0x00007ffff7068ecd in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*, bool) ()
   from /usr/lib64/libkyotocabinet.so.16
#3  0x00007ffff7068f46 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_cache(bool) () from /usr/lib64/libkyotocabinet.so.16
#4  0x00007ffff7093fac in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ()
   from /usr/lib64/libkyotocabinet.so.16
#5  0x00007ffff7093184 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::reorganize_file(unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#6  0x00007ffff7093b88 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) ()
   from /usr/lib64/libkyotocabinet.so.16
#7  0x00007ffff706fe8d in kyotocabinet::PolyDB::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#8  0x00007ffff7023c43 in kcdbopen () from /usr/lib64/libkyotocabinet.so.16
#9  0x00000000004b8ee7 in hcache_open_kc ()
#10 0x00000000004ba070 in mutt_hcache_open ()
#11 0x000000000044de78 in maildir_delayed_parsing ()
#12 0x000000000044e80a in mh_read_dir ()
#13 0x000000000044ea3e in maildir_open_mailbox ()
#14 0x000000000045156a in mx_open_mailbox ()
#15 0x0000000000461997 in mutt_num_postponed ()
#16 0x000000000047af14 in status_format_str ()
#17 0x0000000000485103 in mutt_FormatString ()
#18 0x000000000048524a in mutt_FormatString ()
#19 0x000000000047b823 in menu_status_line ()
#20 0x0000000000426085 in mutt_index_menu ()
#21 0x000000000040ab36 in main ()

@hurricanehrndz
Copy link
Author

More with some debug info, still need to install more debug packages.

#0  0x00007ffff70234e6 in ?? () from /usr/lib64/libkyotocabinet.so.16
#1  0x00007ffff7068ba4 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::save_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*) () from /usr/lib64/libkyotocabinet.so.16
#2  0x00007ffff7068ecd in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*, bool) () from /usr/lib64/libkyotocabinet.so.16
#3  0x00007ffff7068f46 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_cache(bool) ()
   from /usr/lib64/libkyotocabinet.so.16
#4  0x00007ffff7093fac in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#5  0x00007ffff706fe8d in kyotocabinet::PolyDB::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#6  0x00007ffff7023c43 in kcdbopen () from /usr/lib64/libkyotocabinet.so.16
#7  0x00000000004b8ee7 in hcache_open_kc (h=0x8538c0,
    path=path@entry=0x70fd80 <hcpath> "/home/hurricanehrndz/.mutt/cache/01_degmac/headers/b67745c237197f722904d141dd776049") at hcache.c:1156
#8  0x00000000004ba070 in mutt_hcache_open (path=<optimized out>, folder=<optimized out>, namer=namer@entry=0x0) at hcache.c:1545
#9  0x000000000044de78 in maildir_delayed_parsing (ctx=ctx@entry=0x7fffffffbe50, md=md@entry=0x7fffffffba30, progress=progress@entry=0x7fffffffbae0)
    at mh.c:1156
#10 0x000000000044e80a in mh_read_dir (ctx=ctx@entry=0x7fffffffbe50, subdir=subdir@entry=0x4cd26a "new") at mh.c:1277
#11 0x000000000044ea3e in maildir_read_dir (ctx=0x7fffffffbe50) at mh.c:1304
#12 maildir_open_mailbox (ctx=0x7fffffffbe50) at mh.c:1312
#13 0x000000000045156a in mx_open_mailbox (path=<optimized out>, flags=flags@entry=9, pctx=pctx@entry=0x7fffffffbe50) at mx.c:679
#14 0x0000000000461997 in mutt_num_postponed (force=<optimized out>, force@entry=0) at postpone.c:139
#15 0x000000000047af14 in status_format_str (buf=buf@entry=0x7fffffffc360 "", buflen=buflen@entry=1024, col=0, cols=cols@entry=151,
    op=<optimized out>, src=0x7e4417 "", prefix=0x7fffffffc260 "", ifstring=0x7fffffffc2e0 "( %p postponed )", elsestring=0x7fffffffc760 "",
    data=8903952, flags=MUTT_FORMAT_OPTIONAL) at status.c:206
#16 0x0000000000485103 in mutt_FormatString (dest=dest@entry=0x7fffffffcd30 "", destlen=1023, destlen@entry=1024, col=col@entry=0,
    cols=cols@entry=151, src=<optimized out>, src@entry=0x7e4402 "%<p?( %p postponed )>", callback=callback@entry=0x47ac80 <status_format_str>,
    data=8903952, flags=MUTT_FORMAT_OPTIONAL) at muttlib.c:1568
#17 0x000000000048524a in mutt_FormatString (dest=0x7fffffffd6a0 "Folder: 01_degmac", '=' <repeats 21 times>, "   -16 messages (4 new)  ",
    destlen=1023, col=col@entry=0, cols=151, src=0x7e4401 " %<p?( %p postponed )>", callback=callback@entry=0x47ac80 <status_format_str>,
    data=8903952, flags=(unknown: 0)) at muttlib.c:1466
#18 0x000000000047b823 in menu_status_line (buf=buf@entry=0x7fffffffd6a0 "Folder: 01_degmac", '=' <repeats 21 times>, "   -16 messages (4 new)  ",
    buflen=buflen@entry=1024, menu=<optimized out>, p=<optimized out>) at status.c:326
#19 0x0000000000426085 in mutt_index_menu () at curs_main.c:918
#20 0x000000000040ab36 in main (argc=1, argv=<optimized out>) at main.c:882

@hurricanehrndz
Copy link
Author

hurricanehrndz commented Oct 8, 2016

full debug

#0  0x00007ffff70234e6 in kyotocabinet::PlantDB<kyotocabinet::CacheDB, (unsigned char)33>::write_key(char*, int, long) [clone .isra.287] ()
   from /usr/lib64/libkyotocabinet.so.16
#1  0x00007ffff7068ba4 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::save_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*) () from /usr/lib64/libkyotocabinet.so.16
#2  0x00007ffff7068ecd in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_node(kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::LeafNode*, bool) () from /usr/lib64/libkyotocabinet.so.16
#3  0x00007ffff7068f46 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::flush_leaf_cache(bool) ()
   from /usr/lib64/libkyotocabinet.so.16
#4  0x00007ffff7093fac in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#5  0x00007ffff7093184 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::reorganize_file(unsigned int) ()
   from /usr/lib64/libkyotocabinet.so.16
#6  0x00007ffff7093b88 in kyotocabinet::PlantDB<kyotocabinet::HashDB, (unsigned char)49>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#7  0x00007ffff706fe8d in kyotocabinet::PolyDB::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int) () from /usr/lib64/libkyotocabinet.so.16
#8  0x00007ffff7023c43 in kcdbopen () from /usr/lib64/libkyotocabinet.so.16
#9  0x00000000004b8ee7 in hcache_open_kc (h=0x8538c0,
    path=path@entry=0x70fd80 <hcpath> "/home/hurricanehrndz/.mutt/cache/01_degmac/headers/b67745c237197f722904d141dd776049") at hcache.c:1156
#10 0x00000000004ba070 in mutt_hcache_open (path=<optimized out>, folder=<optimized out>, namer=namer@entry=0x0) at hcache.c:1545
#11 0x000000000044de78 in maildir_delayed_parsing (ctx=ctx@entry=0x7fffffffbe50, md=md@entry=0x7fffffffba30, progress=progress@entry=0x7fffffffbae0)
    at mh.c:1156
#12 0x000000000044e80a in mh_read_dir (ctx=ctx@entry=0x7fffffffbe50, subdir=subdir@entry=0x4cd26a "new") at mh.c:1277
#13 0x000000000044ea3e in maildir_read_dir (ctx=0x7fffffffbe50) at mh.c:1304
#14 maildir_open_mailbox (ctx=0x7fffffffbe50) at mh.c:1312
#15 0x000000000045156a in mx_open_mailbox (path=<optimized out>, flags=flags@entry=9, pctx=pctx@entry=0x7fffffffbe50) at mx.c:679
#16 0x0000000000461997 in mutt_num_postponed (force=<optimized out>, force@entry=0) at postpone.c:139
#17 0x000000000047af14 in status_format_str (buf=buf@entry=0x7fffffffc360 "", buflen=buflen@entry=1024, col=0, cols=cols@entry=151,
    op=<optimized out>, src=0x7e4417 "", prefix=0x7fffffffc260 "", ifstring=0x7fffffffc2e0 "( %p postponed )", elsestring=0x7fffffffc760 "",
    data=8903952, flags=MUTT_FORMAT_OPTIONAL) at status.c:206
#18 0x0000000000485103 in mutt_FormatString (dest=dest@entry=0x7fffffffcd30 "", destlen=1023, destlen@entry=1024, col=col@entry=0,
    cols=cols@entry=151, src=<optimized out>, src@entry=0x7e4402 "%<p?( %p postponed )>", callback=callback@entry=0x47ac80 <status_format_str>,
    data=8903952, flags=MUTT_FORMAT_OPTIONAL) at muttlib.c:1568
#19 0x000000000048524a in mutt_FormatString (dest=0x7fffffffd6a0 "Folder: 01_degmac", '=' <repeats 21 times>, "   -16 messages (4 new)  ",
    destlen=1023, col=col@entry=0, cols=151, src=0x7e4401 " %<p?( %p postponed )>", callback=callback@entry=0x47ac80 <status_format_str>,
    data=8903952, flags=(unknown: 0)) at muttlib.c:1466
#20 0x000000000047b823 in menu_status_line (buf=buf@entry=0x7fffffffd6a0 "Folder: 01_degmac", '=' <repeats 21 times>, "   -16 messages (4 new)  ",
    buflen=buflen@entry=1024, menu=<optimized out>, p=<optimized out>) at status.c:326
#21 0x0000000000426085 in mutt_index_menu () at curs_main.c:918
#22 0x000000000040ab36 in main (argc=1, argv=<optimized out>) at main.c:882

@flatcap
Copy link
Member

flatcap commented Oct 9, 2016

Thanks for the improved debugging. That's the sort of backtrace I was hoping for.
Note: if you replace -O2 with -O0 (big o, zero), then variables won't get "".
You may need to pass "CFLAGS=-O0" to configure, for that.

I spent some time looking at this yesterday, but the news isn't good.
I carefully tweaked options and setup to best match yours, but I couldn't get neomutt to crash.

Then, I examined neomutt's call to the database. It creates a database object, then opens the database. The open call takes a path, but at open time, it doesn't do anything with it, AFAICS. As long as the path isn't NULL, the open succeeds.

Please can you compile and run this test program:

/* gcc -Wall -lkyotocabinet -o kc-open kc-open.c */
#include <stdio.h>
#include <kclangc.h>

int
main (int argc, char *argv[])
{
    if (argc != 2)
        return 1;

    KCDB *db = kcdbnew();
    if (db == NULL) {
        printf ("Couldn't create new database\n");
        return 1;
    }

    printf ("Opening database: %s\n", argv[1]);
    if (!kcdbopen (db, argv[1], KCOWRITER | KCOCREATE)) {
        printf ("Couldn't open database\n");
        return 1;
    }

    printf ("Success\n");

    kcdbdel (db);
    return 0;
}

And try running it with a variety of options:

  • ./kc-open file-that-exists
  • ./kc-open file-that-doesnt-exist
  • ./kc-open dir-that-exists
  • ./kc-open dir-that-doesnt-exist

@hurricanehrndz
Copy link
Author

Tests ran, always returned 0.

@flatcap
Copy link
Member

flatcap commented Oct 12, 2016

/* gcc -Wall -lkyotocabinet -o kc-open kc-open.c */
#include <stdio.h>
#include <kclangc.h>

int compress = 1;

int
main (int argc, char *argv[])
{
        if (argc != 2)
                return 1;

        KCDB *db = kcdbnew();
        if (db == NULL) {
                printf ("Couldn't create new database\n");
                return 1;
        }

        char buffer[1024];
        snprintf (buffer, sizeof(buffer), "%s#type=kct#opts=%s#rcomp=lex", argv[1], compress ? "lc" : "l");

        printf ("Opening database: %s\n", buffer);
        if (!kcdbopen (db, argv[1], KCOWRITER | KCOCREATE)) {
                printf ("Couldn't open database\n");
                return 1;
        }

        printf ("Success\n");

        kcdbdel (db);
        return 0;
}

@neverpanic
Copy link
Contributor

So, from further debugging: The crashing instruction is bextr, which requires the bmi1 CPU flag, available from Haswell or later. The CPU in question has the flag, so from the Intel manual I conclude (but may be wrong, since it's been a while) that the only other situations where bextr could cause an illegal instruction exception is incorrect opcode encoding, e.g. if VEX.L is not 0 or there's a different prefix before the VEX prefix in the opcode encoding.

@neverpanic
Copy link
Contributor

OK, this was an interesting ride: SuSE's libkyotocabinet packages (I didn't find the exact package used, but it's probably similar enough) seem to encode the instruction bextr $0x404,%rsi,%r8 as 8f 6a f8 10 c6 04 04. The 8f prefix indicates that this is an amd64 XOP-encoded instruction; AMD's manual suggests that the CPU additionally needs the TBM feature flag to support bextr.

Intel's manual does not indicate support for this type of XOP encoding at all.

I'm convinced this is not a problem with NeoMutt, but with the build of kyoto cabinet you are using. Please report this to your distribution.

@hurricanehrndz
Copy link
Author

@neverpanic Could please provide the documentation that states the prefix 8f indicates an amd64 XOP-encoded instruction?

@neverpanic
Copy link
Contributor

neverpanic commented Oct 13, 2016

@hurricanehrndz See the AMD64 Architecture Programmer’s Manual Volume 3: General-Purpose and System Instructions, section 1.2.8 VEX and XOP Prefixes, paragraph 2, sentence 2:

The three-byte escape sequences, initiated by the VEX C4h prefix or the XOP (8Fh) prefix, select the target opcode map explicitly via the VEX/XOP.map_select field.

Intel's equivalent, the Intel® 64 and IA-32 architectures software developer's manual combined volumes 2A, 2B, 2C, and 2D: Instruction set reference, A-Z has its documentation for the VEX prefix in section 2.3.5, paragraph 1, sentence 1, which does not mention 8Fh at all:

The VEX prefix is encoded in either the two-byte form (the first byte must be C5H) or in the three-byte form (the first byte must be C4H).

Intel's manual mentions the only encoding for the 64-bit variant of the bextr instruction is VEX.NDS.LZ.0F38.W1 F7 /r Section 3.1.1.2 has the details on how to read this:

  • VEX means there must be a VEX prefix. Since bextr requires encoding the VEX.mmmmm field to encode the 0F38 part of the instruction, we must use the 3-byte VEX prefix, which is c4.
  • NDS means that the VEX.vvv field encodes the first source register. I'll ignore the details, since we don't care.
  • LZ means that the VEX.L field must be 0.
  • 0F38 is a prefix that can be encoded using the VEX.mmmmm field. According to Figure 2-9, the field must be 00010b.
  • W1 means that the VEX.W field must be 1.
  • F7 seems to be a literal
  • /r means that the ModR/M byte contains a register operand and an r/m operand.

Note that this variant of bextr has the signature BEXTR r64a, r/m64, r64b, i.e. it specifically does not accept immediate values, which is what the library contains at this point.

Compare this with AMD's manual on bextr which has both a register form encoded with c4 ... (page 85, PDF page 121) and a form that takes an immediate value for the control field (page 87, PDF page 123), but is encoded with 8f ....

So, unless

  • I'm reading the wrong Intel documentation for your CPU
  • your CPU supports the 8Fh XOP encoding but doesn't document it
  • I'm too stupid to understand this stuff and have missed a relevant part of the documentation

I don't think that binary is compatible with Intel CPUs.

Do you disagree?

@hurricanehrndz
Copy link
Author

@neverpanic Thank you so much for the documentation. This will teach me a lot! I appreciate you taking the time. PS I have noticed most rpm base distros are compiling the kyoto library as x86 rather than x86_64 would this have any affect?

See here:
https://build.opensuse.org/package/view_file/openSUSE:Factory/kyotocabinet/kyotocabinet.spec?expand=1

@neverpanic
Copy link
Contributor

Yes, that would certainly have an effect, because there is no VEX prefix for x86, it only exists for x86_64. Consequently, the bextr instruction would not exist in 32-bit protected mode. However, that would mean that you could never build a 64-bit copy of mutt against this library, so I'd be surprised if there was no x86_64 version.

I'm not sure what effect the -march=i586 flag passed by the .spec file would have in an x86_64 build.

@neverpanic
Copy link
Contributor

But I think I see the problem; if you follow the links in the sidebar of https://build.opensuse.org/package/show/openSUSE:Factory/kyotocabinet to the latest x86_64 build and then open the complete log file at https://build.opensuse.org/public/build/openSUSE:Factory/standard/x86_64/kyotocabinet/_log, you'll see -march=native everywhere in the build, which will happily generate AMD64 instructions if the compiler is not a cross-compiler and the build machine happens to be an AMD64 machine.

@hurricanehrndz
Copy link
Author

@neverpanic,

Well thank you. Hopefully I have enough info to open an issue.

@neverpanic
Copy link
Contributor

In any case, it's obvious that this isn't a NeoMutt issue. I'm closing this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants