Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unaligned stack with Xcode 11.0 beta #848

Closed
stv0g opened this issue Jul 23, 2019 · 35 comments
Closed

Unaligned stack with Xcode 11.0 beta #848

stv0g opened this issue Jul 23, 2019 · 35 comments

Comments

@stv0g
Copy link

stv0g commented Jul 23, 2019

During make check the pwhash_argon2i and pwhash_argon2id tests fail with a segmention fault.

Tested on

  • libsodium 0.18.0-RELEASE and fdfca24
  • macOS Catalina 10.15 Beta (19A512f)
  • config.log

Compiler

gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.0 (clang-1100.0.31.5)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Backtraces

pwhash_argon2i

81 $ sudo glibtool --mode=execute lldb test/default/pwhash_argon2i
(lldb) target create "/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2i"
Current executable set to '/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2i' (x86_64).
(lldb) r
Process 85912 launched: '/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2i' (x86_64)
Process 85912 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00007fff704a3ac6 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff704a3ac6 <+0>: movdqa %xmm0, (%rsp)
    0x7fff704a3acb <+5>: int3
    0x7fff704a3acc <+6>: nop
    0x7fff704a3acd <+7>: nop
Target 0: (pwhash_argon2i) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00007fff704a3ac6 libdyld.dylib`stack_not_16_byte_aligned_error
    frame #1: 0x00007ffeefbfd130
    frame #2: 0x0000000100169661 libsodium.23.dylib`_sodium_argon2_fill_memory_blocks(instance=0x00007ffeefbfd180, pass=0) at argon2-core.c:242:13 [opt]
    frame #3: 0x000000010016c88a libsodium.23.dylib`_sodium_argon2_ctx(context=0x00007ffeefbfd1e0, type=<unavailable>) at argon2.c:76:9 [opt]
    frame #4: 0x000000010016c994 libsodium.23.dylib`_sodium_argon2_hash(t_cost=5, m_cost=7086, parallelism=1, pwd=<unavailable>, pwdlen=127, salt=0x00007ffeefbfd670, saltlen=16, hash=0x00007ffeefbfd570, hashlen=155, encoded=0x0000000000000000, encodedlen=0, type=Argon2_i) at argon2.c:129:14 [opt]
    frame #5: 0x000000010016ca88 libsodium.23.dylib`_sodium_argon2i_hash_raw(t_cost=<unavailable>, m_cost=<unavailable>, parallelism=<unavailable>, pwd=<unavailable>, pwdlen=<unavailable>, salt=<unavailable>, saltlen=16, hash=0x00007ffeefbfd570, hashlen=155) at argon2.c:176:12 [opt]
    frame #6: 0x000000010016d179 libsodium.23.dylib`crypto_pwhash_argon2i(out="", outlen=155, passwd="�G�\x92���oYZD\x80�\x9c/���\x14\x8d7\x1e\x94\x87�_\\#", passwdlen=127, salt="UA�ɕ��\x97�)\x03F��YޣG�\x92���oYZD\x80�\x9c/���\x14\x8d7\x1e\x94\x87�_\\#", opslimit=5, memlimit=7256678, alg=1) at pwhash_argon2i.c:168:13 [opt]
    frame #7: 0x000000010016d97e libsodium.23.dylib`crypto_pwhash(out=<unavailable>, outlen=<unavailable>, passwd=<unavailable>, passwdlen=<unavailable>, salt=<unavailable>, opslimit=<unavailable>, memlimit=7256678, alg=1) at crypto_pwhash.c:136:16 [opt]
    frame #8: 0x0000000100000bab pwhash_argon2i`xmain [inlined] tv at pwhash_argon2i.c:93:13 [opt]
    frame #9: 0x0000000100000ae9 pwhash_argon2i`xmain at pwhash_argon2i.c:401 [opt]
    frame #10: 0x0000000100000a23 pwhash_argon2i`main at cmptest.h:190:9 [opt]
    frame #11: 0x00007fff704b2c35 libdyld.dylib`start + 1

pwhash_argon2id

82 $ sudo glibtool --mode=execute lldb test/default/pwhash_argon2id
(lldb) target create "/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2id"
Current executable set to '/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2id' (x86_64).
(lldb) r
Process 85971 launched: '/Users/stv0g/workspace/libsodium/test/default/.libs/pwhash_argon2id' (x86_64)
Process 85971 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00007fff704a3ac6 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff704a3ac6 <+0>: movdqa %xmm0, (%rsp)
    0x7fff704a3acb <+5>: int3
    0x7fff704a3acc <+6>: nop
    0x7fff704a3acd <+7>: nop
Target 0: (pwhash_argon2id) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00007fff704a3ac6 libdyld.dylib`stack_not_16_byte_aligned_error
    frame #1: 0x00007ffeefbfd130
    frame #2: 0x000000010016b661 libsodium.23.dylib`_sodium_argon2_fill_memory_blocks(instance=0x00007ffeefbfd180, pass=0) at argon2-core.c:242:13 [opt]
    frame #3: 0x000000010016e88a libsodium.23.dylib`_sodium_argon2_ctx(context=0x00007ffeefbfd1e0, type=<unavailable>) at argon2.c:76:9 [opt]
    frame #4: 0x000000010016e994 libsodium.23.dylib`_sodium_argon2_hash(t_cost=5, m_cost=7086, parallelism=1, pwd=<unavailable>, pwdlen=127, salt=0x00007ffeefbfd670, saltlen=16, hash=0x00007ffeefbfd570, hashlen=155, encoded=0x0000000000000000, encodedlen=0, type=Argon2_id) at argon2.c:129:14 [opt]
    frame #5: 0x000000010016eac8 libsodium.23.dylib`_sodium_argon2id_hash_raw(t_cost=<unavailable>, m_cost=<unavailable>, parallelism=<unavailable>, pwd=<unavailable>, pwdlen=<unavailable>, salt=<unavailable>, saltlen=16, hash=0x00007ffeefbfd570, hashlen=155) at argon2.c:197:12 [opt]
    frame #6: 0x000000010016f668 libsodium.23.dylib`crypto_pwhash_argon2id(out="", outlen=155, passwd="�G�\x92���oYZD\x80�\x9c/���\x14\x8d7\x1e\x94\x87�_\\#", passwdlen=127, salt="UA�ɕ��\x97�)\x03F��YޣG�\x92���oYZD\x80�\x9c/���\x14\x8d7\x1e\x94\x87�_\\#", opslimit=5, memlimit=7256678, alg=2) at pwhash_argon2id.c:164:13 [opt]
    frame #7: 0x000000010016f989 libsodium.23.dylib`crypto_pwhash(out=<unavailable>, outlen=<unavailable>, passwd=<unavailable>, passwdlen=<unavailable>, salt=<unavailable>, opslimit=<unavailable>, memlimit=7256678, alg=2) at crypto_pwhash.c:139:16 [opt]
    frame #8: 0x000000010000098b pwhash_argon2id`xmain [inlined] tv at pwhash_argon2id.c:93:13 [opt]
    frame #9: 0x00000001000008c9 pwhash_argon2id`xmain at pwhash_argon2id.c:399 [opt]
    frame #10: 0x0000000100000803 pwhash_argon2id`main at cmptest.h:190:9 [opt]
    frame #11: 0x00007fff704b2c35 libdyld.dylib`start + 1
@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

Some more details after debugging:

libsodium throws a general protection fault due to an unaligned memory access in argon2_fill_memory_blocks, or more specifically on my system in argon2_fill_segment_avx512f.

Arguments at time of unaligned access:

print instance
(argon2_instance_t) $3 = {
  region = 0x0000000100202a60
  pseudo_rands = 0x0000000100802e00
  passes = 5
  current_pass = 4294967295
  memory_blocks = 7084
  segment_length = 1771
  lane_length = 7084
  lanes = 1
  threads = 1
  type = Argon2_id
  print_internals = -272641584
}
position.lane = 0
position.index = 0

@stv0g stv0g changed the title Tests fail on macOS Catalina Beta 3: pwhash_argon2i & pwhash_argon2id Unaligned memory access in argon2_fill_memory_blocks Jul 23, 2019
jedisct1 added a commit that referenced this issue Jul 23, 2019
@jedisct1
Copy link
Owner

The stack not being aligned is quite unexpected and sounds like a compiler bug.

How did you compile it? What version of Xcode did you use? Any specific extra optimizations, or did you just use ./configure?

@jedisct1
Copy link
Owner

Meanwhile, this code path has been disabled in the stable branch.

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

How did you compile it?

Just plain autoreconf -i && ./configure && make && make check.

Any specific extra optimizations, or did you just use ./configure?

No special options for ./configure

What version of Xcode did you use?

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version
version: 11.0.0.0.1.1563062694

This is the latest beta of macOS Catalina. See also config.log and the compiler version in my first post.

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

Meanwhile, this code path has been disabled in the stable branch.

Disabled in stable, but not master / 0.18.0?

@jedisct1
Copy link
Owner

That's exactly what stable versions are for.

Can you try different optimization levels, including no optimizations at all (./configure --debug) and see if it makes a difference?

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

I see the same results (pwhash_argon2i failing) when compiling with:

  • ./configure --enable-debug
  • CFLAGS=-O1 ./configure

When compiling with CFLAGS=-O0 ./configure, even more tests fail.

bors bot added a commit to sodiumoxide/sodiumoxide that referenced this issue Jul 23, 2019
355: Fix path to configure script r=kpp a=stv0g

See #341

===

jedisct1/libsodium#848

Co-authored-by: Steffen Vogel <post@steffenvogel.de>
@jedisct1
Copy link
Owner

o_O

What other tests are failing with -O0?

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

All fail with a segmentation fault:

FAIL: box_seal
FAIL: generichash
FAIL: generichash2
FAIL: generichash3
FAIL: kdf
FAIL: kx
FAIL: metamorphic
FAIL: pwhash_argon2i
FAIL: pwhash_argon2id
FAIL: sodium_utils2

@jedisct1
Copy link
Owner

Try the compiler version shipped with Xcode (xcode-select /Application/Xcode or ... /Xcode-beta).

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

I can confirm that all tests succeed when using compiler from my full Xcode installation.
But when using the command line tools, it fails.

Here are again the versions:

/Applications/Xcode.app/

$ sudo xcode-select -s /Applications/Xcode.app/
$ /usr/bin/gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

State: pwhash_argon2i tests are succeeding!

/Applications/Xcode-beta.app/

$ sudo xcode-select -s /Applications/Xcode-beta.app/
$ gcc --version
Configured with: --prefix=/Applications/Xcode-beta.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode-beta.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.0 (clang-1100.0.31.5)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

State: pwhash_argon2i tests are failing

/Library/Developer/CommandLineTools/

$ sudo xcode-select -s /Library/Developer/CommandLineTools/
$ /usr/bin/gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.0 (clang-1100.0.31.5)
Target: x86_64-apple-darwin19.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

State: pwhash_argon2i tests are failing

As you can see, Xcode uses Clang 10 while my Command Line Tools use Clang 11.. o_O

I am downloading now Xcode 11 Beta 4. And try again. I assume that my Command Line Tools already have been updated automatically, while Xcode stayed at version 10.

@jedisct1
Copy link
Owner

Xcode 11 beta 4 ships with what Apple calls LLVM 11, but it's a different version than the one you reported.

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

I updated my previous comment with results using Xcode Beta 4.
And my assumption was confirmed. My command line tools use the same version of LLVM/Clang than the new Xcode beta.

@kpp
Copy link

kpp commented Jul 23, 2019

So what's the state of the issue?

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

The compiler from both Xcode 11 Beta and the Command Line Tools produce code which segfaults in the pwhash_argon2i tests.

I provided Apple some feedback via their Beta Feedback App. I am not entirely sure that this is a compiler bug. Looks strange to me, that only the pwhash_argon2i tests are failing. I guess broken stack alignment would be an issue for much more tests..

@jedisct1
Copy link
Owner

Mmmm... my version of clang with Xcode 11 beta is clang-1100.0.20.17.
`

@stv0g
Copy link
Author

stv0g commented Jul 23, 2019

@jedisct1 Does this version work for you?

@kpp
Copy link

kpp commented Jul 23, 2019

What about macOS Catalina 10.15 Beta (19A512f)? Is it the only difference in your envs?

@jedisct1
Copy link
Owner

It does, but I don't have a good CPU like yours. No AVX512 here.

@jedisct1
Copy link
Owner

This compiler version is completely broken; I installed it and the first project I compiled with it, that has nothing to do with libsodium, crashed the same way (unaligned stack).

@kpp
Copy link

kpp commented Aug 24, 2019

Was it fixed in a new version of the compiler?

@jedisct1
Copy link
Owner

Unfortunately not. And nobody reported it on the Apple developer forums.

@jedisct1
Copy link
Owner

Compiling without optimizations (env CFLAGS="-O0") triggers many more of these.

@jedisct1 jedisct1 changed the title Unaligned memory access in argon2_fill_memory_blocks Unaligned memory access with Xcode 11.0 beta Aug 26, 2019
@jedisct1 jedisct1 changed the title Unaligned memory access with Xcode 11.0 beta Unaligned stack with Xcode 11.0 beta Aug 26, 2019
@kpp
Copy link

kpp commented Aug 26, 2019

Unfortunately not.

Why did you close the issue then?

@jedisct1
Copy link
Owner

jedisct1 commented Aug 26, 2019

Unfortunately, keeping an issue opened here will not do anything,

The solution is simply to use the stable version of Xcode.

@kpp
Copy link

kpp commented Aug 26, 2019

Ok now it sounds as a resolution

@jedisct1
Copy link
Owner

With this version, dsvpn segfaults even before reaching the main() function :/

@jedisct1
Copy link
Owner

jedisct1 commented Aug 26, 2019

Here's a test case, in case someone wants to report this to Apple so that it is fixed.

#include <fcntl.h>
#include <netdb.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

typedef struct {
    char  d[16];
    void *e;
    struct { char b[5536]; } f;
} i;

void g(void) {
    struct addrinfo hints, *k;
    memset(&hints, 0, sizeof hints);
    getaddrinfo(NULL, NULL, &hints, &k);
}

int main(void) {
    puts("Hello world");
    fflush(stdout);
    close(open("/dev/null", O_RDONLY));
    i context;
    context.e = open;
    printf("%p\n", context.d);
    g();
    return 0;
}
cc -mavx -O2 a.c && ./a.out
lldb ./a.out
run

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x00007fff65e73316 libdyld.dylib`stack_not_16_byte_aligned_error
libdyld.dylib`stack_not_16_byte_aligned_error:
->  0x7fff65e73316 <+0>: movdqa %xmm0, (%rsp)

The stack pointer becomes invalid even before we reach the main() function.

This doesn't happen without AVX, or with stable versions of the compiler.

@jedisct1
Copy link
Owner

A workaround is to compile with -ffreestanding. But this is just a workaround.

@zwaldowski
Copy link

It seems like adding -fno-stack-check to CFLAGS also alleviates the problem, making me think it's not a miscompute but a change in defaults.

In the release notes here, you can see:

  • Stack checking is on by default on all platforms to prevent memory corruptions. (25859140)

@jedisct1
Copy link
Owner

jedisct1 commented Sep 4, 2019

The compiler emits invalid code. This has nothing to do with libsodium, see the test case above.

@jedisct1
Copy link
Owner

jedisct1 commented Sep 4, 2019

See https://forums.developer.apple.com/thread/121887

This compiler version is broken, has obviously received insufficient testing, and even with additional compilation flags, I wouldn’t recommend using it for anything serious.

@ELLIOTTCABLE
Copy link

FYI, this is still an issue with the release version of Catalina; as the above cross-refs show, it's showing up in a lot of other libraries, as well. (Hello from bcmyers/argonautica#20!)

@eliaslevy
Copy link

FWIW, this issue appears fixed in Xcode 11.2 and the crate builds successfully with it. Note that if you had installed the command line tools, using sudo xcode-select -s /Applications/Xcode.app after updating Xcode is insufficient. You must remove the command line tools (sudo rm -rf /Library/Developer/CommandLineTools) or the build will still fail.

@nlfiedler
Copy link

I can confirm that the suggestion by eliaslevy works as advertised. Thanks.

Repository owner locked and limited conversation to collaborators Nov 23, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants