Internal memory fault at 0x00000000 when using python numpy library.

[Seb5ferrari]

Hi,

I am using graphene for sgx to run python scripts. I'm able to run all the native apps and bash scripts. Even python worked fine until I tried to import packages like numpy and pandas. I've changed my manifest by adding all the suggested file here #244 (comment), here #267, and here #308 (comment). None of them worked for me.

The files in the scripts folder 1) HelloWorld, 2) Test http, 3) Fibonacci, 4) Dummy web server are working fine and gives the required output. I get Internal memory fault only when I use numpy or panda. To test numpy, I used this file given by @ihepburn

Here are the log files. Thanks in advance.

fork_output.txt
HelloWorld_output.txt
Pyhton_dummy_webServer.txt
Python_csv.txt
Python_Fibonacci.txt
Python_Hello_world.txt
Python_numpy_output.txt
Python_Test_Http.txt

[ihepburn]

Hi @Seb5ferrari, in order to run NumPy in Graphene, you should allocate more than 1G memory.
Add the following setting to your manifest:
sgx.enclave_size = 1G

[Seb5ferrari]

I've increased it to 1G still no luck.
I copied the trusted files from this file.
I got the exit code 0 with HelloWorld and Fibonacci
But I'm getting Internal Memory Fault at 0x0 with NumPy, Pandas and SciKit etc.

Please see the attached files for more info.

hello_world_log.txt
hello_world_terminal_log.txt
numpy_log.txt
numpy_terminal_log.txt
python.manifest.template.txt

Thanks in advance.

[ihepburn]

@Seb5ferrari, can you paste the source code with NumPy script?

[Seb5ferrari]

Hi @ihepburn

This is the code taken from your script

import numpy as np
print np.arange(15).reshape(3,5)

[lanhaoxiang]

@ihepburn Same problem, any progress on this?

[Seb5ferrari]

I've cloned the latest repo and built it again. But no luck.

addr2line -e libm.so.6 0x1e423
/home/ubuntu/graphene/LibOS/glibc-2.19/math/../sysdeps/ieee754/dbl-64/s_sin.c:575

addr2line -e libpal-Linux-SGX.so 0x1e423
/home/ubuntu/graphene/Pal/lib/crypto/mbedtls/sha256.c:223 (discriminator 3)

addr2line -e libsysdb.so 0x1e423
/home/ubuntu/graphene/LibOS/shim/src/bookkeep/shim_signal.c:337

Still stuck. @lanhaoxiang any progress?

[Seb5ferrari]

@dimakuv Please look into this.
Got the same error in simulation mode as well.

Thanks in advance.

[kamathhrishi]

Any idea why this error is occurring? I am getting a similar error while importing TensorFlow. Numpy and scipy work fine. This happens in normal Linux and not with SGX, I tried increasing enclave size to 8G, but no luck.

[dimakuv]

@Seb5ferrari @kamathhrishi Have you looked into your examples provided with Graphene?

Please check:

• https://github.com/oscarlab/graphene-tests/tree/master/python-scipy-insecure
• https://github.com/oscarlab/graphene-tests/tree/master/tensorflow

If you experience problems on these examples, please tell us. If you experience problems on your own versions of NumPy and TensorFlow, please provide more details: attach logs in debug mode, attach the manifest, describe the OS/CPU and specific versions of all relevant software.

[kamathhrishi]

@dimakuv I did try those examples.

I am running it on Ubuntu 16.04 and all I ran inside python was a program that imported tensorflow (just a .py file with import tensorflow)

I got the error while trying to import tensorflow inside Python. I had got the following error

Internal memory fault at 0x78d6f4b7e000 (IP = +0x31f49, VMID = 1456744796, TID = 1)
Trace/breakpoint trap (core dumped)

below being the debug log before the error occured.

[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/misc/doccer.py",0x78d736b36960) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/misc/doccer.py",0x78d736b36f40) = 0
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/misc/__pycache__/doccer.cpython-35.pyc",O_RDONLY|2000000,0000) = 7
[     1] ---- shim_fstat(7,0x78d736b37040) = 0
[     1] ---- shim_lseek(7,0,SEEK_CUR) = 0
[     1] ---- shim_fstat(7,0x78d736b37080) = 0
[     1] ---- shim_read(7,0x78d6f61cc7a0,5933) ... read
[     1] bkeep_unmapped: 0x7f4f6651d000-0x7f4f66521000 => filebuf
[     1] ---- return from shim_read(...) = 5932
[     1] ---- shim_read(7,0x78d6f61cdecc,1) ... read
[     1] ---- return from shim_read(...) = 0
[     1] bkeep_munmap: 0x7f4f6651d000-0x7f4f66521000
[     1] ---- shim_close(7) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/misc",0x78d736b372a0) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/misc/common.py",0x78d736b37040) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/misc/common.py",0x78d736b37620) = 0
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/misc/__pycache__/common.cpython-35.pyc",O_RDONLY|2000000,0000) = 7
[     1] ---- shim_fstat(7,0x78d736b37720) = 0
[     1] ---- shim_lseek(7,0,SEEK_CUR) = 0
[     1] ---- shim_fstat(7,0x78d736b37760) = 0
[     1] ---- shim_read(7,0x78d6f619ea10,11996) ... read
[     1] bkeep_unmapped: 0x7f4f6651d000-0x7f4f66521000 => filebuf
[     1] ---- return from shim_read(...) = 11995
[     1] ---- shim_read(7,0x78d6f61a18eb,1) ... read
[     1] ---- return from shim_read(...) = 0
[     1] bkeep_munmap: 0x7f4f6651d000-0x7f4f66521000
[     1] ---- shim_close(7) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/_lib",0x78d736b364f0) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/_lib/_util.py",0x78d736b36290) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/_lib/_util.py",0x78d736b36870) = 0
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/_lib/__pycache__/_util.cpython-35.pyc",O_RDONLY|2000000,0000) = 7
[     1] ---- shim_fstat(7,0x78d736b36970) = 0
[     1] ---- shim_lseek(7,0,SEEK_CUR) = 0
[     1] ---- shim_fstat(7,0x78d736b369b0) = 0
[     1] ---- shim_read(7,0x78d6f61cc7a0,11544) ... read
[     1] bkeep_unmapped: 0x7f4f6651d000-0x7f4f66521000 => filebuf
[     1] ---- return from shim_read(...) = 11543
[     1] ---- shim_read(7,0x78d6f61cf4b7,1) ... read
[     1] ---- return from shim_read(...) = 0
[     1] bkeep_munmap: 0x7f4f6651d000-0x7f4f66521000
[     1] ---- shim_close(7) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy",0x78d736b372a0) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/__init__.cpython-35m-x86_64-linux-gnu.so",0x78d736b37040) = -2
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/__init__.abi3.so",0x78d736b37040) = -2
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/__init__.so",0x78d736b37040) = -2
[     1] bkeep_unmapped: 0x7f4f66500000-0x7f4f66521000 => slab
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/__init__.py",0x78d736b37040) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/__init__.py",0x78d736b37620) = 0
[     1] bkeep_unmapped: 0x7f4f664fe000-0x7f4f66500000 => slab
[     1] bkeep_munmap: 0x7f4f664fe000-0x7f4f66500000
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/special/__pycache__/__init__.cpython-35.pyc",O_RDONLY|2000000,0000) = 7
[     1] ---- shim_fstat(7,0x78d736b37720) = 0
[     1] ---- shim_lseek(7,0,SEEK_CUR) = 0
[     1] ---- shim_fstat(7,0x78d736b37760) = 0
[     1] ---- shim_read(7,0x78d6f5ba80d0,23384) ... read
[     1] bkeep_unmapped: 0x7f4f664f8000-0x7f4f66500000 => filebuf
[     1] ---- return from shim_read(...) = 23383
[     1] ---- shim_read(7,0x78d6f5badc27,1) ... read
[     1] ---- return from shim_read(...) = 0
[     1] bkeep_munmap: 0x7f4f664f8000-0x7f4f66500000
[     1] ---- shim_close(7) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special",0x78d736b365b0) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special",0x78d736b36030) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special",0x78d736b364f0) = 0
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/special",O_RDONLY|2204000,0000) = 7
[     1] ---- shim_fstat(7,0x78d736b36480) = 0
[     1] bkeep_unmapped: 0x7f4f664fe000-0x7f4f66500000 => slab
[     1] bkeep_munmap: 0x7f4f664fe000-0x7f4f66500000
[     1] bkeep_unmapped: 0x7f4f664fe000-0x7f4f66500000 => slab
[     1] bkeep_munmap: 0x7f4f664fe000-0x7f4f66500000
[     1] bkeep_unmapped: 0x7f4f664fe000-0x7f4f66500000 => slab
[     1] bkeep_munmap: 0x7f4f664fe000-0x7f4f66500000
[     1] bkeep_unmapped: 0x7f4f664df000-0x7f4f66500000 => slab
[     1] ---- shim_getdents(7,0x78d6f5ba80e0,32768) = 986
[     1] ---- shim_getdents(7,0x78d6f5ba80e0,32768) = 0
[     1] ---- shim_close(7) = 0
[     1] ---- shim_stat("/usr/lib/python3.5/site-packages/scipy/special/_ufuncs.cpython-35m-x86_64-linux-gnu.so",0x78d736b36290) = 0
[     1] ---- shim_openat(AT_FDCWD,"/usr/lib/python3.5/site-packages/scipy/special/_ufuncs.cpython-35m-x86_64-linux-gnu.so",O_RDONLY|2000000,0000) = 7
[     1] ---- shim_read(7,0x78d736b35e58,832) ... read
[     1] bkeep_unmapped: 0x7f4f664db000-0x7f4f664df000 => filebuf
[     1] ---- return from shim_read(...) = 832
[     1] ---- shim_lseek(7,4706304,SEEK_SET) = 4706304
[     1] ---- shim_read(7,0x78d736b35b60,504) ... read
[     1] bkeep_munmap: 0x7f4f664db000-0x7f4f664df000
[     1] bkeep_unmapped: 0x7f4f664db000-0x7f4f664df000 => filebuf
[     1] ---- return from shim_read(...) = 504
[     1] ---- shim_fstat(7,0x78d736b35cf0) = 0
[     1] ---- shim_lseek(7,4706304,SEEK_SET) = 4706304
[     1] ---- shim_read(7,0x78d736b35a70,504) ... read
[     1] ---- return from shim_read(...) = 504
[     1] ---- shim_mmap(0x0,3749216,PROT_READ|PROT_EXEC,MAP_PRIVATE|0x800,7,0) ... mmap
[     1] bkeep_unmapped: 0x78d6f4701000-0x78d6f4a95000
[     1] heap top adjusted to 0x78d6f4701000
[     1] ---- return from shim_mmap(...) = 0x78d6f4701000
[     1] ---- shim_mprotect(0x78d6f487b000,2191360,PROT_NONE) ... mprotect
[     1] bkeep_mprotect: 0x78d6f487b000-0x78d6f4a92000
[     1] ---- return from shim_mprotect(...) = 0
[     1] ---- shim_mmap(0x78d6f4a7b000,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|0x800,7,1548288) ... mmap
[     1] mmap: allowing overlapping MAP_FIXED allocation at 0x78d6f4a7b000 with length 32768
[     1] bkeep_mmap: 0x78d6f4a7b000-0x78d6f4a83000
[     1] ---- return from shim_mmap(...) = 0x78d6f4a7b000
[     1] ---- shim_mmap(0x78d6f4a83000,33176,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_FIXED,-1,0) ... mmap
[     1] mmap: allowing overlapping MAP_FIXED allocation at 0x78d6f4a83000 with length 36864
[     1] bkeep_mmap: 0x78d6f4a83000-0x78d6f4a8c000
[     1] ---- return from shim_mmap(...) = 0x78d6f4a83000
[     1] ---- shim_mmap(0x78d6f4a8c000,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|0x800,7,4681728) ... mmap
[     1] mmap: allowing overlapping MAP_FIXED allocation at 0x78d6f4a8c000 with length 12288
[     1] bkeep_mmap: 0x78d6f4a8c000-0x78d6f4a8f000
[     1] ---- return from shim_mmap(...) = 0x78d6f4a8c000
[     1] ---- shim_mmap(0x78d6f4a8f000,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|0x800,7,4694016) ... mmap
[     1] mmap: allowing overlapping MAP_FIXED allocation at 0x78d6f4a8f000 with length 12288
[     1] bkeep_mmap: 0x78d6f4a8f000-0x78d6f4a92000
[     1] ---- return from shim_mmap(...) = 0x78d6f4a8f000
[     1] ---- shim_mmap(0x78d6f4a92000,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|0x800,7,4706304) ... mmap
[     1] mmap: allowing overlapping MAP_FIXED allocation at 0x78d6f4a92000 with length 12288
[     1] bkeep_mmap: 0x78d6f4a92000-0x78d6f4a95000
[     1] ---- return from shim_mmap(...) = 0x78d6f4a92000
[     1] ---- shim_close(7) = 0
[     1] glibc register library /usr/lib/python3.5/site-packages/scipy/special/_ufuncs.cpython-35m-x86_64-linux-gnu.so loaded at 0x78d6f4701000

[kamathhrishi]

This was my manifest file

share.txt

[dimakuv]

I am confused by your two statements:

I am running it on Ubuntu 16.04 ...

I got the error on a Mac while trying to import tensorflow inside Python.

So what are using? Is it a VM with Ubuntu 16.04 on a MacOS being a VMM? Or did you try it on two different computers? Or do you use Ubuntu 16.04 on a Mac machine?

[kamathhrishi]

@dimakuv Sorry for the confusion, its a Ubuntu 16.04 itself.

[dimakuv]

I see that you have stack size of 100MB. This seems like a very big number. Could you decrease it to 1-2MB and rebuild your Python example again? It could be the reason for this bug you see.

[kamathhrishi]

No luck :/

[duanbing]

Any progress on this issue? we have the same error when we run TensorFlow. @dimakuv @kamathhrishi

[mkow]

@duanbing: Would you be able to produce a small repro for this?

[duanbing]

@mkow I fixed it by increasing the enclave_size as far as I recall.

[dimakuv]

Ok, let's close this issue then.