Apache | Couldn't create the mpm-accept mutex

[dzygann]

Hi,

I have tried to run the Apache example with this command:

SGX=1 make start-gramine-server & 

I've got the following error :

Listen on 127.0.0.1:8001
gramine-sgx ./httpd -D FOREGROUND \
        -f conf/httpd-gramine.conf \
        -C "LoadModule mpm_worker_module modules/mod_mpm_worker.so" \
        -C "Listen 127.0.0.1:8001" \
        -C "ServerName 127.0.0.1" \
        -C "PidFile logs/httpd-127.0.0.1-8001.pid"
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.log_level = warning|debug|trace|all (verbose log level, may leak information)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

[Wed Jan 26 20:03:52.118415 2022] [core:warn] [pid 1:tid 358529984] (92)Protocol not available: AH00076: Failed to enable APR_TCP_DEFER_ACCEPT
[Wed Jan 26 20:03:52.118535 2022] [core:warn] [pid 1:tid 358529984] (92)Protocol not available: AH00076: Failed to enable APR_TCP_DEFER_ACCEPT
[Wed Jan 26 20:03:52.119930 2022] [core:emerg] [pid 1:tid 358529984] (38)Function not implemented: AH00023: Couldn't create the mpm-accept mutex 
(38)Function not implemented: could not create accept mutex
AH00015: Unable to open logs
make: *** [Makefile:202: start-gramine-multithreaded-server] Error 1

(38)Function not implemented: could not create accept mutex this one seems to be necessary. Does this example work with Gramine SGX enabled?

Here is the debug log.
In the end, some files are removed. Is this a valid behaviour?
The system call semget is not implemented. Is this causing the issue?

[P1:T1:httpd] debug: Allocating stack at 0 (size = 262144)
[P1:T1:httpd] debug: loading "file:./install/bin/httpd"
[P1:T1:httpd] debug: append_r_debug: adding file:./install/bin/httpd at 0x16307000
[P1:T1:httpd] debug: find_interp: searching for interpreter: /lib/ld-linux-x86-64.so.2
[P1:T1:httpd] debug: loading "file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2"
[P1:T1:httpd] debug: append_r_debug: adding file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/ld-linux-x86-64.so.2 at 0x162d3000
[P1:T1:httpd] debug: Creating pipe: pipe.srv:1
debug: sock_getopt (fd = 14, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
[P1:T1:httpd] debug: Shim process initialized
[P1:shim] debug: IPC worker started
[P1:T1:httpd] warning: Not supported flag (0x3001) passed to arch_prctl
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libpcre.so.3 loaded at 0x1625d000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libpcre.so.3 at 0x1625d000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libaprutil-1.so.0 loaded at 0x1622f000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libaprutil-1.so.0 at 0x1622f000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libapr-1.so.0 loaded at 0x161f6000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libapr-1.so.0 at 0x161f6000
[P1:T1:httpd] debug: glibc register library /lib/libpthread.so.0 loaded at 0x161f1000
[P1:T1:httpd] debug: append_r_debug: adding file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/libpthread.so.0 at 0x161f1000
[P1:T1:httpd] debug: glibc register library /lib/libc.so.6 loaded at 0x15ff7000
[P1:T1:httpd] debug: append_r_debug: adding file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/libc.so.6 at 0x15ff7000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libcrypt.so.1 loaded at 0x15fbc000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libcrypt.so.1 at 0x15fbc000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libexpat.so.1 loaded at 0x15f8c000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libexpat.so.1 at 0x15f8c000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libuuid.so.1 loaded at 0x15f83000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libuuid.so.1 at 0x15f83000
[P1:T1:httpd] debug: glibc register library /lib/libdl.so.2 loaded at 0x15f7e000
[P1:T1:httpd] debug: append_r_debug: adding file:/usr/local/lib/x86_64-linux-gnu/gramine/runtime/glibc/libdl.so.2 at 0x15f7e000
[P1:T1:httpd] warning: shim_socket: unknown socket domain 16
debug: sock_getopt (fd = 24, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
[P1:T1:httpd] debug: shim_connect: reconnect on a stream socket
debug: sock_getopt (fd = 25, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_mpm_prefork.so loaded at 0x15f38000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_mpm_prefork.so at 0x15f38000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authn_file.so loaded at 0x15f31000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authn_file.so at 0x15f31000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authn_core.so loaded at 0x15f2c000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authn_core.so at 0x15f2c000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authz_host.so loaded at 0x15f24000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authz_host.so at 0x15f24000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authz_groupfile.so loaded at 0x15f1e000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authz_groupfile.so at 0x15f1e000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authz_user.so loaded at 0x15f19000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authz_user.so at 0x15f19000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_authz_core.so loaded at 0x15f10000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_authz_core.so at 0x15f10000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_access_compat.so loaded at 0x15f09000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_access_compat.so at 0x15f09000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_auth_basic.so loaded at 0x15f03000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_auth_basic.so at 0x15f03000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_reqtimeout.so loaded at 0x15efd000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_reqtimeout.so at 0x15efd000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_filter.so loaded at 0x15ef4000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_filter.so at 0x15ef4000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_mime.so loaded at 0x15eeb000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_mime.so at 0x15eeb000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_log_config.so loaded at 0x15ee0000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_log_config.so at 0x15ee0000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_env.so loaded at 0x15edb000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_env.so at 0x15edb000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_headers.so loaded at 0x15ed3000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_headers.so at 0x15ed3000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_setenvif.so loaded at 0x15ecb000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_setenvif.so at 0x15ecb000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_version.so loaded at 0x15ec6000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_version.so at 0x15ec6000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_ssl.so loaded at 0x15e71000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_ssl.so at 0x15e71000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libssl.so.1.1 loaded at 0x15dde000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libssl.so.1.1 at 0x15dde000
[P1:T1:httpd] debug: glibc register library /lib/x86_64-linux-gnu/libcrypto.so.1.1 loaded at 0x15b08000
[P1:T1:httpd] debug: append_r_debug: adding file:/lib/x86_64-linux-gnu/libcrypto.so.1.1 at 0x15b08000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_unixd.so loaded at 0x15afe000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_unixd.so at 0x15afe000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_status.so loaded at 0x15af4000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_status.so at 0x15af4000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_autoindex.so loaded at 0x15ae7000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_autoindex.so at 0x15ae7000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_dir.so loaded at 0x15adf000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_dir.so at 0x15adf000
[P1:T1:httpd] debug: glibc register library /home/developer/workspaces/enclaive/examples/apache/install/modules/mod_alias.so loaded at 0x15ad7000
[P1:T1:httpd] debug: append_r_debug: adding file:./install/modules/mod_alias.so at 0x15ad7000
[P1:T1:httpd] warning: shim_socket: unknown socket domain 16
[P1:T1:httpd] warning: shim_socket: unknown socket domain 16
[P1:T1:httpd] warning: shim_socket: unknown socket domain 16
debug: sock_getopt (fd = 26, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
debug: sock_getopt (fd = 51, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
[P1:T1:httpd] debug: Creating pipe: pipe.srv:1b98f986340392a97c7c5c47ae431e01e738b38e9114428194e2f6d485564d1c
debug: sock_getopt (fd = 52, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
debug: sock_getopt (fd = 53, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
debug: sock_getopt (fd = 54, sockopt addr = 0x7fffaaa5dc10) is not implemented and always returns 0
[P1:T1:httpd] warning: Unsupported system call semget
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_alias.so at 0x15ad7000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_dir.so at 0x15adf000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_autoindex.so at 0x15ae7000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_status.so at 0x15af4000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_unixd.so at 0x15afe000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_ssl.so at 0x15e71000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_version.so at 0x15ec6000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_setenvif.so at 0x15ecb000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_headers.so at 0x15ed3000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_env.so at 0x15edb000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_log_config.so at 0x15ee0000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_mime.so at 0x15eeb000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_filter.so at 0x15ef4000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_reqtimeout.so at 0x15efd000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_auth_basic.so at 0x15f03000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_access_compat.so at 0x15f09000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authz_core.so at 0x15f10000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authz_user.so at 0x15f19000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authz_groupfile.so at 0x15f1e000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authz_host.so at 0x15f24000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authn_core.so at 0x15f2c000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_authn_file.so at 0x15f31000
[P1:T1:httpd] debug: remove_r_debug: removing file:./install/modules/mod_mpm_prefork.so at 0x15f38000
[P1:T1:httpd] debug: ---- shim_exit_group (returning 1)
[P1:T1:httpd] debug: clearing POSIX locks for pid 1
[P1:T1:httpd] debug: sync client shutdown: closing handles
[P1:T1:httpd] debug: sync client shutdown: waiting for confirmation
[P1:T1:httpd] debug: sync client shutdown: finished
[P1:shim] debug: IPC worker: exiting worker thread
[P1:T1:httpd] debug: process 1 exited with status 1
debug: DkProcessExit: Returning exit code 1

The example runs smoothly if I run it without the SGX=1 option at the start.

[dimakuv]

[P1:T1:httpd] warning: Unsupported system call semget

This is a problem. Sys-V semaphores (https://man7.org/linux/man-pages/man2/semget.2.html) are not supported currently in Gramine.

The example runs smoothly if I run it without the SGX=1 option at the start.

So you mean the example runs on gramine-direct but not on gramine-sgx? Interesting. I assume that Apache doesn't find semget() syscall and falls back to some kind of shared files for synchronization. Shared files work on gramine-direct but do not work on gramine-sgx (for the obvious reason that Gramine-SGX doesn't allow untrusted sharing for security reasons).

I would suggest to read more about Apache internals and try to figure out if there's a way to disable Sys-V semaphores and/or shared files/shared memory.

[dimakuv]

We removed Apache from the examples, since it didn't work. Closing this issue.