Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opensc-notify: forks inside C_Initialize #1507

Closed
loskutov opened this issue Oct 13, 2018 · 16 comments · Fixed by #1621
Closed

opensc-notify: forks inside C_Initialize #1507

loskutov opened this issue Oct 13, 2018 · 16 comments · Fixed by #1621
Labels

Comments

@loskutov
Copy link

Problem Description

Here's a stack backtrace:

#0  0x00007ff6b32026a0 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffdc57bfd20, remaining=remaining@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007ff6b32357d4 in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:32
#2  0x00007ff6b36ff232 in  () at /usr/lib/x86_64-linux-gnu/libpkcs11-helper.so.1
#3  0x00007ff6b324d578 in __run_fork_handlers (who=who@entry=atfork_run_prepare) at register-atfork.c:122
#4  0x00007ff6b3202707 in __libc_fork () at ../sysdeps/nptl/fork.c:58
#5  0x00007ff6b1f0b712 in fork_exec_with_fds
    (intermediate_child=intermediate_child@entry=0, working_directory=working_directory@entry=0x0, argv=argv@entry=0x55be22fdf8e0, envp=envp@entry=0x0, close_descriptors=close_descriptors@entry=1, search_path=search_path@entry=1, search_path_from_envp=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, cloexec_pipes=0, child_setup=0x0, user_data=0x0, child_pid=0x7ffdc57c0008, child_close_fds=0x7ffdc57bff20, stdin_fd=-1, stdout_fd=5, stderr_fd=7, error=0x7ffdc57c01c8) at ../../../../glib/gspawn.c:1644
#6  0x00007ff6b1f0bfab in fork_exec_with_pipes
    (intermediate_child=intermediate_child@entry=0, working_directory=working_directory@entry=0x0, argv=0x55be22fdf8e0, envp=envp@entry=0x0, close_descriptors=close_descriptors@entry=1, search_path=search_path@entry=1, search_path_from_envp=0, stdout_to_null=0, stderr_to_null=0, child_inherits_stdin=0, file_and_argv_zero=0, cloexec_pipes=0, child_setup=0x0, user_data=0x0, child_pid=0x7ffdc57c0008, standard_input=0x0, standard_output=0x7ffdc57c0000, standard_error=0x7ffdc57c0004, error=0x7ffdc57c01c8) at ../../../../glib/gspawn.c:1955
#7  0x00007ff6b1f0c3c6 in g_spawn_sync
    (working_directory=working_directory@entry=0x0, argv=<optimized out>, envp=envp@entry=0x0, flags=flags@entry=G_SPAWN_SEARCH_PATH, child_setup=child_setup@entry=0x0, user_data=user_data@entry=0x0, standard_output=0x7ffdc57c0158, standard_error=0x7ffdc57c0160, exit_status=0x7ffdc57c0154, error=0x7ffdc57c01c8) at ../../../../glib/gspawn.c:391
#8  0x00007ff6b1f0cc07 in g_spawn_command_line_sync
    (command_line=command_line@entry=0x55be22fdf800 "dbus-launch --autolaunch=790ef33180164b2a91d23c8c584cb1c2 --binary-syntax --close-stderr", standard_output=standard_output@entry=0x7ffdc57c0158, standard_error=standard_error@entry=0x7ffdc57c0160, exit_status=exit_status@entry=0x7ffdc57c0154, error=error@entry=0x7ffdc57c01c8) at ../../../../glib/gspawn.c:932
#9  0x00007ff6b209ecd0 in get_session_address_dbus_launch (error=error@entry=0x7ffdc57c01c8) at ../../../../gio/gdbusaddress.c:1131
#10 0x00007ff6b20a080a in get_session_address_platform_specific (error=0x7ffdc57c01c8) at ../../../../gio/gdbusaddress.c:1559
#11 0x00007ff6b20a080a in g_dbus_address_get_for_bus_sync (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0)
    at ../../../../gio/gdbusaddress.c:1639
#12 0x00007ff6b20abd96 in get_uninitialized_connection (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0)
    at ../../../../gio/gdbusconnection.c:7187
#13 0x00007ff6b20b16ad in g_bus_get_sync (bus_type=bus_type@entry=G_BUS_TYPE_SESSION, cancellable=cancellable@entry=0x0, error=error@entry=0x0) at ../../../../gio/gdbusconnection.c:7282
#14 0x00007ff6b20925fa in g_application_impl_register
    (application=application@entry=0x55be22fd4890 [GApplication], appid=0x55be22fd47c0 "org.opensc.notify", flags=G_APPLICATION_NON_UNIQUE, exported_actions=0x55be22fcdcd0, remote_actions=remote_actions@entry=0x55be22fd4838, cancellable=cancellable@entry=0x0, error=0x0) at ../../../../gio/gapplicationimpl-dbus.c:545
#15 0x00007ff6b208f464 in g_application_register (application=0x55be22fd4890 [GApplication], cancellable=cancellable@entry=0x0, error=error@entry=0x0) at ../../../../gio/gapplication.c:2120
#16 0x00007ff6b24f1586 in sc_notify_init () at notify.c:396
#17 0x00007ff6b278b1e5 in C_Initialize (pInitArgs=0x0) at pkcs11-global.c:241
#18 0x00007ff6b37086e9 in pkcs11h_addProvider () at /usr/lib/x86_64-linux-gnu/libpkcs11-helper.so.1
#19 0x000055be2166c133 in pkcs11_addProvider
    (provider=0x55be22fb6008 "/usr/lib/x86_64-linux-gnu/opensc-pkcs11.so", protected_auth=<optimized out>, private_mode=0, cert_private=<optimized out>) at pkcs11.c:399
#20 0x000055be2164fd05 in context_init_1 (c=0x7ffdc57c0570) at init.c:632
#21 0x000055be2166d62c in openvpn_main (argc=2, argv=0x7ffdc57c1568) at openvpn.c:297
#22 0x00007ff6b314809b in __libc_start_main (main=
    0x55be216398c0 <main>, argc=2, argv=0x7ffdc57c1568, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdc57c1558) at ../csu/libc-start.c:308
#23 0x000055be216398fa in _start () at openvpn.c:388

I'm trying to use OpenVPN with PKCS#11 authentication and OpenSC as the provider. On the frame #16 sc_notify_init calls some glib functions. As glib is unaware of the limitations that PKCS#11 spec puts on the code that's executed after C_Initialize is called, it calls fork internally. In this particular case, it results in pkcs11-helper hang (see also OpenSC/pkcs11-helper#16).

Proposed Resolution

Probably the notifications mechanism should be redesigned to avoid external library calls inside C_Initialize (no specific suggestions though).

@frankmorgner
Copy link
Member

You can compile OpenSC with --disable-notify, which avoids the problem. Currently we don't have a runtime configuration for this. Given the problem we had in the past (#1346), it would be advisable to put notifications into a seperate program as done on Windows.

@frankmorgner
Copy link
Member

@dwmw2 has already argued in OpenSC/pkcs11-helper#5 that fork()ing isn't forbidden per se. However, I cannot immediately see where the deadlock happens. OpenSC's code only locks a mutex in C_Finalize(), but not in C_Initialize()... Is the deadlock really happening inside OpenSC?

@loskutov
Copy link
Author

You mean, who locks the mutex in the first place?
Everything happens inside the pkcs11h_addProvider function of pkcs11-helper. First, it locks the mutex (pkcs11h-core.c:702), and then it calls the provider's C_Initialize (pkcs11h-core.c:778), which in our case forks and runs the atfork handler from the issue you referenced which tries to lock the mutex again (pkcs11h-core.c:1262).

However, I don't have a good understanding of the PKCS#11 spec and whether the Interface Usage Guide should be considered a part of it, but pkcs11-helper is obviously designed having this requirement in mind. Given it's a part of the OpenSC project, I suppose there should be some consistent understanding of this issue.

@frankmorgner
Copy link
Member

I didn't realize that it is already the initialization that fails. I've moved that part to the load time of the library (even before C_Initialize). If that doesn't work, I think there's no other possibility than to move it to C_GetSlotList...

frankmorgner added a commit to frankmorgner/OpenSC that referenced this issue Mar 7, 2019
@loskutov
Copy link
Author

loskutov commented Mar 7, 2019

Even before C_Initialize, the fork seems to be still intercepted by the handler: https://paste.ubuntu.com/p/g3hQHRtxqw/

@frankmorgner
Copy link
Member

frankmorgner commented Mar 7, 2019

  1. In pkcs11h_addProvider() _g_pkcs11h_data->mutexes.global is locked.
  2. When OpenSC forks at dlopen(), __pkcs11h_threading_atfork_prepare() calls _pkcs1h_threading_mutexLockAll(), which then also tries to lock _g_pkcs11h_data->mutexes.global, which deadlocks

I just realize that pkcs11-helper disallows forking in any PKCS#11 provider this way, because every PKCS#11 library call is protected with a mutex. This is very restrictive.

For tasting this forbidden fruit, we could add a daemon mode to opensc-notify outside of the PKCS#11 module (which we already have for Windows), but unfortunately I don't currently have time for that.

I think initializing OpenSC at dlopen() with a fork (outside of a PKCS#11 call) is the best I can do at the moment. pkcs11-helper should create a seperate mutexe for creation of a PKCS#11 module. __pkcs11h_threading_atfork_prepare() can then prevent usage of initialized modules, while pkcs11h_addProvider() can complete independently.

@alonbl, what do you think?

@alonbl
Copy link
Member

alonbl commented Mar 8, 2019

@frankmorgner: you present this as pkcs11-helper issue, while PKCS#11 is a standard... the problem of opensc forks within the provider is a generic common violation an unexpected behavior of PKCS#11 provider or any standard library.

I am truly sorry, but having a code specific for a provider implementation only because there is no time to solve it properly is not a good reason, the fact that OpenSC causes leak of handles for any application into subprocess or if daemon runs as root then user interface programs runs as root, is not something that people should accept, regardless if this works or not with pkcs11-helper.

I guess that if you already have this for windows, it should be possible to have this for all platforms without a complete write. It should be as simple as opening a unix domain socket and if success send notifications into it, the implementation of the peer can be done at later time.

@dwmw2
Copy link

dwmw2 commented Mar 8, 2019

This OpenSC bug looks invalid to me. You're allowed to fork() when you like, otherwise glib wouldn't be permitted to do it implicitly on a library call as it does.

The POSIX atfork documentation clearly forbids calling any but a limited set of safe system calls from the handler. Yet nothing in PKCS#11 says that a C_Initialize() method is limited to those calls — and it would be far too limited if it did so. You may not call C_Initialize() from an atfork handler. It's as simple as that. Just set a flag and then re-initialise the module the next time it's called.

The bug here is with pkcs11-helper violating the POSIX specification and common sense, which has been discussed at length. But doesn't it already have an option to turn that off?

@frankmorgner
Copy link
Member

This change to pkcs11-tool together with #1621 should at least make it work (without leaking ressources):

diff --git a/lib/pkcs11h-core.c b/lib/pkcs11h-core.c
index dd3133e..737dfe0 100644
--- a/lib/pkcs11h-core.c
+++ b/lib/pkcs11h-core.c
@@ -698,13 +698,6 @@ pkcs11h_addProvider (
 		provider_location
 	);
 
-#if defined(ENABLE_PKCS11H_THREADING)
-	if ((rv = _pkcs11h_threading_mutexLock (&_g_pkcs11h_data->mutexes.global)) != CKR_OK) {
-		goto cleanup;
-	}
-	mutex_locked = TRUE;
-#endif
-
 	if ((rv = _pkcs11h_mem_malloc ((void *)&provider, sizeof (struct _pkcs11h_provider_s))) != CKR_OK) {
 		goto cleanup;
 	}
@@ -742,6 +735,13 @@ pkcs11h_addProvider (
 		goto cleanup;
 	}
 
+#if defined(ENABLE_PKCS11H_THREADING)
+	if ((rv = _pkcs11h_threading_mutexLock (&_g_pkcs11h_data->mutexes.global)) != CKR_OK) {
+		goto cleanup;
+	}
+	mutex_locked = TRUE;
+#endif
+
 #if defined(_WIN32)
 	gfl = (CK_C_GetFunctionList)GetProcAddress (
 		provider->handle,
@@ -825,6 +825,13 @@ pkcs11h_addProvider (
 
 cleanup:
 
+#if defined(ENABLE_PKCS11H_THREADING)
+	if (mutex_locked) {
+		_pkcs11h_threading_mutexRelease (&_g_pkcs11h_data->mutexes.global);
+		mutex_locked = FALSE;
+	}
+#endif
+
 	if (provider != NULL) {
 		if (provider->handle != NULL) {
 #if defined(_WIN32)
@@ -839,13 +846,6 @@ cleanup:
 		provider = NULL;
 	}
 
-#if defined(ENABLE_PKCS11H_THREADING)
-	if (mutex_locked) {
-		_pkcs11h_threading_mutexRelease (&_g_pkcs11h_data->mutexes.global);
-		mutex_locked = FALSE;
-	}
-#endif
-
 #if defined(ENABLE_PKCS11H_SLOTEVENT)
 	_pkcs11h_slotevent_notify ();
 #endif

@frankmorgner
Copy link
Member

@alonbl what do you think about the two comments above? Within the OpenSC Project, we should at least find a compromise that doesn't put the work on the user...

@alonbl
Copy link
Member

alonbl commented Mar 25, 2019 via email

@frankmorgner
Copy link
Member

Hmm, it's a pity that you simply ignore other objections as invalid and classify any attempt to gracefully resolve this as forcing.

Anyway, reading the source code it seems like glib tries to forks to run dbus-launch, because it doesn't detect to be running in a valid dbus session (specifically, it looks for DBUS_SESSION_BUS_ADDRESS via getenv()).

@loskutov, before starting OpenVPN, please try running

eval `dbus-launch --sh-syntax`

Afterwards, glib should use it's standard IPC (which should not involve forking).

By the way, isn't openvpn running as root with setuid? If so, glib should normally not launch dbus (checked with g_check_setuid())...

@frankmorgner frankmorgner added invalid and removed bug labels Mar 26, 2019
@alonbl
Copy link
Member

alonbl commented Mar 26, 2019 via email

@frankmorgner
Copy link
Member

glib should detect both, running as root and running with sudo. In both cases, glib will not fork.

@alonbl
Copy link
Member

alonbl commented Mar 26, 2019 via email

@frankmorgner
Copy link
Member

Sure, but for some reason it's obviously not the case in @loskutov's setup.

frankmorgner added a commit to OpenSC/pkcs11-helper that referenced this issue Apr 23, 2019
alonbl added a commit to alonbl/pkcs11-helper that referenced this issue Apr 24, 2019
resolve a deadlock caused when provider fork in C_Initialize/C_Finalize.

Thanks: Frank Morgner <frankmorgner@gmail.com>
Bug: OpenSC/OpenSC#1507
Bug: OpenSC#16
frankmorgner added a commit to frankmorgner/OpenSC that referenced this issue May 22, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants