XrdCl::DefaultEnv interferes with fork()

[davidgcameron]

I am using the XrdCl libraries in a C++ framework which uses libc fork() heavily. When I create a DefaultEnv object, for example to set the log level, the static methods defined there somehow interfere with fork() and cause it to hang. A simple example program to recreate the problem:

> cat fork.cpp
#include <sys/wait.h>
#include <XrdCl/XrdClDefaultEnv.hh>
#include <XrdCl/XrdClLog.hh>

int main () {
  XrdCl::DefaultEnv env;
  XrdCl::Log *log = XrdCl::DefaultEnv::GetLog();
  log->SetLevel(XrdCl::Log::DumpMsg);

  int status;
  pid_t pid;

  pid = fork ();
  if (pid == 0) {
    // child
    sleep(1);
  } else if (pid < 0) {
    // The fork failed.  Report failure.
    status = -1;
  } else {
    // This is the parent process.  Wait for the child to complete.
    if (waitpid (pid, &status, 0) != pid)
      status = -1;
  }
  return status;
}

> g++ -o fork -I /usr/include/xrootd -l XrdCl -l pthread fork.cpp

> ./fork 
[2019-03-04 14:18:58.927546 +0100][Debug  ][Utility           ] In the prepare fork handler for process 1576837
[2019-03-04 14:18:58.927602 +0100][Debug  ][Utility           ] Running the prepare fork handler for process 1576837
[2019-03-04 14:18:58.927609 +0100][Debug  ][Utility           ] Locking File and FileSystem objects for process: 1576837
[2019-03-04 14:18:58.927615 +0100][Debug  ][Utility           ] In the prepare fork handler for process 1576837
[2019-03-04 14:18:58.927620 +0100][Debug  ][Utility           ] Running the prepare fork handler for process 1576837

The process hangs at this point. Removing the XrdCl part, i.e. commenting out the first 3 lines of main(), allows the process to complete normally.

Attaching to the hanging process, one can see that the XrdCl prepare handler is being used:

#0  0x00007ff6914be4ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ff6914b9dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007ff6914b9c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ff6917834b8 in XrdCl::ForkHandler::Prepare() () from /lib64/libXrdCl.so.2
#4  0x00007ff6917087f0 in prepare () from /lib64/libXrdCl.so.2
#5  0x00007ff690988eac in fork () from /lib64/libc.so.6
#6  0x0000000000400fb7 in main ()

Is there a way to avoid this problem?

[simonmichal]

Hi David,

Have you tried setting XRD_RUNFORKHANDLER=1 ?

Cheers,
Michal

[davidgcameron]

You mean as an environment variable? Or in the code?

export XRD_RUNFORKHANDLER=1 doesn't have any effect.

[simonmichal]

Hi David,

I beg your pardon, I haven't read full description of your problem, and thanks for the reproducer!

The problem is you create an instance of XrdCl::DefaultEnv, you mustn't be doing that (that's bad design on our side, the constructor should have been private). Instead please use the getter methods. For example, in order to get access to the logger do:

XrdCl::Log *log = XrdCl::DefaultEnv::GetLog();

or if you want access to the environment itself do:

XrdCl::Env *env = XrdCl::DefaultEnv::GetEnv();

Hope this solves your problem. Let me know if I can close this issue.

Cheers,
Michal

[davidgcameron]

Hi Michal,

Right, now that I look at the code again I see the DefaultEnv object is completely unecessary, since the next line is using the static GetLog() as you suggest. When I remove the call to the DefaultEnv constructor the problem goes away. Thanks for the fast answers! I think you can close the ticket.

Cheers,
David

[simonmichal]

Any time :-)

Cheers,
Michal