Using xdist with libfaketime

[seporaitis]

Hi,

We would like to use python-libfaketime (a wrapper around the excellent libfaketime). It works by re-executing the interpreter with LD_PRELOAD environment variable pointing to libfaketime.so file. This works in a single-threaded pytest run, but it fails with xdist - after some digging it seems that LD_PRELOAD is not passed along to the child process. Using --dist=load --tx 2*popen//env:LD_PRELOAD=/path/to/libfaketime.so.1 does not seem to help.

We've tried to run libfaketime.reexec_if_needed in pytest_configure and pytest_configure_node, but they are both executed in the parent process - thus useless. Forcing libfaketime.reexec_if_needed in the child produces a loop, since it re-execs the parent command: py.test - collect tests - initialize nodes - reexec py.test - .... As well as some other attempts to make LD_PRELOAD reach child process - all without success.

To reproduce:

---

files:

# assuming some sort of venv
$ pip install pytest pytest-xdist libfaketime
$ mkdir libfaketime_xdist
$ cd libfaketime_xdist
$ touch conftest.py
$ touch test_issue.py

conftest.py:

from libfaketime import reexec_if_needed

def pytest_configure():
    reexec_if_needed()

test_issue.py:

import datetime
from libfaketime import fake_time

def test_faketime():
    with fake_time("2016-05-05 00:00:00"):
        assert datetime.datetime.now() == datetime.datetime(2016, 5, 5, 0, 0, 0)

execute:

$ py.test  # succeeds
$ py.test -n 2  # fails

---

Any tips on how to approach this would be very welcome.

[RonnyPfannschmidt]

this one needs extra support, atm execnet does set env vars after fork (ssh vs non-ssh)

its not clear how to handle interpreter restart correctly off-hand, please try to get a hold of me in #pylib on freenode tommorow

[seporaitis]

So we had a brief chat and this is difficult to tackle, especially for ssh gateways. A crude plan now is to add an additional gateway to execnet that sets env variables before spawning the remote python process. This would enable the libfaketime use-case for us. In the end I imagine it would work something like this in pytest.ini:

addopts = --tx '2*pre-popen//env:LD_PRELOAD=/path/to/libfaketime.so.1'

The gateway name - to be decided. The timeframe - I'll sit on it around mid-week next week.

Meanwhile, with some plain hacking around I verified that this plan should work - this patch and py.test --dist=load --tx '2*pre-popen//env:LD_PRELOAD=/path/to/libfaketime.so.1' above described failure case passed.

[adamchainz]

Hi, I'm @seporaitis' colleague.

This works:

libfaketime.reexec_if_needed()
os.environ['LD_PRELOAD'] = libfaketime.get_reload_information()[1]['LD_PRELOAD']  # for pytest-xdist

libfaketime by default doesn't re-add the environment var into the process after the preloader has removed it, but doing it manually causes it to be inherited by the subprocess interpreters. Magic!

[seporaitis]

I was about to write the same :-)

@adamchainz solution works on Linux. Updating conftest.py as per below should make it work more robustly across linux and osx:

import os
import libfaketime

def pytest_configure():
    libfaketime.reexec_if_needed()
    _, env_additions = libfaketime.get_reload_information()
    os.environ.update(env_additions)

Verified that it works and I'll close the issue.

[RonnyPfannschmidt]

good find, this is a entirely different kind of solution - can i rope you into providing a more complete example for xdist or pytest documentation? or perhaps even creating a pytest plugin for this