gh-139482: Add posix._clearenv() function

[vstinner]

• Issue: os.environ.clear() has a quadratic time complexity #139482

[vstinner]

Benchmark:

import os, time

VARIABLES = 10_000

env = os.environ
t1 = time.perf_counter()
for i in range(VARIABLES):
    env[f"k{i}"] = "1"
t2 = time.perf_counter()
print(f"create {VARIABLES:,} variables: {(t2-t1)*1e3:.1f} ms")

t1 = time.perf_counter()
env.clear()
t2 = time.perf_counter()
print(f"clear {VARIABLES:,} variables: {(t2-t1)*1e3:.1f} ms")

Without the change:

create 10,000 variables: 856.5 ms
clear 10,000 variables: 3276.3 ms

With the change (on Linux with posix.clearenv()):

create 10,000 variables: 856.5 ms
clear 10,000 variables: 1.1 ms

It's around 3,000x faster 😄

[picnixz]

Is it enabled by default on your machine or did you need to use some feature macros when compiling the project?

[picnixz]

I should note that I've seen many times the Windows failure today (the failure is in test_profiling).

[vstinner]

Is it enabled by default on your machine or did you need to use some feature macros when compiling the project?

I just modified the configure script to detect it, it's detected on Linux, and it just works. I didn't have to touch macros.

I should note that I've seen many times the Windows failure today (the failure is in test_profiling).

I created #139970 to track the test_profiling failure on Windows.

Do you mind documenting os.clearenv as well? or do you prefer not to document it? if you don't plan to, I would suggest having os._clearenv so that users don't use os.clearenv and end up with a desynchronized os.environ.

I prefer to not document it. I just rename it to posix._clearenv() to make it really private.

[vstinner]

I tested my PR on my FreeBSD VM. clearenv() function is detected by configure and Pythn builds successfully.

main branch:

create 10,000 variables: 1781.4 ms
clear 10,000 variables: 4726.3 ms

With posix._clearenv():

create 10,000 variables: 1402.2 ms
clear 10,000 variables: 23.5 ms

It's 201x faster.

[itamaro]

Shouldn't this also be inside ifdef guard?

[picnixz]

AC handles this by creating a no-op macro

[vstinner]

Yeah, AC is magic :-) It just works. Look at Modules/clinic/posixmodule.c.h:

#if defined(HAVE_CLEARENV)
...
#define OS__CLEARENV_METHODDEF    \
    {"_clearenv", (PyCFunction)os__clearenv, METH_NOARGS, os__clearenv__doc__}, 
... 
#endif /* defined(HAVE_CLEARENV) */ 

...

#ifndef OS__CLEARENV_METHODDEF
    #define OS__CLEARENV_METHODDEF
#endif /* !defined(OS__CLEARENV_METHODDEF) */