nested temporary directories for easystack items

[bedroge]

We use easystack files for our software stack, and I've run into weird issues several times where a build would fail due to what looked like an issue with the temporary directory. In all cases, the temporary directory looked like:
/tmp/eb_something/eb_something/eb_something/....etc...
and sometimes the error clearly stated that this path got too long.

It seems that each easystack item sets its own temporary directory as a subdirectory of the previous one, leading to this nested structure. While looking into it a bit more, I found that it's probably being caused by this call to set_up_configuration, which sets up the eb_tmpdir:
https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/main.py#L277
Adding a print there indeed shows the issue with a simple easystack:

$ cat test.yml 
easyconfigs:
  - GCCcore-11.2.0.eb
  - foss-2021b.eb
  - attr-2.5.1-GCCcore-11.2.0.eb
  - zlib-1.2.12.eb

$ $ eb -M --experimental --easystack ./test.yml 
== Temporary log file in case of crash /tmp/eb-5_bt16c0/easybuild-0qwq_ptk.log
eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

1 out of 61 required modules missing:

* pkgconf/1.8.0 (pkgconf-1.8.0.eb)

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow/eb-9m3qhlsm
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow/eb-9m3qhlsm/eb-u43uav4s
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

== Temporary log file(s) /tmp/eb-5_bt16c0/easybuild-0qwq_ptk.log* have been removed.
== Temporary directory /tmp/eb-5_bt16c0 has been removed.

[bedroge]

Every item in the easystack file calls set_up_configuration, which will set the tmpdir (via the option parser?), and I guess this nesting then happens on lines 1878-1882 of https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/options.py#L1878. I'm not entirely sure yet why it happens, as the environment variables should be reset for every easystack item. However, inserting an tempfile.tempdir = None in the easystack loop here seems to solve the issue, so it definitely seems related to tempfile remembering the old tmpdir.

[bedroge]

It only happens when --tmpdir is not set, by the way. But it does happen when $TMPDIR is defined.

[bedroge]

The nested directory gets created here: https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/options.py#L1860.
os.environ['TMPDIR'] is not defined here, but tempfile.tempdir is set and points to the temp dir that was created before, leading to the nested structure.

I suspect it has to do with setting tempfile.tempdir = None and setting TMPDIR to the new temp dir, wich then leads to tempfile.tempdir being set to the new one later on as well (when e.g. tempfile.mkdtemp() is called). For any next easystack item, the os environment are restored/reset, but the tempfile.tempdir is not.

[bedroge]

Reproducer based on what more or less happens in the actual code:

import os
import tempfile

def set_tmpdir():
    tmpdir = tempfile.mkdtemp(prefix="eb-")
    os.environ['TMPDIR'] = tmpdir
    tempfile.tempdir = None
    print("tempfile.gettempdir(): %s" % tempfile.gettempdir())
    return tmpdir

for i in range(3):
    if 'TMPDIR' in os.environ:
        del(os.environ['TMPDIR'])
    tempfile.tempdir = None
    tmpdir = set_tmpdir()
    print("new tmpdir: %s" % tmpdir)
    print("tempfile.tempdir: %s" % tempfile.tempdir)

This results in:

tempfile.gettempdir() = /tmp/eb-e5ihmqst
tmpdir = /tmp/eb-e5ihmqst
tempfile.tempdir = /tmp/eb-e5ihmqst
tempfile.gettempdir() = /tmp/eb-e5ihmqst/eb-p1snbzpj
tmpdir = /tmp/eb-e5ihmqst/eb-p1snbzpj
tempfile.tempdir = /tmp/eb-e5ihmqst/eb-p1snbzpj
tempfile.gettempdir() = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh
tmpdir = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh
tempfile.tempdir = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh

Uncommenting the tempfile.tempdir = None in the loop solves the issue:

tempfile.gettempdir(): /tmp/eb-carcmvfb
new tmpdir: /tmp/eb-carcmvfb
tempfile.tempdir: /tmp/eb-carcmvfb
tempfile.gettempdir(): /tmp/eb-7udoyzcp
new tmpdir: /tmp/eb-7udoyzcp
tempfile.tempdir: /tmp/eb-7udoyzcp
tempfile.gettempdir(): /tmp/eb-81phvqcx
new tmpdir: /tmp/eb-81phvqcx
tempfile.tempdir: /tmp/eb-81phvqcx

Alternatively, the tempfile.tempdir = None could also be added to the top of the set_tmpdir function.