Global: supporting OPENPYPE_TMPDIR in staging dir maker

[jakubjezek001]

Brief description

Productions can use OPENPYPE_TMPDIR for staging temp publishing directory

Description

Studios were demanding to be able to configure their own shared storages as temporary staging directories.
Template formatting is also supported with optional keys formatting and following anatomy keys:
- root[work | ]
- project[name | code]

Documentation:

future documentataion url

Testing notes:

1. configure OPENPYPE_TMPDIR on a testing project with {root[work]}/{project[name]}/temp
2. publish anything from any host.
3. all should work as expected.
4. notice temp foder in root of your project will appear.

[ynbot]

Task linked: OP-3663 Hiero - export of large number of shots fill up disk

[BigRoy]

Nice feature! :)

Is it to be expected that there are no Studio Settings to set the temp dir and its up to the machine itself to "choose" the temp directory by setting a global OPENPYPE_TEMP_DIR variable from outside of OpenPype itself?

I like that a particular user/machine can set its own custom temporary directory, e.g. for if that machine has a strong local cache disk to use. But I felt that with the complex formatting that is possible using anatomy this feels more like a studio setting of sorts with a potential override in local settings?

Anyway, only other thing I'm worried about is that it breaks some logic I had set up for Substance Painter. In particular this line of code of using publish.get_instance_staging_dir(instance) from within a Collector because that is in a CollectorOrder BEFORE instance.context.data["anatomy"] already exists. Keeping that in mind this might be a change breaking other areas of code too.

EDIT:

Missing documentation!

Also, this feature must have a mention in the documentation too. So please make sure the Admin Docs contain an explicit mention about the feature and how it's intended to be used by a studio or artist.

[jakubjezek001]

Is it to be expected that there are no Studio Settings to set the temp dir and its up to the machine itself to "choose" the temp directory by setting a global OPENPYPE_TEMP_DIR variable from outside of OpenPype itself?

Yes that was the idea. Yes the documentation would be important for this.

[iLLiCiTiT]

Few comments:

• Current imlementation is using OPENPYPE_TMPDIR only at one place and there is no prepared function to get it from any other place. The implementation expects that it is used only for instance staging dir but it should be possible to get it out of instance scope. So it is not usable for other places where new temp directory should be created without using instance staging dir.

• The value is targeted for publishing and nothing else in that case it should be maybe called OPENPYPE_PUBLISH_TMPDIR because I would expect more usages then just publishing.

• The possible keys is overkill which is limiting usage of the value. I would limit possible keys only to project name/code and roots. I can't imagine any usecase where other keys from anatomy data would make sense.

• I would expect a function to just create tempdir (e.g. in openpype/pipeline/tempdir.py -> just made up) which is then used in publish staging dir.

def _format_tempdir(temp_dir, project_name, anatomy):
    if "{" not in temp_dir:
        return temp_dir

    if anatomy is None:
        anatomy = Anatomy(project_name)

    data = {
        "root": anatomy.roots,
        "project": {
            "name": project_name,
            "code": anatomy.project_code
        }
    }
    return StringTemplate.format_template(temp_dir, data).normalized()


def create_tempdir(project_name, anatomy=None, prefix=None):
    if prefix is None:
        prefix = "op_tmp_"

    kwargs = {"prefix": prefix}
    openpype_temp_dir = os.getenv("OPENPYPE_TMPDIR")
    if openpype_temp_dir:
        openpype_temp_dir = _format_tempdir(
            openpype_temp_dir,
            project_name,
            anatomy
        )
        if not os.path.exists(openpype_temp_dir):
            os.makedirs(openpype_temp_dir)
        kwargs["dir"] = openpype_temp_dir

    return os.path.normpath(
        tempfile.mkdtemp(**kwargs)
    )

[antirotor]

Some small typos.

Question - shouldn't this be set per context? I mean - it would be useful to influence the staging dir per family (or asset). Considering cases like special share for transient renders or big caches...

[iLLiCiTiT]

Question - shouldn't this be set per context? I mean - it would be useful to influence the staging dir per family (or asset). Considering cases like special share for transient renders or big caches...

It is still creating subdirectories in the temp dir, and must create them because of filename clashes.

[antirotor]

This is unfortunately breaking Python 2 hosts.