Package checks extremely slow with large build environments

[mdekstrand]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

When the build environment is very large, the post-packaging checks take an extremely long time. The culprit seems to be https://github.com/conda/conda-build/blob/main/conda_build/post.py#L988 (or somewhere in its call chain), where it it seems to wind up calling stat on every file in the build environment. This becomes a problem when building a Rust package, because the current Rust compiler toolchain installs documentation (35K files) and — after cargo build has completed — a copy of the cargo index (37K files). On Linux, this takes several minutes; on Windows, it takes hours. For an example of a public build that is affected, see conda-forge/deno-feedstock#79.

I determined this by two things:

1. running strace on the conda-build while it appeared stuck, saw many stat calls
2. ^C the stuck conda-build, found this call chain in the backtrace

I just started noticing this recently; I don't know if it is a change in conda-build or a change in how the rust compiler is packaged in conda-forge.

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

[travishathaway]

Hi @mdekstrand,

Thanks for reporting this. I've looked into this, and it does appear to be a performance regression from version 3.27.x to 3.28.x.

My hunch right now is that it is coming from this function:

• https://github.com/conda/conda-build/blob/main/conda_build/utils.py#L2225

Tagging @kenodegard as he works much more in the conda-build code base than I do.

[kenodegard]

Resolved in #5130