jsii-pacmak very slow running python in fresh pipenv

[jsteinich]

🐛 Bug Report

Affected Languages

• TypeScript or Javascript
• Python
• Java
• .NET (C#, F#, ...)

General Information

• JSII Version: 1.7.0
• Platform: Linux docker-desktop 4.19.104-microsoft-standard

What is the problem?

jsii-pacmak is very slow when python is compiling 700+ files (hashicorp/terraform-cdk#276) in a fresh pipenv. On the order of 10 minutes.

I managed to track the main slowness of trying to integrate jsii-srcmak into cdktf to running python setup. I'm not certain how much of the time is sdist vs wheel though.
Subsequent runs in the same environment are much faster. Still slower than ideal, but #1854 and #1855 are more relevant in that case.

@skorfmann also observed "one python process takes up the majority of the time and consumes up to 2.8 GB of memory at 100% cpu" while building the cdk tf aws provider (https://github.com/skorfmann/cdktf-provider-aws)

Verbose Log

[RomainMuller]

A while ago, jsii-pacmak started running black on the generated Python to improve formatting (users had requested this). We are checking for it's presence in $PATH and installing it in a cache location ($HOME/.jsii-cache/...) if it's not there.

I wonder if the difference between your "cold" and "hot" environments is simply the pip install black that is happening there?

Second - could the large python process turn out to be black itself?

[jsteinich]

I was just able to confirm that it is indeed running black that is taking almost 10 minutes in my test.

Perhaps an option to disable it would work well. It doesn't seem like a necessary step for just running tests.

[RomainMuller]

@jsteinich that makes perfect sense! If you don't mind I'll convert this to a feature request (to allow opting out of black for python)... Which I'll get to once I'm done with my current problems 😅

[RomainMuller]

Also - we could run black with --fast (instead of the default --safe), so I'd stop checking for AST coherence, which apparently would make it significantly faster...

[eladb]

Is it possible to only run black if it is installed? I am sure installation is also a huge time waste