Windows environment variable limit reached when installing packages

[NFSpeedy]

Issue description

When you have an environment where you install packages from an in-house source with a hash for authentication (and the hash is long), you are risking hitting the environment variable limit of Windows. This issue prevents the use of pipenv on Windows machines.

The Windows limit is 32767 characters, but with in-house projects with many dependencies and sub dependencies, the chance of hitting the limit is absolute, especially when you are using JFrog's Artifactory.

Expected result

Installed packages.

Actual result

We are exceeding the character limit for Windows environment variables.

  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\<user_account>\AppData\Local\Programs\Python\Python37\Scripts\pipenv.exe\__main__.py", line 7, in <module>
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\vendor\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\cli\command.py", line 338, in lock
    write=not state.quiet,
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1059, in do_lock
    keep_outdated=keep_outdated
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\site-packages\pipenv\utils.py", line 1312, in venv_resolve_deps
    os.environ["PIPENV_PACKAGES"] = str("\n".join(constraints))
  File "c:\users\<user_account>\appdata\local\programs\python\python37\lib\os.py", line 687, in __setitem__
    self.putenv(key, value)
ValueError: the environment variable is longer than 32767 characters

Steps to replicate

1. Use a secured source such as JFrog Artifactory
2. Be given a huge key for authentication with the explanation that it is not possible to be shortened.
3. Define a list of packages to be installed by pipenv (10-20 will be enough)
4. Try to run pipenv install, pipenv install --dev, pipenv sync or pipenv sync --dev
5. Fail
   Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

I am not allowed to share the contents of this command — company policy.

[NFSpeedy]

The code in the fork is verified and works with passing tests.
I am new to contributing to open-source. If I need to do something more, please notify me. The code can be viewed in the commits above.

[frostming]

I would prefer a temporary file as input.

[NFSpeedy]

Sorry for the delay.

Here is an example pipfile. After creating the file, you have to run pipenv install and you will immediately get ValueError: the environment variable is longer than 32767 characters.
The user and key are the same lengths that we have been provided by the JFrog admins.

# The user and key are the same lengths used in JFrog

[[source]]
url = "https://user_exampl:C7NV0UmS3tdeDONM2CiXpw-FJIfzq2ME2eiuCL6KnP1StbmRcZR8qYFvo2Y84KJdkVFLVA4cCNRHIBiV5NpxLJiISi27ykJQzi5jGJj2gU4TlzDK1qi3ymIpBhxCV-2SmdwAoeb0lp31aCEoGhfXfK3sLlrPdOnSuajyctIwWYt2MMai5vgRQlN2Et6_JtGMA3v2U8SQXd4VZFbuGVhPTIm8Sb9KD91hOkVDWI563pvwFfULItTz76m_JuS3tL_55Fpkpzf26AJfcI0jPz_FTMnd9xWRDMfCfTy067c4YdfIJHPoBlg2n8_1uyVXqeYD2m-QM-wmCmjbXjmqcbbvKtMFfSx-pQZScAgX1bSQ-fUP25Qdb9Q3r6zMAS478b2qw3dkS8R-d6RKm-tA20KGg2h_xcExnym14fXtaIEQjC4w_0S35f5XhhTAnNIelxfQv2b_A8O6hiNsGIiQynuIscvczqEc4wafIJUZ2TFCXn1411WCKGKlsZe4IcCpqQlbDEDsczsDPUI2Ve4XhA4S0nb0wjNnpDurRJlzqWiuqoj6Xqw3_0FaRCAlyWn8SAtwZ-DkGF5JwKCfppiAPEXX9g-H1Hb_kL744e49bL_hIvmzx7AXeJtk0d9viGaK_22OpBv3KnK8F0qyY9Jxz7i0hHblls7htfBn7QpV5uW4_aXtly2nT9lc-1jJ9lUk7OyjQojLSnte8_lZQLHubBT-E5kyKJQRz69onQiQK-AL-KBF5oA1LFZDjZTPqgrnVsUXm48hH4dFXKt786-AUqNXWNdCMsFCzbMAkoGZ5JBoXPsIRRvkMIazNraRsDiNbWWhTQTyXv31R267RMKfrELrvYQq9bG_aOkNYc_DahZVE_kBU7eQ4xBZINZY7URads8h-_ygU1vcN4L2j1m9JuEpb73tYEc-TLJK-Ar0TpJhZiUlda8GsSnp8Aqcg56eyTVGDd0B6KZmmyMXGv2hrbqL7pquslYNRFJp9GAbCqrS7TTX2-19NeI9c2uVgNKiM2J_N6q9CZ5fCXYx4exWSKXRXaeMWWChHGZWgmhlW0EqxtAmicsSQiLLd1ZglyFRGlfEMjZqs-L6uxw2HM9QAM93VVnpb6jbyoj4uF4LGuje8NBDNHHu-ub44Wc8iB8Z7y8LoYC2R7_CibDUCY6XbwFnNX1q4Nwde2yFERh6F4naBIAoxmmuSFeTOT8@pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "*"
django-503 = "*"
django-pg-zero-downtime-migrations = "*"
django-redis = "*"
wagtail-headless-preview = "*"
django-scribbler = "*"
django-countries-plus = "*"
django-s3-sqlite = "*"
django-developmentemaildashboard = "*"
django-recaptcha429 = "*"
django-convenient-formsets = "*"
django-rql = "*"
django-jinja = "*"
django-tagulous = "*"
xss-utils = "*"
fluentcms-cookielaw = "*"
django-reactify = "*"
edx-rest-api-client = "*"
paper-rq = "*"
django-db-logging = "*"
easy-thumbnails = "*"
django-bleach = "*"
django-rdtwt = "*"
django-awesomplete = "*"
django-dirtyfields = "*"
django-errors = "*"
django-guest-user = "*"
django-treenode = "*"
django-browser-reload = "*"
django-testing-utils = "*"
django-dkim = "*"
django-cors-headers = "*"
django-apiblueprint-view = "*"
django-rest-friendship = "*"

[dev-packages]

[requires]
python_full_version = "3.7.9"

[NFSpeedy]

@frostming is this what you requested?

[frostming]

Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.

[NFSpeedy]

You might have permission issues in some environments because of the security limitations of the technical user/bot.
I am unsure that it will be a good solution.

[frostming]

You might have permission issues in some environments because of the security limitations of the technical user/bot. I am unsure that it will be a good solution.

Or feed the packages via stdin/command line arguments? Nothing is more ugly than multiple env vars IMHO.

[NFSpeedy]

32000+ cmd argument? Well... that's a new level of Python craziness. :D
IMHO it would be better to generate the links when you require them. organize the function that generates the links as a generator, then you pass it to the loop that installs the packages.
This way, you would not need any vars, env vars, args, temps etc.
What do you think?

[Hristiyan-Andreev]

Instead of passing packages via multiple env vars, I would prefer to write them into a temporary file and make the resolver read from it.

Hello, I've finally found that there is an issue with this problem. We experience the same issue with artifactory, since Jfrog artifactory keys are quite big and they overflow the ENV.

When a fix will be applied? Otherwise, we will start using our own fork of Pipenv, I guess.

BTW, do you actually require to use persistent storage for passing the package links? Do the generation and resolution happen in the same python process? If they do, can't you just use a regular Python variable?

[NFSpeedy]

@frostming ? Are we getting a resolution?