Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cli Docs?] Conflicting Info in regards to pipenv lock --dev --requirements #3316

Closed
gtalarico opened this issue Nov 28, 2018 · 16 comments · Fixed by #3878 or #4183
Closed

[Cli Docs?] Conflicting Info in regards to pipenv lock --dev --requirements #3316

gtalarico opened this issue Nov 28, 2018 · 16 comments · Fixed by #3878 or #4183
Labels
good first issue help wanted Type: Documentation 📖

Comments

@gtalarico
Copy link
Contributor

@gtalarico gtalarico commented Nov 28, 2018

Overview

Docs, Actual Results, and CLI Help are in conflict:
Should --dev include only dev dependencies? Or both dev + default dependencies?

Actual Results

pipenv lock --dev --requiremnts

Outputs only dev dependencies

Docs

States that --dev should output only dev dependencies

If you wish to generate a requirements.txt with only the development requirements you can do that too! ... pipenv lock -r --dev

https://pipenv.readthedocs.io/en/latest/advanced/#generating-a-requirements-txt

CLI usage

States --dev should includes both dev + default dependencies:

  -d, --dev           Install *both* develop and default packages.  [env var: PIPENV_DEV]

pipenv, version 2018.11.26

@frostming frostming added help wanted Type: Documentation 📖 good first issue labels Nov 28, 2018
@gtalarico
Copy link
Contributor Author

@gtalarico gtalarico commented Nov 28, 2018

@frostming please confirm: is CLI docs incorrect and expected behaviour is dev deps only

@frostming
Copy link
Contributor

@frostming frostming commented Nov 28, 2018

Let me try to clarify it. We re-organized the commands and options to reduce repetitions. And the --dev option is taken from a common option, which is documented as "install both develop and default packages", whereas, as you found, behaves differently between commands.

In the context of lock -r, it is likely that people use it to generate the old-fashioned requirements.txt and dev-requirements.txt, so it is more expected the two sections are generated separately. Now what we should do is to replace the common option to a specific one with the correct document string to reflect what it does. Contributions are welcomed.

It is just my opinion and it might be wrong. Kenneth should have some words for it.

@frostming
Copy link
Contributor

@frostming frostming commented Nov 28, 2018

@gtalarico I just changed my mind and modified the previous comment a lot, sorry for the trouble.

@gtalarico
Copy link
Contributor Author

@gtalarico gtalarico commented Nov 28, 2018

I understand the logic, but using the same exact --dev flag for opposite behaviors smells funny to me
🙊

Edit

Did i get this right?

Command Flags Default Deps Dev Deps
pipenv install none
pipenv install --dev
pipenv lock none
pipenv lock --requirements
pipenv lock --requirements --dev
  • installing only dev dependencies is possible using (ok, edge case use)

pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt

@jxltom
Copy link
Contributor

@jxltom jxltom commented Nov 28, 2018

@gtalarico You are right as shown in the table. Currently --dev has different logic for install and lock -r

Install only dev packages is not a typical usage but you can do that by pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt.

By default, pipenv lock is locking both default and dev. I think --dev is not working for pipenv lock --dev (correct me if i'm wrong) which needs enhancement.

@gtalarico
Copy link
Contributor Author

@gtalarico gtalarico commented Nov 28, 2018

install only dev packages is not a typical usage but you can do that by pipenv lock -r -d > requirements-dev.txt && pip install requirements-dev.txt.

I think that's good enough given how unusual of a use case that is.

By default, pipenv lock is locking both default and dev

Updated the table above to reflect this

@techalchemy
Copy link
Member

@techalchemy techalchemy commented Nov 29, 2018

@gtalarico the api is backwards when you dump requirements, I didn't like that when it was implemented but it's probably too late to undo it

@jxltom it's technically impossible to lock dev packages on their own in any meaningful way with the current implementation of locking. Even if we keep the dev dependencies separate, they can't ever be allowed to conflict with the production dependencies, which means we currently just overwrite dev dependencies as they are generated if they conflict.

It's not possible to generate them independently because by nature the dev dependencies require the default dependencies to exist and be up to date first, otherwise there is no way to be sure they are not in conflict.

@jxltom
Copy link
Contributor

@jxltom jxltom commented Nov 29, 2018

@techalchemy Thanks for explanation. I feels same for that locking default and dev deps independently is not possible and also it is not a typical use. Maybe we could consider removing --dev options for pipenv lock otherwise it will confuse users.

@frostming
Copy link
Contributor

@frostming frostming commented Nov 29, 2018

@jxltom I doubt if there is a way to do this since --dev is needed by lock -r.

@jxltom
Copy link
Contributor

@jxltom jxltom commented Nov 29, 2018

@frostming Oh, yes, I missed that... Thanks for the kind reminder. I don't know whether click has this kind of support which accepts only lock or lock -r -d but not lock -d. The other way is to raise a warning manually for user when pipenv lock --dev is used.

@gtalarico
Copy link
Contributor Author

@gtalarico gtalarico commented Nov 29, 2018

Maybe we could propose a change... ?
Remove --dev flag due to its amiguitiy and opposite behavior of install dev flag.

Use lock --requirements and lock --requirements-dev to generate requirements txt and requirements dev txt

@layoaster
Copy link

@layoaster layoaster commented Mar 1, 2019

In my case, I'm using the old-fashioned requirements.txt to build Docker images.

Therefore, it's extremely needful to have pipenv lock -r generating only default deps and pipenv lock -r -d generating both (default & development) dependencies together so I can build separate images for development and production.

As workaround, I have to manually add -r ./requirements.txt to my requirements_dev.txt which minimizes extra work but still far from optimal.

Honestly, I don't know if there is a better way to work with Docker images while using/adopting pipenv since this is the first time I use pipenv. if you know a better way, please point me to the right direction.

adborden added a commit to GSA/data.gov that referenced this issue Apr 4, 2019
Use `pipenv run pip freeze > requirements.txt` to generate the requirements.txt

pypa/pipenv#3316
adborden added a commit to GSA/data.gov that referenced this issue Apr 8, 2019
Use `pipenv run pip freeze > requirements.txt` to generate the requirements.txt

pypa/pipenv#3316
@philtonite
Copy link

@philtonite philtonite commented May 22, 2019

@layoaster I struggled with this while trying to build docker images too - also the first time I'm using pipenv.

I found a way to use the Pipfile.lock and this command: pipenv install --ignore-pipfile --system --deploy --dev when building the image.

Currently I'm installing all the packages in the image, as I use the same image to test and run in production.

@nickwilliams-eventbrite
Copy link

@nickwilliams-eventbrite nickwilliams-eventbrite commented Aug 1, 2019

Yet another very frustrating roadblock for us. We're trying to work around #1356 and #3586, which are basically killing us. We decided to just convert the Pipfile.lock to a requirements.txt because we know we can rely on pip install to actually fail when it fails. But pipenv lock -r --dev was not working as expected/documented. So then I came across this issue.

We completely disagree with the assertion that the documentation is wrong. It makes no sense for --dev to work differently here than it does in install, et. al. If you see a use case for dev-only requirements files, then add --dev-only, or, at the very least, PLEASE give us a --default-and-dev option so we can generate one file with everything. What should be a simple pipenv install --dev --deploy for us has now become the following, and it's horribly frustrating:

    pipenv lock --requirements > .tmp_requirements.txt
    pipenv lock --requirements --dev > .tmp_requirements_dev.txt
    pip install -r .tmp_requirements.txt
    pip install -r .tmp_requirements_dev.txt
    rm .tmp_requirements*.txt

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented Mar 23, 2020

+1 from me for @frostming's proposal (cc @techalchemy).

We just ran into this in a docker build after switching to pip-sync to work around #3057 (since pip-sync removes non-matching packages, unlike pipenv sync, it's more sensitive to transitive dependencies being missing from the locked output).

In our case, we can work around it by removing the production/dev distinction (as our production deployments don't go through pipenv, they use a separate pip-compile invocation), but the general case needs --dev to work the same way it does in other commands.

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented Apr 10, 2020

(Sorry about that - this isn't closed yet, there was just a magic comment in the PEEP 006 PR merge)

techalchemy pushed a commit that referenced this issue May 20, 2020
* Implements PEEP-006
* `pipenv lock -r --dev` is now consistent with other commands
  and the CLI help output, and includes both default and dev
  dependencies in the result
* New `--dev-only` option allows requesting the previous behaviour
  (which was specifically designed to support the traditional
  `requirements.txt`/`dev-requirements.txt` split)
fwojciak pushed a commit to fwojciak/pipenv that referenced this issue May 29, 2020
2020.5.28 (2020-05-28)
======================

Features & Improvements
-----------------------

-   `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506
-   Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc.
    -   Improved error handling and formatting.
    -   Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298
-   For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316
-   Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328
-   Added support for `--verbose` output to `pipenv run`. pypa#3348
-   Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368
-   Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577
-   Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595
-   Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652
-   Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738
-   Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763
-   Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810
-   Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018
-   Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245
-   Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257

Behavior Changes
----------------

-   Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553
-   Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386
-   Re-enable `--help` option for `pipenv run` command. pypa#3844
-   Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199

Bug Fixes
---------

-   Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373
-   Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722
-   Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some
    systems.
-   Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783
-   Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053
-   Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071
-   Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances.
-   Fixed a bug with package discovery when running `pipenv clean`. pypa#3298
-   Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307
-   Handle alternate names for UTF-8 encoding. pypa#3313
-   Abort pipenv before adding the non-exist package to Pipfile. pypa#3318
-   Don\'t normalize the package name user passes in. pypa#3324
-   Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339
-   Fix a bug that `--site-packages` flag is not recognized. pypa#3351
-   Fix a bug where `pipenv --clear` is not working pypa#3353
-   Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384
-   Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396
-   Fixed a keyerror which could occur when locking VCS dependencies in
    some cases. pypa#3404
-   Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427
-   Updated the index names in lock file when source name in Pipfile is changed. pypa#3449
-   Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479
-   Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502
-   Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537
-   Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584
-   Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647
-   The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies.
    -   In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656
-   Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718
-   Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745
-   `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766
-   Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768
-   Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786
-   Fix a bug that installation errors are displayed as a list. pypa#3794
-   Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807
-   Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting
    successful installation. pypa#3809
-   `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819
-   Resolve the symlinks when the path is absolute. pypa#3842
-   Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879
-   Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with
    subdirectories. pypa#3976
-   Honor `PIPENV_SPINNER` environment variable pypa#4045
-   Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188
-   Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217
-   Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220
-   `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226
-   Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229
-   Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231
-   Don\'t fallback to system Python when no matching Python version is found. pypa#4232

Vendored Libraries
------------------

- Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766
- Update vendored dependencies and invocations
  - Update vendored and patched dependencies
  - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit`
  - Fix invocations of dependencies
  - Fix custom `InstallCommand` instantiation
  - Update `PackageFinder` usage
  - Fix `Bool` stringify attempts from `tomlkit`
  - Updated vendored dependencies:
    -   **attrs**: `18.2.0 => `19.1.0`
    -   **certifi**: `2018.10.15 => `2019.3.9`
    -   **cached\_property**: `1.4.3 => `1.5.1`
    -   **cerberus**: `1.2.0 => `1.3.1`
    -   **click**: `7.0.0 => `7.1.1`
    -   **click-completion**: `0.5.0 => `0.5.1`
    -   **colorama**: `0.3.9 => `0.4.3`
    -   **contextlib2**: `(new) => `0.6.0.post1`
    -   **distlib**: `0.2.8 => `0.2.9`
    -   **funcsigs**: `(new) => `1.0.2`
    -   **importlib\_metadata** `1.3.0 => `1.5.1`
    -   **importlib-resources**: `(new) => `1.4.0`
    -   **idna**: `2.7 => `2.9`
    -   **jinja2**: `2.10.0 => `2.11.1`
    -   **markupsafe**: `1.0 => `1.1.1`
    -   **more-itertools**: `(new) => `5.0.0`
    -   **orderedmultidict**: `(new) => `1.0`
    -   **packaging**: `18.0 => `19.0`
    -   **parse**: `1.9.0 => `1.15.0`
    -   **pathlib2**: `2.3.2 => `2.3.3`
    -   **pep517**: `(new) => `0.5.0`
    -   **pexpect**: `4.6.0 => `4.8.0`
    -   **pip-shims**: `0.2.0 => `0.5.1`
    -   **pipdeptree**: `0.13.0 => `0.13.2`
    -   **pyparsing**: `2.2.2 => `2.4.6`
    -   **python-dotenv**: `0.9.1 => `0.10.2`
    -   **pythonfinder**: `1.1.10 => `1.2.2`
    -   **pytoml**: `(new) => `0.1.20`
    -   **requests**: `2.20.1 => `2.23.0`
    -   **requirementslib**: `1.3.3 => `1.5.4`
    -   **scandir**: `1.9.0 => `1.10.0`
    -   **shellingham**: `1.2.7 => `1.3.2`
    -   **six**: `1.11.0 => `1.14.0`
    -   **tomlkit**: `0.5.2 => `0.5.11`
    -   **urllib3**: `1.24 => `1.25.8`
    -   **vistir**: `0.3.0 => `0.5.0`
    -   **yaspin**: `0.14.0 => `0.14.3`
    -   **zipp**: `0.6.0`
    - Removed vendored dependency **cursor**. pypa#4169

-   Add and update vendored dependencies to accommodate `safety` vendoring:
    -   **safety** `(none)` => `1.8.7`
    -   **dparse** `(none)` => `0.5.0`
    -   **pyyaml** `(none)` => `5.3.1`
    -   **urllib3** `1.25.8` => `1.25.9`
    -   **certifi** `2019.11.28` => `2020.4.5.1`
    -   **pyparsing** `2.4.6` => `2.4.7`
    -   **resolvelib** `0.2.2` => `0.3.0`
    -   **importlib-metadata** `1.5.1` => `1.6.0`
    -   **pip-shims** `0.5.1` => `0.5.2`
    -   **requirementslib** `1.5.5` => `1.5.6` pypa#4188

-   Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215
-   Updated vendored dependencies to latest versions for security and bug fixes:
    -   **requirementslib** `1.5.8` => `1.5.9`
    -   **vistir** `0.5.0` => `0.5.1`
    -   **jinja2** `2.11.1` => `2.11.2`
    -   **click** `7.1.1` => `7.1.2`
    -   **dateutil** `(none)` => `2.8.1`
    -   **backports.functools\_lru\_cache** `1.5.0` => `1.6.1`
    -   **enum34** `1.1.6` => `1.1.10`
    -   **toml** `0.10.0` => `0.10.1`
    -   **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226
-   Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267

Improved Documentation
----------------------

-   Added documenation about variable expansion in `Pipfile` entries. pypa#2317
-   Consolidate all contributing docs in the rst file pypa#3120
-   Update the out-dated manual page. pypa#3246
-   Move CLI docs to its own page. pypa#3346
-   Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499
-   Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522
-   Ensure docs show navigation on small-screen devices pypa#3527
-   Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629
-   Updated the documentation with the new `pytest` entrypoint. pypa#3759
-   Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911
-   Added a line describing potential issues in fancy extension. pypa#3912
-   Documental description of how Pipfile works and association with Pipenv. pypa#3913
-   Clarify the proper value of `python_version` and `python_full_version`. pypa#3914
-   Write description for `--deploy` extension and few extensions differences. pypa#3915
-   More documentation for `.env` files pypa#4100
-   Updated documentation to point to working links. pypa#4137
-   Replace docs.pipenv.org with pipenv.pypa.io pypa#4167
-   Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue help wanted Type: Documentation 📖
Projects
None yet
8 participants