Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate requirements.txt from Pipfile.lock #3493

Closed
Zebradil opened this issue Jan 31, 2019 · 16 comments
Closed

Generate requirements.txt from Pipfile.lock #3493

Zebradil opened this issue Jan 31, 2019 · 16 comments

Comments

@Zebradil
Copy link

@Zebradil Zebradil commented Jan 31, 2019

How to generate requirements.txt file from existing Pipfile.lock without locking?
When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

There is a workaround for this:

$ pipenv sync
$ pipenv run pip freeze

In my particular situation I'm building docker image and using requirements.txt in Dockerfile. I'd like to avoid creating virtual environment on the host machine just to be able to create requirements.txt.

@frostming

This comment has been minimized.

Copy link
Collaborator

@frostming frostming commented Feb 1, 2019

Pipenv doesn't provide a way for this, you can seek for other pipfile utility libraries such as pipfile-requirements

@frostming frostming closed this Feb 1, 2019
@anuj9196

This comment has been minimized.

Copy link

@anuj9196 anuj9196 commented Jul 16, 2019

you can run

pipenv run pip freeze > requirements.txt
@Zebradil

This comment has been minimized.

Copy link
Author

@Zebradil Zebradil commented Jul 16, 2019

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

@frostming

This comment has been minimized.

Copy link
Collaborator

@frostming frostming commented Jul 16, 2019

LOL, I have already mentioned that library.

@Zebradil

This comment has been minimized.

Copy link
Author

@Zebradil Zebradil commented Jul 16, 2019

I, personally, dislike to have a dedicated library for that. Also, there is a higher chance that a team member already has jq or some other general purpose tool installed.

@anuj9196

This comment has been minimized.

Copy link

@anuj9196 anuj9196 commented Jul 16, 2019

you can even run

 pipenv lock --requirements > requirements.txt
@Zebradil

This comment has been minimized.

Copy link
Author

@Zebradil Zebradil commented Jul 16, 2019

It will not work as you expect, because, as I wrote:

When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

In other words, it performs an update, which potentially can destroy a distribution. Imagine, you generate requirements.txt to use it in Dockerfile to build a docker image. Locally your application works, but when you generate requirements.txt using pipenv lock, requirements might get updated to incompatible or just broken versions (hopefully, it's a rare case, though). And you will not know this before running the image. So, you'll need to test the app again after running pipenv lock.

If you don't want to use jq, then better use the approach I proposed in the first post with pipenv sync (which doesn't do update).

@ye

This comment has been minimized.

Copy link

@ye ye commented Nov 10, 2019

@Zebradil your jq oneliner approach is much simpler than @frostming 's own pipfile-requirements package (100+ lines python code) since I've already installed jq, no other dependencies required, which is great.

However, after a few git commits I noticed the difference between what pipenv lock --requirements outputs and what jq gleans through Pipfile.lock file and prints out:

  • jq output doesn't have -i https://pypi.org/simple as the very first line, as opposed to what pipenv lock --r always insert as the very first line.
  • jq output doesn't include the annotation for packages. For example: pipenv lock --r output has this lineappnope==0.1.0 ; sys_platform == 'darwin', but in jq output, it's appnope==0.1.0. Another example is pipenv lock -r generates pexpect==4.7.0 ; sys_platform != 'win32' whereas jq generates pexpect==4.7.0, not sure if this matters or not.
  • the requirements file package ordering is different between the two outputs. jq which presumably takes the package order in Pipfile.lock file, which always sorts by ascending order in alphabets and character length, for example, flask is in front of flask-sqlalchemy or any flask-XXXXX packages, whereas pipenv lock --r outputs flask behind flask-sqlalchemy, which is different from the order in the Pipfile.lock. This is a major annoyance because it doesn't generate a minimum git diff. I would consider this is a bug in pipenv.
@Zebradil

This comment has been minimized.

Copy link
Author

@Zebradil Zebradil commented Nov 12, 2019

Hi @ye, nice comparison of the methods. It might help people choosing proper solution for their particular situation and avoid caveats.

Yes, as you said, the proposed approach with jq has limited functionality. It is possible to extend it to add annotations and package index URL, but I have no need for this right now.

To avoid having differences in generated requirements.txt one should consider using the same approach every time. In a similar way, using different code formatting tools might lead to inconsistent results. So, I don't see a problem here.

@rcastill

This comment has been minimized.

Copy link

@rcastill rcastill commented Nov 26, 2019

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

Hi,

Thanks for your solution. I ran into the same problem but I also need the definition of sources created by pipenv lock -r, i.e.: -i, --extra-index-url. This is because I work with private sources.

@Zebradil I think you mentioned that.

So I created yet another minimal no-dependencies script in python that includes that functionality. It also expands env vars in case that you have your sources defined that way in your Pipfile.

If anybody wants to take a look I'll leave it here: https://gist.github.com/rcastill/dab85c234dd10fa7af56755116c75aee

@dms-cat

This comment has been minimized.

Copy link

@dms-cat dms-cat commented Dec 2, 2019

In case it helps anyone else, here's how to include the hashes in the results:

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock

This creates entries like

paramiko==2.6.0 --hash=sha256:99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf --hash=sha256:f4b2edfa0d226b70bd4ca31ea7e389325990283da23465d572ed1f70a7583041

Which makes pip enforce the hashes.

If you want to include only the requirements which were in the original requirements file (provided they were already locked to a specific version with ==):

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock | grep --file=<(grep --only-matching --perl-regexp '^.*(?===)' requirements.txt | tr '[:upper:]' '[:lower:]') > new.txt && mv new.txt requirements.txt

tr is necessary because requirements.txt files may contain mixed case package names but pipenv install -r requirements.txt lowercases them in Pipfile.

@BeyondEvil

This comment has been minimized.

Copy link

@BeyondEvil BeyondEvil commented Feb 3, 2020

Here's a small python script in case you want to turn the Pipfile (not the lock-file) into a requirements.txt file.

import configparser


def main():
    parser = configparser.ConfigParser()
    parser.read("Pipfile")

    packages = "packages"
    with open("requirements.txt", "w") as f:
        for key in parser[packages]:
            value = parser[packages][key]
            f.write(key + value.replace("\"", "") + "\n")


if __name__ == "__main__":
    main()
@linusguan

This comment has been minimized.

Copy link

@linusguan linusguan commented Feb 10, 2020

@frostming Hi, I found https://github.com/frostming/pipfile-requirements useful but why it didn't get integrated into pipenv?

@frostming

This comment has been minimized.

Copy link
Collaborator

@frostming frostming commented Feb 10, 2020

@linusguan The tool exists for those who don't want to install the big pipenv library, when you have installed pipenv, you can use pipenv lock -r

@linusguan

This comment has been minimized.

Copy link

@linusguan linusguan commented Feb 10, 2020

@frostming I find it quite useful for use with other tools that do not support pipfile.lock.
The problem with pipenv lock -r is it updates pipfile.lock so I cannot use it to produce deterministic build along with other tools. Something like pipenv lock -r --ignore-pipfile would be ideal.

@sergeyt

This comment has been minimized.

Copy link

@sergeyt sergeyt commented Feb 22, 2020

Here is yet another python script to generate requirements.txt from Pipfile.lock file with hashes:

import os
import json

__dir__ = os.path.dirname(os.path.realpath(__file__))


def read_json_file(path):
    with open(path) as f:
        return json.load(f)


def main():
    root = read_json_file(os.path.join(__dir__, 'Pipfile.lock'))

    for name, pkg in root["default"].items():
        version = pkg["version"]
        sep = lambda i: "" if i == len(pkg["hashes"]) - 1 else " \\"
        hashes = [f'--hash={t}{sep(i)}' for i, t in enumerate(pkg["hashes"])]
        tail = '' if len(hashes) == 0 else f' {hashes[0]}'
        print(f'{name} {version}{tail}')
        for h in hashes[1:]:
            print(f'    {h}')


if __name__ == "__main__":
    main()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.