Pipenv lock changes markers in a non deterministic way

[matpompili]

Issue description

I'm running pipenv version 2020.11.15 in a Gitlab CI pipeline.
I use pipenv lock and then git diff Pipfile.lock to see if the Pipfile.lock was updated, meaning that the version on the repository is not up-to-date anymore.

Occasionally, pipenv updates the lock file just changing the Python 4 specifier as follows:

diff --git a/Pipfile.lock b/Pipfile.lock
index c8d17c6..0228d56 100644
--- a/Pipfile.lock
+++ b/Pipfile.lock
@@ -225,7 +225,7 @@
                 "sha256:0a943902919f65c5684ac4e0154b1ad4fac6dcaa5d9f3426b732f1c8b5419be6",
                 "sha256:2bb1680aad211e3c9944dbce1d4ba09a989f04e238296c87fe2139faa26d655d"
             ],
-            "markers": "python_version >= '3.6' and python_version < '4.0'",
+            "markers": "python_version >= '3.6' and python_version < '4'",
             "version": "==5.8.0"
         },
         "jedi": {
@@ -881,7 +881,7 @@
                 "sha256:2f4da4594db7e1e110a944bb1b551fdf4e6c136ad42e4234131391e21eb5b0df",
                 "sha256:e7b021f7241115872f92f43c6508082facffbd1c048e3c6e2bb9c2a157e28937"
             ],
-            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4.0'",
+            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'",
             "version": "==1.26.4"
         },
         "wcwidth": {

And sometimes it does the opposite thing, changing from python_version < '4' to python_version < '4.0'.
This of course breaks the pipeline, because there is difference in the Pipfile.lock, even though that difference is not significant.

Expected result

pipenv lock should change the lock file in a deterministic way, either always with python_version < '4' or python_version < '4.0'.

[matpompili]

As a temporary workaround, I am removing the "markers": "python_version lines before comparing the old and new Pipfiles.

Here the section of my .gitlab-ci.yml:

pipenv:
  stage: analysis
  script:
    - cp Pipfile.lock Pipfile.lock.old
    - pipenv lock
    - sed -i '/\"markers\":\ \"python_version/d' Pipfile.lock # remove markers
    - sed -i '/\"markers\":\ \"python_version/d' Pipfile.lock.old # remove markers
    - git --no-pager diff --no-index Pipfile.lock Pipfile.lock.old | tee diff.log
    - RESULT=$( head -n 1 diff.log | { grep -o 'diff' || true; } )
    - RETURN=0
    - if [ "$RESULT" = "diff" ]; then RETURN=1; echo "Something changed!"; fi 
    - exit $RETURN
  after_script:
    - pip install anybadge
    - RESULT=$( head -n 1 diff.log | { grep -o 'diff' || true; } )
    - SCORE=passed
    - if [ "$RESULT" = "diff" ]; then SCORE=outdated; echo "Something changed!"; fi 
    - anybadge -l pipenv -v $SCORE -f pipenv.svg outdated=red passed=green
  artifacts:
    paths:
      - pipenv.svg
      - diff.log
    when: always

[jshwi]

Hi, I am also getting this result, but on every pipenv lock call. It just goes back and forth between '4' and '4.0'.

[matpompili]

Something changed in the latest version, now I am getting both python_version and python_full_version markers.
The following change to the .gitlab-ci.yml works

 - sed -i '/\"markers\":[[:print:]]*python_[[:print:]]*version/d' Pipfile.lock  # remove markers
 - sed -i '/\"markers\":[[:print:]]*python_[[:print:]]*version/d' Pipfile.lock.old  # remove markers

[matteius]

@matpompili and @jshwi please see: #4967
TLDR: Pip 22.0.4 upgrade branch seems to make this deterministic again.

Closing this ticket now since we are already tracking it in the newer ticket.

[porn]

Still having the same behaviour after upgrading to pip 22.0.4. Using pipenv 2022.5.2:

diff --git tasks/Pipfile.lock tasks/Pipfile.lock
index f039f8175..086dbcb0c 100644
--- tasks/Pipfile.lock
+++ tasks/Pipfile.lock
@@ -68,7 +68,7 @@
                 "sha256:211800f725cdecedcbcf4c753bbd22d248312b37d130f06045434acb7d9b34e1",
                 "sha256:35dd9063244263e63bd0bd24ea61e3015b00272cead084b2c40d788b0f857c46"
             ],
-            "markers": "python_version >= '3.7' and python_version < '4.0'",
+            "markers": "python_version >= '3.7' and python_version < '4'",
             "version": "==1.10.0"
         },
         "certifi": {

reproduced with pip 22.1 as well.

...is it just me?

[gitpushdashf]

Still having the same behaviour after upgrading to pip 22.0.4. Using pipenv 2022.5.2:

diff --git tasks/Pipfile.lock tasks/Pipfile.lock
index f039f8175..086dbcb0c 100644
--- tasks/Pipfile.lock
+++ tasks/Pipfile.lock
@@ -68,7 +68,7 @@
                 "sha256:211800f725cdecedcbcf4c753bbd22d248312b37d130f06045434acb7d9b34e1",
                 "sha256:35dd9063244263e63bd0bd24ea61e3015b00272cead084b2c40d788b0f857c46"
             ],
-            "markers": "python_version >= '3.7' and python_version < '4.0'",
+            "markers": "python_version >= '3.7' and python_version < '4'",
             "version": "==1.10.0"
         },
         "certifi": {

reproduced with pip 22.1 as well.

...is it just me?

I'm getting this as well.

[matteius]

Sounds like pip released a new version that may have changed in it, but pipenv vendors in pip 22.0.4 and nothing has been done to test the new version of pip. Perhaps this is a pip bug.

[matteius]

Want to check the behavior of the newly vendored pip==22.2.1 that is in main against this ticket because I saw something odd working on vistir where re-locking changed the markers from my initial lock (which had problematic markers). If anyone has energy to see if they can reproduce this issue on pipenv main or not, will let me know if I need to track down a Pip issue report I read about once (where perhaps the fix for this was reverted).

[porn]

Hey @matteius ,

using pip 22.2.1 and pipenv 2022.7.24 keeps altering python_full_version and python_version randomly:

diff --git tasks/Pipfile.lock tasks/Pipfile.lock
index 2149cfcdd0..2eeb60add2 100644
--- tasks/Pipfile.lock
+++ tasks/Pipfile.lock
@@ -84,7 +84,7 @@
                 "sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
                 "sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
             ],
-            "markers": "python_full_version >= '3.6.0'",
+            "markers": "python_version >= '3.6'",
             "version": "==2022.6.15"
         },
         "cffi": {

[matteius]

I believe this is not an issue on the main branch as we force the usage of the vendor'd pip which eliminates the using pip VERSION_X with pipenv, as it uses a consistent version of pip for resolution.

[gitpushdashf]

Still having this on the latest pipenv which should have the fix, right?

[matteius]

@gitpushdashf Well technically it was vendoring of pip that fixed it and then we of course continue to vendor in new versions of pip as they are released. We recently merged in a change that hasn't been released to force the environment to use the vendor'd version of pip in more places but it has not been released yet. There are two possibilities -- one is that the version of pip you have installed doesn't match the version pipenv has vendored and that could perhaps create inconsistencies. Or it is possible that something changed again in pip creating this inconsistency, but it was definitely pip vendoring that fixed it the prior time.