poetryup not updating 0.x.y dependencies

[lyz-code]

Hi @MousaZeidBaker , first of all, thanks for the awesome package, it's much needed.

I'm encountering some trouble when updating libraries that are on the 0.x.y version, as they are ignored by poetryup.

Steps to reproduce:

The latest version of dlint is 0.12.0, first we set up the environment to use the 0.11.0

$: poetry add --dev dlint@^0.11.0

Updating dependencies
Resolving dependencies... (2.3s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing dlint (0.11.0)

$: poetryup
Updating dependencies
Resolving dependencies... (5.8s)

Writing lock file

No dependencies to install or update

Desired behavior

The dependency in pyproject.toml should be updated to ^0.12.0

Version

poetryup version 0.3.15

[lyz-code]

I just saw that I can reproduce the error with 1.x version packages. For example, pylint's last version is 2.12.2.

$: poetry add --dev pylint@^1

Updating dependencies
Resolving dependencies... (2.8s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing lazy-object-proxy (1.6.0)
  • Installing astroid (1.6.6)
  • Installing pylint (1.9.4)

$: poetryup
Updating dependencies
Resolving dependencies... (1.8s)

No dependencies to install or update

[MousaZeidBaker]

Hi @lyz-code, glad you like it!

This is a desired behavior and it's not poetryup that is in control over it but rather poetry itself. poetryup runs poetry --update and then modifies the pyproject.toml file, so if poetry doesn't update the package there is nothing poetryup can do. From the poetry dependency docs you can read the following

Caret requirements allow SemVer compatible updates to a specified version. An update is allowed if the new version number does not modify the left-most non-zero digit in the major, minor, patch grouping.

So the left-most non-zero digit in the major, minor, patch grouping will never be updated. The reason for this is to avoid introducing breaking changes.

They also provide a table with examples

[lyz-code]

Ouch, I saw your program in this issue, and I thought that it was able to upgrade the constrains of the package beyond the defined ones.

Would it be possible to add a flag like --upgrade or something similar to add this behaviour?

The problem that I want to solve is not to manually change the constrains of a package, but run a command that changes them all for me, then I can test if the package keeps on working.

Thanks!

[MousaZeidBaker]

@lyz-code created a PR that add the possibility to update dependencies to their latest available version. Could you please try it and come back with some feedback?

Steps to install from the branch:

• in your pyproject.toml file replace poetryup = x.y.z with poetryup = { git = "https://github.com/MousaZeidBaker/poetryup.git", branch = "poetryup-latest" }
• run poetry lock && poetry install
• run poetryup --help
• run poetryup --latest

[lyz-code]

Hey @MousaZeidBaker, thanks for implementing the solution so fast! Oo

I've created an empty virtualenv and run the commands that you told me.

pip install poetry
poetry new package
poetry add git+https://github.com/MousaZeidBaker/poetryup.git#poetryup-latest
poetry lock && poetry install

poetryup --help shows the --latest in the message, but when I run poetryup --latest it downgrades poetryup:

$: poetryup --latest

INFO:root:Updating dependencies to their latest available version
Using version ^0.3.15 for poetryup

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 1 update, 2 removals

  • Removing click (8.0.3)
  • Removing typer (0.4.0)
  • Updating poetryup (0.4.0 -> 0.3.15)
Using version ^6.2.5 for pytest

Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because package depends on both pytest (^6.2.5) and pytest (^5.2), version solving failed.

  at /tmp/tmp.vZGdVbq1wp/env/lib/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
INFO:root:Found 2 dependencies in pyproject.toml
INFO:root:Bumping skipped for dependency named: poetryup

I've tried to fool poetry by commenting out the poetryup dependency but then
it removes it. Can you think how can we bypass this problem?

I see however that we can run once poetryup --latest before it downgrades it,
so I've tried with dlint

$: poetry add --dev dlint@^0.11.0
$: poetryup --latest

INFO:root:Updating dependencies to their latest available version
Using version ^0.12.0 for dlint

Updating dependencies
Resolving dependencies... (3.3s)

  SolverProblemError

  Because package depends on both dlint (^0.12.0) and dlint (^0.11.0), version solving failed.

  at /tmp/tmp.vZGdVbq1wp/env/lib/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
Using version ^0.3.15 for poetryup

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 0 installs, 1 update, 2 removals

  • Removing click (8.0.3)
  • Removing typer (0.4.0)
  • Updating poetryup (0.4.0 -> 0.3.15)
INFO:root:Found 2 dependencies in pyproject.toml
INFO:root:Bumping skipped for dependency named: poetryup

The same error happens as with pytest, it tries to update it to the correct
version, but it looks like the constrains are not updated in time and they get
in conflict.

Funnily though, if after the downgrade of poetryup you try to install again the
dlint dependency, poetryup is upgraded again xD:

poetry add --dev dlint@^0.11.0

Updating dependencies
Resolving dependencies... (6.5s)

Writing lock file

Package operations: 2 installs, 1 update, 0 removals

  • Installing click (8.0.3)
  • Installing typer (0.4.0)
  • Updating poetryup (0.3.15 -> 0.4.0 74bd22b)

If you directly use poetry add --dev dlint@latest the upgrade works as
expected:

Using version ^0.12.0 for dlint

Updating dependencies
Resolving dependencies... (3.8s)

Writing lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating dlint (0.11.0 -> 0.12.0)

[MousaZeidBaker]

Hmm yes poetryup will update itself since it loops through all packages in pyproject.toml file and then runs poetry add package@latest on each package. To avoid that, completely remove poetryup from pyproject.toml file and instead install it with pip python -m pip install git+https://github.com/MousaZeidBaker/poetryup.git@poetryup-latest.

I tried it with dlint and it worked however I didn't have any other package can you try that? Again its poetry that will do the update but sometimes if you have multiple packages they may have their own dependencies and if they have specified a specific version constraint then poetry will fail.

[lyz-code]

Hi, installing it with pip solved the update of poetryup but I still encounter the dependency error :S. Here is the full traceback:

$: virtualenv -p `which python3.9` env
created virtual environment CPython3.9.8.final.0-64 in 188ms
  creator CPython3Posix(dest=/tmp/tmp.WNwwD7mGGr/env, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle,
  wheel=bundle, via=copy, app_data_dir=~/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==58.5.3, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivato
r

$: source env/bin/activate

$: python -m pip install git+https://github.com/MousaZeidBaker/poetryup
.git@poetryup-latest
Collecting git+https://github.com/MousaZeidBaker/poetryup.git@poetryup-latest
  Cloning https://github.com/MousaZeidBaker/poetryup.git (to revision poetryup-latest) to /tmp/pip-req-bui
ld-k35mbao9
  Running command git clone --filter=blob:none -q https://github.com/MousaZeidBaker/poetryup.git /tmp/pip-
req-build-k35mbao9
  Running command git checkout -b poetryup-latest --track origin/poetryup-latest
  Switched to a new branch 'poetryup-latest'
  Branch 'poetryup-latest' set up to track remote branch 'poetryup-latest' from 'origin'.
  Resolved https://github.com/MousaZeidBaker/poetryup.git to commit 74bd22bd15b3c3cd2b461f6a2f91d6c5016393
f7
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting tomlkit<0.8.0,>=0.7.2
  Using cached tomlkit-0.7.2-py2.py3-none-any.whl (32 kB)
Collecting typer<0.5.0,>=0.4.0
  Using cached typer-0.4.0-py3-none-any.whl (27 kB)
Collecting click<9.0.0,>=7.1.1
  Using cached click-8.0.3-py3-none-any.whl (97 kB)
Building wheels for collected packages: poetryup
  Building wheel for poetryup (pyproject.toml) ... done
  Created wheel for poetryup: filename=poetryup-0.4.0-py3-none-any.whl size=5836 sha256=c43cc94613b28e6e9b
5e5c3ee186ac28c8d8a0c34dcb7c43efd0b2acf581cea4
  Stored in directory: /tmp/pip-ephem-wheel-cache-no3d4qmk/wheels/19/87/6b/1fc6a1f95bf2f208fffaba4e3c77c05
e255f01f3e20b26993b
Successfully built poetryup
Installing collected packages: click, typer, tomlkit, poetryup
Successfully installed click-8.0.3 poetryup-0.4.0 tomlkit-0.7.2 typer-0.4.0

$: poetry new test-project                                       [env]
Created package test_project in test-project

$: cd test-project                                               [env]

$: poetry add --dev dlint@^0.11.0                   [env]

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 13 installs, 0 updates, 0 removals

  • Installing mccabe (0.6.1)
  • Installing pycodestyle (2.7.0)
  • Installing pyflakes (2.3.1)
  • Installing pyparsing (3.0.6)
  • Installing attrs (21.2.0)
  • Installing flake8 (3.9.2)
  • Installing more-itertools (8.12.0)
  • Installing packaging (21.3)
  • Installing pluggy (0.13.1)
  • Installing py (1.11.0)
  • Installing wcwidth (0.2.5)
  • Installing dlint (0.11.0)
  • Installing pytest (5.4.3)

$: poetryup --latest                                [env]
INFO:root:Updating dependencies to their latest available version
Using version ^0.12.0 for dlint

Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because test-project depends on both dlint (^0.12.0) and dlint (^0.11.0), version solving failed.

  at ~/.local/lib/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
Using version ^6.2.5 for pytest

Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because test-project depends on both pytest (^6.2.5) and pytest (^5.2), version solving failed.

  at ~/.local/lib/python3.7/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes
INFO:root:Found 2 dependencies in pyproject.toml

[lyz-code]

I think I realized the problem. The dependency was configured as a development one, instead of a main one, so when we try to add it as a normal one with poetry add dlint@latest they enter in conflict. Maybe we need to iterate first over the main dependencies and then over the development and add them as poetry add --dev {dep}@latest

[MousaZeidBaker]

Aha that explains why you were facing errors while I did not. That will be hard to support especially since poetry is introducing dependency groups that can have custom names...

[lyz-code]

Maybe we can start by providing support to the normal and dev dependencies, and once those groups are introduced and a user needs to upgrade them to the latest we can see how to deal with it?

Probably poetry will provide a command or something that lists them and we can iterate through

[MousaZeidBaker]

Could be a start to begin with the "main" and dev dependencies and then later consider the other groups. I think we need to add this new logic to the lookup operation that poetryup currently have. However, I don't feel I have the time to dig deeper into this right now, but you are more than welcome to dig deeper if you can.

[lyz-code]

I completely understand. Sadly after reading Should You Use Upper Bound Version Constraints? by Henry Schreiner I'm moving away from poetry and into pdm, so I won't have time either for this :(.

Thank you for all the time invested in this issue. It's been a nice experience :)

If you feel like closing the issue, it's fine by me

[MousaZeidBaker]

@lyz-code FYI: poetryup can now update all dependencies to their LATEST available version by setting a flag as following poetryup --latest

[mbnoimi]

I've a similar issue with the recent version:

  Because app depends on both pytest (^5.4.3) and pytest (^7.1.2), version solving failed.

  at /opt/poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

root@ab266c0ea618:/app# poetry --version
Poetry version 1.1.14
root@ab266c0ea618:/app# poetry show
Skipping virtualenv creation, as specified in config file.
alembic            1.8.1     A database migration tool for SQLAlchemy.
amqp               2.6.1     Low-level AMQP client for Python (fork of amqplib).
appdirs            1.4.4     A small Python module for determining appropriate platform-specific dirs, e.g. a ...
attrs              21.4.0    Classes Without Boilerplate
autoflake          1.4       Removes unused imports and unused variables
bcrypt             3.2.2     Modern password hashing for your software and your servers
billiard           3.6.4.0   Python multiprocessing fork with improvements and bugfixes
black              19.10b0   The uncompromising code formatter.
cachetools         5.2.0     Extensible memoizing collections and decorators
celery             4.4.7     Distributed Task Queue.
certifi            2022.6.15 Python package for providing Mozilla's CA Bundle.
cffi               1.15.1    Foreign Function Interface for Python calling C code.
chardet            5.0.0     Universal encoding detector for Python 3
charset-normalizer 2.1.0     The Real First Universal Charset Detector. Open, modern and actively maintained a...
click              7.1.2     Composable command line interface toolkit
coverage           6.4.2     Code coverage measurement for Python
cryptography       37.0.4    cryptography is a package which provides cryptographic recipes and primitives to ...
cssselect          1.1.0     cssselect parses CSS3 Selectors and translates them to XPath 1.0
cssutils           2.5.0     A CSS Cascading Style Sheets library for Python
dnspython          2.2.1     DNS toolkit
ecdsa              0.18.0    ECDSA cryptographic signature library (pure python)
email-validator    1.2.1     A robust email syntax and deliverability validation library.
emails             0.5.15    Modern python library for emails.
fastapi            0.54.2    FastAPI framework, high performance, easy to learn, fast to code, ready for produ...
flake8             3.9.2     the modular source code checker: pep8 pyflakes and co
greenlet           1.1.2     Lightweight in-process concurrent programming
gunicorn           20.1.0    WSGI HTTP Server for UNIX
h11                0.9.0     A pure-Python, bring-your-own-I/O implementation of HTTP/1.1
httptools          0.1.2     A collection of framework independent HTTP protocol utils.
idna               3.3       Internationalized Domain Names in Applications (IDNA)
isort              4.3.21    A Python utility / library to sort Python imports.
jinja2             2.11.3    A very fast and expressive template engine.
kombu              4.6.11    Messaging library for Python.
lxml               4.9.1     Powerful and Pythonic XML processing library combining libxml2/libxslt with the E...
mako               1.2.1     A super-fast templating language that borrows the best ideas from the existing te...
markupsafe         2.1.1     Safely add untrusted strings to HTML/XML markup.
mccabe             0.6.1     McCabe checker, plugin for flake8
more-itertools     8.13.0    More routines for operating on iterables, beyond itertools
mypy               0.770     Optional static typing for Python
mypy-extensions    0.4.3     Experimental type system extensions for programs checked with the mypy typechecker.
packaging          21.3      Core utilities for Python packages
passlib            1.7.4     comprehensive password hashing framework supporting over 30 schemes
pathspec           0.9.0     Utility library for gitignore style pattern matching of file paths.
pluggy             0.13.1    plugin and hook calling mechanisms for python
poetryup           0.9.0     Update dependencies and bump their version in the pyproject.toml file
premailer          3.10.0    Turns CSS blocks into style attributes
psycopg2-binary    2.9.3     psycopg2 - Python-PostgreSQL Database Adapter
py                 1.11.0    library with cross-python path, ini-parsing, io, code, log facilities
pyasn1             0.4.8     ASN.1 types and codecs
pycodestyle        2.7.0     Python style guide checker
pycparser          2.21      C parser in Python
pydantic           1.9.1     Data validation and settings management using python type hints
pyflakes           2.3.1     passive checker of Python programs
pyparsing          3.0.9     pyparsing module - Classes and methods to define and execute parsing grammars
pytest             5.4.3     pytest: simple powerful testing with Python
pytest-cov         2.12.1    Pytest plugin for measuring coverage.
python-dateutil    2.8.2     Extensions to the standard Python datetime module
python-jose        3.3.0     JOSE implementation in Python
python-multipart   0.0.5     A streaming multipart parser for Python
pytz               2022.1    World timezone definitions, modern and historical
raven              6.10.0    Raven is a client for Sentry (https://getsentry.com)
regex              2022.7.9  Alternative regular expression module, to replace re.
requests           2.28.1    Python HTTP for Humans.
rsa                4.8       Pure-Python RSA implementation
six                1.16.0    Python 2 and 3 compatibility utilities
sqlalchemy         1.4.39    Database Abstraction Library
sqlalchemy-stubs   0.3       SQLAlchemy stubs and mypy plugin
starlette          0.13.2    The little ASGI library that shines.
tenacity           6.3.1     Retry code until it succeeds
toml               0.10.2    Python Library for Tom's Obvious, Minimal Language
tomlkit            0.11.1    Style preserving TOML library
typed-ast          1.4.3     a fork of Python 2 and 3 ast modules with type comment support
typer              0.4.2     Typer, build great CLIs. Easy to code. Based on Python type hints.
typing-extensions  4.3.0     Backported and Experimental Type Hints for Python 3.7+
urllib3            1.26.10   HTTP library with thread-safe connection pooling, file post, and more.
uvicorn            0.11.8    The lightning-fast ASGI server.
uvloop             0.16.0    Fast implementation of asyncio event loop on top of libuv
vine               1.3.0     Promises, promises, promises.
wcwidth            0.2.5     Measures the displayed width of unicode strings in a terminal
websockets         8.1       An implementation of the WebSocket Protocol (RFC 6455 & 7692)
root@ab266c0ea618:/app# 

[mbnoimi]

pyproject.toml

[tool.poetry]
name = "app"
version = "0.1.0"
description = ""
authors = ["Admin <admin@example.com>"]

[tool.poetry.dependencies]
python = "^3.7"
uvicorn = "^0.11.8"
fastapi = "^0.54.2"
python-multipart = "^0.0.5"
email-validator = "^1.2.1"
requests = "^2.28.1"
celery = "^4.4.7"
passlib = {extras = ["bcrypt"], version = "^1.7.4"}
tenacity = "^6.3.1"
pydantic = "^1.9.1"
emails = "^0.5.15"
raven = "^6.10.0"
gunicorn = "^20.1.0"
jinja2 = "^2.11.3"
psycopg2-binary = "^2.9.3"
alembic = "^1.8.1"
sqlalchemy = "^1.4.39"
pytest = "^5.4.3"
python-jose = {extras = ["cryptography"], version = "^3.3.0"}

[tool.poetry.dev-dependencies]
mypy = "^0.770"
black = "^19.10b0"
isort = "^4.3.21"
autoflake = "^1.4"
flake8 = "^3.9.2"
pytest = "^5.4.3"
sqlalchemy-stubs = "^0.3"
pytest-cov = "^2.12.1"
poetryup = "^0.9.0"

[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
line_length = 88
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[MousaZeidBaker]

@mbnoimi its because you've declared pytest package in both dependencies and dev-dependencies so when poetryup tries to update one of the groups (it takes one group at a time) it results in solver error, you need to decide whether you want pytest to be part of dependencies or dev-dependencies.

[mbnoimi]

@MousaZeidBaker Thanks. It works fine