Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dependency versions does not prevent duplicate dependency resolution process. #6089

Closed
3 tasks done
krlng opened this issue Jul 29, 2022 · 3 comments
Closed
3 tasks done
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged

Comments

@krlng
Copy link

krlng commented Jul 29, 2022

  • I am on the latest Poetry version.
  • I have searched the issues of this repo: Related to issue 5121, but while they want a more intelligent resolution, I wonder why version pinning of dependencies does not work.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

Infos:

Description:
The pyproject I linked is a minimal version of our real pyorject.yaml (see below). It contains two major dependencies, that both rely on different versions of numpy depending on the python-version, platform, and chipset you are on:

pandas:

install_requires =
    numpy>=1.18.5; platform_machine!='aarch64' and platform_machine!='arm64' and python_version<'3.10'
    numpy>=1.19.2; platform_machine=='aarch64' and python_version<'3.10'
    numpy>=1.20.0; platform_machine=='arm64' and python_version<'3.10'
    numpy>=1.21.0; python_version>='3.10'

opencv-python:

  "numpy==1.13.3; python_version=='3.6' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.14.5; python_version=='3.7' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.17.3; python_version=='3.8' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.19.3; python_version<='3.9' and sys_platform == 'linux' and platform_machine == 'aarch64'",
  "numpy==1.21.0; python_version<='3.9' and sys_platform == 'darwin' and platform_machine == 'arm64'",
  "numpy==1.19.3; python_version=='3.9' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.21.2; python_version>='3.10'"

When you resolve the project (poetry lock -vvv), poetry realizes that there are 16 different possible combinations and resolves all of them.

Excerpt of the logs:


   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.21.2) with markers python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64", numpy (>=1.19.3) with markers python_version >= "3.6" and platform_system == "Linux" and platform_machine == "aarch64" or python_version >= "3.9", numpy (>=1.14.5) with markers python_version >= "3.7" and numpy (>=1.17.3) with markers python_version >= "3.8".
   1: Version solving took 0.902 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('opencv-python-headless', '4.6.0.66'): {'numpy': <Dependency numpy (>=1.21.2)>}}).
   1: fact: resolve-dependencies is 0.1.0
   1: derived: resolve-dependencies
   1: fact: resolve-dependencies depends on opencv-python-headless (^4.6)
   1: fact: resolve-dependencies depends on pandas (^1.3)
   1: fact: resolve-dependencies depends on numpy (=1.21.2)
   1: selecting resolve-dependencies (0.1.0)
   1: derived: numpy (==1.21.2)
   1: derived: pandas (>=1.3,<2.0)
   1: derived: opencv-python-headless (>=4.6,<5.0)
   1: selecting numpy (1.21.2)
   1: fact: opencv-python-headless (4.6.0.66) depends on numpy (>=1.21.2)
   1: selecting opencv-python-headless (4.6.0.66)
PyPI: Getting info for pandas (1.4.3) from PyPI
   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.18.5) with markers platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", numpy (>=1.19.2) with markers platform_machine == "aarch64" and python_version < "3.10" and numpy (>=1.20.0) with markers platform_machine == "arm64" and python_version < "3.10".
   1: Version solving took 0.138 seconds.
   1: Tried 1 solutions.

This will continue for 15 more strategies, so in the end, there will be these combinations:


({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) >	}
}), ({
	Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >	},
	Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) >	}
})

It can quickly be seen, that no library has a more upper bound than numpy < 2.0, so using the newest version of numpy should just be fine. As the resolution system does not realize that on its own, I pinned my numpy to 1.21.2. However, this simply gets ignored, and the process is exactly the same, no matter if I specified this.

Why is this the case? I would suspect it is related to the different platform_machine and sys_platform markers. This is why I also tried to pin numpy for every possible combination of those, but this also did not help.

Is there any way to prevent poetry from resolving all those different strategies?

While this is more a small blemish in the posted example, it is a major issue in our real use case, as that one includes multiple packages living in a private package repository, which will interrupt the connection after a few minutes because of hitting the rate limit, which makes it practically impossible to use poetry at all.

@krlng krlng added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Jul 29, 2022
@krlng
Copy link
Author

krlng commented Aug 16, 2022

Any comments or ideas? Would not like to see this issue going stale

@radoering
Copy link
Member

It can quickly be seen, that no library has a more upper bound than numpy < 2.0, so using the newest version of numpy should just be fine.

That's too short-sighted. Even if there is no upper bound, it is possible that a (transient) dependency of the latest numpy version conflicts with another dependency for a specific platform marker so that an older numpy version has to be used for that platform.

I pinned my numpy to 1.21.2. However, this simply gets ignored, and the process is exactly the same, no matter if I specified this.

That's just another constraint. It probably helps to avoid backtracking, so that the only version that is tried for all overrides is numpy 1.21.2 and if there is a conflict, no other versions are tried.

Is there any way to prevent poetry from resolving all those different strategies?

At the moment you can only restrict the python version (which you have already done in the given pyproject.toml). This avoids those overrides where the python version does not match. (I'm not sure how good this works in poetry 1.1.x so you may have to try 1.2 beta.)

If only specific platforms are relevant to you, you may be interested in #4956.

@finswimmer finswimmer closed this as not planned Won't fix, can't repro, duplicate, stale Jan 24, 2023
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Something isn't working as expected status/triage This issue needs to be triaged
Projects
None yet
Development

No branches or pull requests

3 participants