Remove hard dependency on urllib3>2

[alanvgreen]

It would be very helpful if one of anehx's suggestions in #1378 could be implemented, allowing either a later version of urllib3 1.x.x or urllib3 2 to be used.

botocore3 - a package that many minio users would have installed simultaneously with minio - has the requirement 'urllib3>=1.25.4,<2.1 ; python_version >= "3.10"'. minio is obviously not the first project to encounter versioning drama due to urllib3's v1 vs v2 changes.

For our organisation, mandating urllib3>2 at this point is difficult. There are many unfortunate first and third party dependencies and a surprising number of conda environments that need to be updated to use urllib3, and we are only part way through the process. Happy to discuss the situation further, but I don't believe we are unique in finding version pinning issues time consuming to deal with. It wouldn't surprise me to learn that, across all of minio's users, adding the requirement of urllib3>2 caused a thousand hours' unscheduled work.

Implementing a softer transition from urllib3 v1 to v2 will make it easier to upgrade our other package dependencies on urllib3.

[harshavardhana]

We will see how this can be done @alanvgreen

[dhilazo]

Agree with @alanvgreen this can lead to ImportError in any package using minio too. It happened to me in pymilvus where I got:

File "/usr/local/lib/python3.11/site-packages/minio/api.py", line 44, in <module>
    from urllib3.response import BaseHTTPResponse
ImportError: cannot import name 'BaseHTTPResponse' from 'urllib3.response'

due to the v2 breaking change

[suyuchenxm]

@dhilazo any solutions to fix this issue? Thanks!

[dhilazo]

@suyuchenxm if you are referring to the pymilvus case, you might have to manually specify version lower than 7.2.1 for minio in your requirements. Since pymilvus doesn't specify any version for minio you will have to do it yourself unfortunately.

The bugfix 7.2.2 doesn't have this problem as it specifies the hard requirement of having urllib3>2 but since pypi by default tries to install the latest version, it will install version 7.2.1 which contains this bug. Imho, release 7.2.1 should be removed from the pypi registry

[harshavardhana]

@suyuchenxm if you are referring to the pymilvus case, you might have to manually specify version lower than 7.2.1 for minio in your requirements. Since pymilvus doesn't specify any version for minio you will have to do it yourself unfortunately.

The bugfix 7.2.2 doesn't have this problem as it specifies the hard requirement of having urllib3>2 but since pypi by default tries to install the latest version, it will install version 7.2.1 which contains this bug. Imho, release 7.2.1 should be removed from the pypi registry

7.2.1 will stay as is, move to v7.2.2 if you can if not stay at v7.2.0 - until a different workaround is found.

You simply have conflicting dependency using two different SDKs for the same work.

You could use a custom namespaced deployment where you are using virtualenv.

There are various workarounds.

[harshavardhana]

I upgraded all the dependencies successfully, and everything checks out fine and works without anything breaking.

pip install awscli botocore boto3 --upgrade
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: awscli in /home/harsha/.local/lib/python3.10/site-packages (1.32.9)
Requirement already satisfied: botocore in /home/harsha/.local/lib/python3.10/site-packages (1.34.9)
Requirement already satisfied: boto3 in /home/harsha/.local/lib/python3.10/site-packages (1.24.2)
Collecting boto3
  Downloading boto3-1.34.9-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 745.1 kB/s eta 0:00:00
Requirement already satisfied: PyYAML<6.1,>=3.10 in /usr/lib/python3/dist-packages (from awscli) (5.4.1)
Requirement already satisfied: docutils<0.17,>=0.10 in /home/harsha/.local/lib/python3.10/site-packages (from awscli) (0.16)
Requirement already satisfied: rsa<4.8,>=3.1.2 in /home/harsha/.local/lib/python3.10/site-packages (from awscli) (4.7.2)
Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/harsha/.local/lib/python3.10/site-packages (from awscli) (0.10.0)
Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /usr/lib/python3/dist-packages (from awscli) (0.4.4)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore) (2.8.1)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from botocore) (0.10.0)
Requirement already satisfied: urllib3<2.1,>=1.25.4 in /home/harsha/.local/lib/python3.10/site-packages (from botocore) (2.0.7)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python3/dist-packages (from rsa<4.8,>=3.1.2->awscli) (0.4.8)
Installing collected packages: boto3
  Attempting uninstall: boto3
    Found existing installation: boto3 1.24.2
    Uninstalling boto3-1.24.2:
      Successfully uninstalled boto3-1.24.2
Successfully installed boto3-1.34.9

[notice] A new release of pip available: 22.2.2 -> 23.3.2
[notice] To update, run: python3 -m pip install --upgrade pip

~ pip install minio --upgrade
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: minio in /home/harsha/.local/lib/python3.10/site-packages (7.2.2)
Requirement already satisfied: pycryptodome in /home/harsha/.local/lib/python3.10/site-packages (from minio) (3.19.1)
Requirement already satisfied: urllib3>=2.0 in /home/harsha/.local/lib/python3.10/site-packages (from minio) (2.0.7)
Requirement already satisfied: argon2-cffi in /home/harsha/.local/lib/python3.10/site-packages (from minio) (23.1.0)
Requirement already satisfied: certifi in /home/harsha/.local/lib/python3.10/site-packages (from minio) (2022.12.7)
Requirement already satisfied: typing-extensions in /usr/lib/python3/dist-packages (from minio) (3.10.0.2)
Requirement already satisfied: argon2-cffi-bindings in /home/harsha/.local/lib/python3.10/site-packages (from argon2-cffi->minio) (21.2.0)
Requirement already satisfied: cffi>=1.0.1 in /home/harsha/.local/lib/python3.10/site-packages (from argon2-cffi-bindings->argon2-cffi->minio) (1.15.1)
Requirement already satisfied: pycparser in /home/harsha/.local/lib/python3.10/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->minio) (2.21)

[notice] A new release of pip available: 22.2.2 -> 23.3.2
[notice] To update, run: python3 -m pip install --upgrade pip

>>> from minio import Minio
>>> 
>>> client = Minio("play.min.io",
...     access_key="Q3AM3UQ867SPQQA43P2F",
...     secret_key="zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG",
... )
>>> for i in client.list_objects("testbucket"):
...     print(i.object_name)
... 
testdata
>>> 
>>> import minio
>>> minio.__version__
'7.2.2'

aws s3api list-buckets --profile s3
{
    "Buckets": [
        {
            "Name": "219432304946-us-east-1-bhdl-talend-dev-va",
            "CreationDate": "2022-02-25T12:54:02.000Z"
        },

Both of them work fine along with urllib2.0 upgraded dependency.

[balamurugana]

Please check PR #1384 solves your problem.

[alonme-loops]

@balamurugana when can we expect a release with this fix?

[minio-trusted]

Removed hard dependency with v7.2.3