Potential dependency conflicts between plaso, chardet, idna and urllib3 when using pip

[NeolithEra]

Hi, plaso directly and transitively introduced multiple versions of chardet, idna and urllib3.

As shown in the following full dependency graph of plaso, plaso requires chardet (the latest version), while the installed version of requests(2.22.0) requires chardet>=3.0.2,<3.1.0, urllib3>=1.21.1,<1.26 and idna>=2.5,<2.9.

According to pip's “first found wins” installation strategy, chardet 3.0.4, urllib3 1.25.3 and idna 2.8 are the actually installed versions.

Although the first found package versions chardet 3.0.4, urllib3 1.25.3 and idna 2.8 just satisfy their corresponding later dependency constraints (chardet>=3.0.2,<3.1.0), (urllib3>=1.21.1,<1.26) and (idna>=2.5,<2.9), it will lead to a build failure once developers release a newer version of chardet, urllib3 and idna.

Dependency tree--------

plaso(version range:)
| +-artifacts(version range:>=20190305)
| | +-pip(version range:>=7.0.0)
| | +-pyyaml(version range:>=3.10)
| +-backports.lzma(version range:)
| +-bencode.py(version range:)
| +-biplist(version range:>=1.0.3)
| +-certifi(version range:>=2016.9.26)
| +-chardet(version range:>=2.0.1)
| +-defusedxml(version range:>=0.5.0)
| +-dfdatetime(version range:>=20180704)
| | +-pip(version range:>=7.0.0)
| +-dfvfs(version range:>=20190609)
| | +-backports.lzma(version range:)
| | +-dfdatetime(version range:>=20181025)
| | | +-pip(version range:>=7.0.0)
| | +-dtfabric(version range:>=20170524)
| | | +-pip(version range:>=7.0.0)
| | | +-pyyaml(version range:>=3.10)
| | +-libbde-python(version range:>=20140531)
| | +-libewf-python(version range:>=20131210)
| | +-libfsapfs-python(version range:>=20181205)
| | +-libfsntfs-python(version range:>=20151130)
| | +-libfvde-python(version range:>=20160719)
| | +-libfwnt-python(version range:>=20160418)
| | +-libqcow-python(version range:>=20131204)
| | +-libsigscan-python(version range:>=20150627)
| | +-libsmdev-python(version range:>=20140529)
| | +-libsmraw-python(version range:>=20140612)
| | +-libvhdi-python(version range:>=20131210)
| | +-libvmdk-python(version range:>=20140421)
| | +-libvshadow-python(version range:>=20160109)
| | +-libvslvm-python(version range:>=20160109)
| | +-pip(version range:>=7.0.0)
| | +-pycrypto(version range:>=2.6)
| | +-pytsk3(version range:>=20160721)
| | +-pyyaml(version range:>=3.10)
| +-dfwinreg(version range:>=20180712)
| +-dtfabric(version range:>=20181128)
| | +-pip(version range:>=7.0.0)
| | +-pyyaml(version range:>=3.10)
| +-elasticsearch(version range:>=6.0)
| +-future(version range:>=0.16.0)
| +-idna(version range:>=2.5)
| +-libbde-python(version range:>=20140531)
| +-libesedb-python(version range:>=20150409)
| +-libevt-python(version range:>=20120410)
| +-libevtx-python(version range:>=20141112)
| +-libewf-python(version range:>=20131210)
| +-libfsapfs-python(version range:>=20181205)
| +-libfsntfs-python(version range:>=20151130)
| +-libfvde-python(version range:>=20160719)
| +-libfwnt-python(version range:>=20180117)
| +-libfwsi-python(version range:>=20150606)
| +-liblnk-python(version range:>=20150830)
| +-libmsiecf-python(version range:>=20150314)
| +-libolecf-python(version range:>=20151223)
| +-libqcow-python(version range:>=20131204)
| +-libregf-python(version range:>=20150315)
| +-libscca-python(version range:>=20190605)
| +-libsigscan-python(version range:>=20190629)
| +-libsmdev-python(version range:>=20140529)
| +-libsmraw-python(version range:>=20140612)
| +-libvhdi-python(version range:>=20131210)
| +-libvmdk-python(version range:>=20140421)
| +-libvshadow-python(version range:>=20160109)
| +-libvslvm-python(version range:>=20160109)
| +-lz4(version range:>=0.10.0)
| +-pefile(version range:>=2018.8.8)
| | +-future(version range:)
| +-pip(version range:>=7.0.0)
| +-psutil(version range:>=5.4.3)
| +-pycrypto(version range:>=2.6)
| +-pyparsing(version range:>=2.3.0)
| +-python-dateutil(version range:>=1.5)
| +-pytsk3(version range:>=20160721)
| +-pytz(version range:)
| +-pyyaml(version range:>=3.10)
| +-pyzmq(version range:>=2.1.11)
| +-requests(version range:>=2.18.0)
| | +-chardet(version range:>=3.0.2,<3.1.0)
| | +-idna(version range:>=2.5,<2.9)
| | +-urllib3(version range:>=1.21.1,<1.26)
| | +-certifi(version range:>=2017.4.17)
| +-six(version range:>=1.1.0)
| +-urllib3(version range:>=1.21.1)
| +-xlsxwriter(version range:>=0.9.3)
| +-yara-python(version range:>=3.4.0)

Thanks for your attention.
Best,
Neolith

[NeolithEra]

Solution

1. Fix your direct dependencies to be chardet>=3.0.2,<3.1.0, idna>=2.5,<2.9, urllib3>=1.21.1,<1.26 and requests==2.22.0, to remove this conflict.

I have checked this revision will not affect your downstream projects now.

2. Remove your direct dependency **chardet, urllib3 and idna **, and use chardet transitively introduced by request.

@joachimmetz Please let me know your choice. I can submit a PR to solve this issue.
Build a good dependency ecosystem for python projects is our common goal ^_^.

[joachimmetz]

@NeolithEra what installation method are you referring to? It seems like requirements.txt

Which we explicitly discourage. Per: https://plaso.readthedocs.io/en/latest/sources/developer/Developing-Virtualenv.html

Note that this is intended for development use only, and if you aren’t comfortable debugging package installation, this is not for you.

Fix your direct dependencies to be chardet>=3.0.2,<3.1.0, idna>=2.5,<2.9, urllib3>=1.21.1,<1.26 and requests==2.22.0, to remove this conflict.

Pinning to exact versions is such a bad practice for many reasons.

Remove your direct dependency **chardet, urllib3 and idna **, and use chardet transitively introduced by request.

Remove from where? requirements.txt?

[NeolithEra]

@joachimmetz When users use intall -r requirement to debug or use plaso, the the multiple versions of these libraries will bring potential build failures.

I mean that maybe we can remove direct dependencies to be chardet>=3.0.2,<3.1.0, idna>=2.5,<2.9, urllib3>=1.21.1,<1.26 and requests==2.22.0 from requirement.txt.

[joachimmetz]

When users use intall -r requirement to debug or use plaso, the the multiple versions of these libraries will bring potential build failures.

As I said we don't recommend using this installation method in the first place, especially due to issues like these. I'll have a look, as you proposed, to remove direct dependency on chardet, urllib3 and idna, and use them transitively.

[joachimmetz]

Pending changes, closing.