ImportError: cannot import name 'metadata' with v0.6.1 on python 3.6

[XN137]

we have automated tests that install tcconfig==0.24.1 on python 3.6 and they started failing 1-2 days ago with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyroute2/__init__.py", line 44, in <module>
    from importlib import metadata
ImportError: cannot import name 'metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcdel", line 5, in <module>
    from tcconfig.tcdel import main
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/tcdel.py", line 21, in <module>
    from ._error import NetworkInterfaceNotFoundError
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/_error.py", line 11, in <module>
    from pyroute2 import IPRoute
  File "/usr/local/lib/python3.6/dist-packages/pyroute2/__init__.py", line 46, in <module>
    import importlib_metadata as metadata
ModuleNotFoundError: No module named 'importlib_metadata'

pinning pyroute2 to 0.5.19 fixes the issue, see reproduction below:

myhost:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.5 LTS
Release:	18.04
Codename:	bionic

myhost:~$ python3 -V
Python 3.6.9

myhost:~$ sudo pip3 -V
pip 20.3.3 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)

myhost:~$ sudo pip3 list | grep pyroute
WARNING: The directory '/home/myuser/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: You are using pip version 20.3.3; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

myhost:~$ sudo pip3 install tcconfig==0.24.1 >/dev/null 2>&1; echo $?
0

myhost:~$ sudo pip3 list | grep pyroute
WARNING: The directory '/home/myuser/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
pyroute2            0.6.1
pyroute2.core       0.6.1
pyroute2.ethtool    0.6.1
pyroute2.ipdb       0.6.1
pyroute2.ipset      0.6.1
pyroute2.ndb        0.6.1
pyroute2.nftables   0.6.1
pyroute2.nslink     0.6.1
WARNING: You are using pip version 20.3.3; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

myhost:~$ tcdel
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pyroute2/__init__.py", line 44, in <module>
    from importlib import metadata
ImportError: cannot import name 'metadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tcdel", line 5, in <module>
    from tcconfig.tcdel import main
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/tcdel.py", line 21, in <module>
    from ._error import NetworkInterfaceNotFoundError
  File "/usr/local/lib/python3.6/dist-packages/tcconfig/_error.py", line 11, in <module>
    from pyroute2 import IPRoute
  File "/usr/local/lib/python3.6/dist-packages/pyroute2/__init__.py", line 46, in <module>
    import importlib_metadata as metadata
ModuleNotFoundError: No module named 'importlib_metadata'

myhost:~$ sudo pip3 install pyroute2==0.5.19 >/dev/null 2>&1; echo $?
0

myhost:~$ tcdel
usage: tcdel [-h] [-V] [--tc-command | --tc-script] [--debug | --quiet]
             [--debug-query] [--stacktrace] [-a] [--id FILTER_ID]
             [--direction {outgoing,incoming}] [--network DST_NETWORK]
             [--src-network SRC_NETWORK] [--port DST_PORT]
             [--src-port SRC_PORT] [--ipv6] [--docker]
             [--src-container SRC_CONTAINER] [--dst-container DST_CONTAINER]
             device
tcdel: error: the following arguments are required: device

looking at
https://docs.python.org/3.7/library/importlib.html
https://docs.python.org/3.8/library/importlib.metadata.html

it seems like the import and fallback here can only work on python 3.8+:

pyroute2/pyroute2/pyroute2/__init__.py

Lines 43 to 46 in 59d6546

	try:
	from importlib import metadata
	except ImportError:
	import importlib_metadata as metadata

somewhat related (i.e. a potential fix):
python-poetry/poetry#1586

tox.ini on the project suggests py36 is being tested as part of CI, so not sure how this did not fail early.
maybe one of the test dependencies happens to also install importlib-metadata along the way already.

[svinota]

Strange enough it works for me in local tox as well as on travis: https://travis-ci.org/github/svinota/pyroute2/builds/771564718

maybe one of the test dependencies happens to also install importlib-metadata along the way already.

yep, that may be the case.

Let me dig a bit around.

[adamgreenhalgh]

A pip3 install of pyroute2 picking up 0.6.1 results in the following output for a pip3 freeze

pyroute2===file-.VERSION
pyroute2.core===file-.VERSION
pyroute2.ethtool===file-.VERSION
pyroute2.ipdb===file-.VERSION
pyroute2.ipset===file-.VERSION
pyroute2.ndb===file-.VERSION
pyroute2.nftables===file-.VERSION
pyroute2.nslink===file-.VERSION

And on this we see the importlib-metadata issue above.

[svinota]

@XN137 yep, looks like importlib-metadata is installed as a dependency for the project test environment.

I include it into the deps for python < 3.8

[svinota]

0.6.2 is on it's way

[XN137]

looks like its working:

~$ python3 -V
Python 3.6.9

~$ sudo pip3 list 2>&1 | egrep 'pyroute2|importlib'

~$ sudo pip3 install tcconfig==0.24.1 >/dev/null 2>&1; echo $?
0

~$ sudo pip3 list 2>&1 | egrep 'pyroute2|importlib'
importlib-metadata  4.0.1
pyroute2            0.6.2
pyroute2.core       0.6.2
pyroute2.ethtool    0.6.2
pyroute2.ipdb       0.6.2
pyroute2.ipset      0.6.2
pyroute2.ndb        0.6.2
pyroute2.nftables   0.6.2
pyroute2.nslink     0.6.2

~$ tcdel
usage: tcdel [-h] [-V] [--tc-command | --tc-script] [--debug | --quiet]
             [--debug-query] [--stacktrace] [-a] [--id FILTER_ID]
             [--direction {outgoing,incoming}] [--network DST_NETWORK]
             [--src-network SRC_NETWORK] [--port DST_PORT]
             [--src-port SRC_PORT] [--ipv6] [--docker]
             [--src-container SRC_CONTAINER] [--dst-container DST_CONTAINER]
             device
tcdel: error: the following arguments are required: device

thanks for the quick fix!

issue should be good to close now, wdyt?

[svinota]

Yep. Thanks!