Bootstrap script fails to build python netifaces

[nickoe]

I just tried to run the scripts/bootstrap.sh script on Archlinux and it failed at installing the netifaces python module, or rather it failed to build it! :S

It looks like this dependency was added as part of a renode dependency at some point 4ebc207

Installing netifaces (python module)
Collecting netifaces
  Downloading netifaces-0.10.9.tar.gz (28 kB)
Building wheels for collected packages: netifaces
  Building wheel for netifaces (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/nickoe/litex_test/litex-buildenv/build/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ssarfmf1
       cwd: /tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/
  Complete output (29 lines):
  running bdist_wheel
  running build
  running build_ext
  checking for getifaddrs...not found.
  checking for getnameinfo...not found.
  checking for socket IOCTLs...not found.
  checking for optional header files...netash/ash.h netatalk/at.h netax25/ax25.h neteconet/ec.h netipx/ipx.h netpacket/packet.h netrose/rose.h linux/atm.h linux/llc.h linux/tipc.h linux/dn.h.
  checking whether struct sockaddr has a length field...no.
  checking which sockaddr_xxx structs are defined...none!
  checking for routing socket support...no.
  checking for sysctl(CTL_NET...) support...no.
  checking for netlink support...no.
  building 'netifaces' extension
  gcc -pthread -B /home/nickoe/litex_test/litex-buildenv/build/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNETIFACES_VERSION=0.10.9 -DHAVE_NETASH_ASH_H=1 -DHAVE_NETATALK_AT_H=1 -DHAVE_NETAX25_AX25_H=1 -DHAVE_NETECONET_EC_H=1 -DHAVE_NETIPX_IPX_H=1 -DHAVE_NETPACKET_PACKET_H=1 -DHAVE_NETROSE_ROSE_H=1 -DHAVE_LINUX_ATM_H=1 -DHAVE_LINUX_LLC_H=1 -DHAVE_LINUX_TIPC_H=1 -DHAVE_LINUX_DN_H=1 -I/home/nickoe/litex_test/litex-buildenv/build/conda/include/python3.7m -c netifaces.c -o build/temp.linux-x86_64-3.7/netifaces.o
  netifaces.c:210:6: error: #error You need to add code for your platform.
    210 | #    error You need to add code for your platform.
        |      ^~~~~
  netifaces.c: In function ‘gateways’:
  netifaces.c:1466:22: warning: unused variable ‘defaults’ [-Wunused-variable]
   1466 |   PyObject *result, *defaults;
        |                      ^~~~~~~~
  At top level:
  netifaces.c:689:1: warning: ‘add_to_family’ defined but not used [-Wunused-function]
    689 | add_to_family (PyObject *result, int family, PyObject *obj)
        | ^~~~~~~~~~~~~
  netifaces.c:438:1: warning: ‘string_from_netmask’ defined but not used [-Wunused-function]
    438 | string_from_netmask (struct sockaddr *addr,
        | ^~~~~~~~~~~~~~~~~~~
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for netifaces
  Running setup.py clean for netifaces
Failed to build netifaces
Installing collected packages: netifaces
    Running setup.py install for netifaces ... error
    ERROR: Command errored out with exit status 1:
     command: /home/nickoe/litex_test/litex-buildenv/build/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-glludawt/install-record.txt --single-version-externally-managed --compile --install-headers /home/nickoe/litex_test/litex-buildenv/build/conda/include/python3.7m/netifaces
         cwd: /tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/
    Complete output (29 lines):
    running install
    running build
    running build_ext
    checking for getifaddrs...not found.
    checking for getnameinfo...not found.
    checking for socket IOCTLs...not found.
    checking for optional header files...netash/ash.h netatalk/at.h netax25/ax25.h neteconet/ec.h netipx/ipx.h netpacket/packet.h netrose/rose.h linux/atm.h linux/llc.h linux/tipc.h linux/dn.h.
    checking whether struct sockaddr has a length field...no.
    checking which sockaddr_xxx structs are defined...none!
    checking for routing socket support...no.
    checking for sysctl(CTL_NET...) support...no.
    checking for netlink support...no.
    building 'netifaces' extension
    gcc -pthread -B /home/nickoe/litex_test/litex-buildenv/build/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNETIFACES_VERSION=0.10.9 -DHAVE_NETASH_ASH_H=1 -DHAVE_NETATALK_AT_H=1 -DHAVE_NETAX25_AX25_H=1 -DHAVE_NETECONET_EC_H=1 -DHAVE_NETIPX_IPX_H=1 -DHAVE_NETPACKET_PACKET_H=1 -DHAVE_NETROSE_ROSE_H=1 -DHAVE_LINUX_ATM_H=1 -DHAVE_LINUX_LLC_H=1 -DHAVE_LINUX_TIPC_H=1 -DHAVE_LINUX_DN_H=1 -I/home/nickoe/litex_test/litex-buildenv/build/conda/include/python3.7m -c netifaces.c -o build/temp.linux-x86_64-3.7/netifaces.o
    netifaces.c:210:6: error: #error You need to add code for your platform.
      210 | #    error You need to add code for your platform.
          |      ^~~~~
    netifaces.c: In function ‘gateways’:
    netifaces.c:1466:22: warning: unused variable ‘defaults’ [-Wunused-variable]
     1466 |   PyObject *result, *defaults;
          |                      ^~~~~~~~
    At top level:
    netifaces.c:689:1: warning: ‘add_to_family’ defined but not used [-Wunused-function]
      689 | add_to_family (PyObject *result, int family, PyObject *obj)
          | ^~~~~~~~~~~~~
    netifaces.c:438:1: warning: ‘string_from_netmask’ defined but not used [-Wunused-function]
      438 | string_from_netmask (struct sockaddr *addr,
          | ^~~~~~~~~~~~~~~~~~~
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/nickoe/litex_test/litex-buildenv/build/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_irg_6x3/netifaces_becf8350b74f4a6191e1434ac412afbc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-glludawt/install-record.txt --single-version-externally-managed --compile --install-headers /home/nickoe/litex_test/litex-buildenv/build/conda/include/python3.7m/netifaces Check the logs for full command output.

[mithro]

@mateusz-holenko Could you take a quick look at this? I think as netifaces requires a C extension might need to be provided by a conda package rather than installed via pip?

Looks like there are multiple conda packages already build, so we might just need to add something to the litex-hub conda repos?

• https://github.com/anguslees/conda-netifaces
• https://anaconda.org/conda-forge/netifaces
• https://github.com/conda-forge/netifaces-feedstock

GitHub

anguslees/conda-netifaces

Conda packaging for netifaces. Contribute to anguslees/conda-netifaces development by creating an account on GitHub.

Netifaces :: Anaconda Cloud

GitHub

conda-forge/netifaces-feedstock

A conda-smithy repository for netifaces. Contribute to conda-forge/netifaces-feedstock development by creating an account on GitHub.

[mateusz-holenko]

@nickoe netifaces are required only for some selected tests in Renode and I don't see that we are running them in litex-buildenv. Perhaps we have just too much dependencies listed.

Could you try to comment out the following fragments and see if it works for you?

• litex-buildenv/scripts/download-env.sh

  Lines 646 to 649 in 690901d

  	echo
  	echo "Installing netifaces (python module)"
  	pip install netifaces
  	check_import netifaces

• litex-buildenv/scripts/enter-env.sh

  Line 550 in 690901d

  check_import netifaces || return 1

[nickoe]

@mateusz-holenko mm, that is not really enough, I did cd into the new clone of litex-buildenv in the failed build. Then removed the lines you requested:

$ git diff -w
diff --git a/scripts/download-env.sh b/scripts/download-env.sh
index bb77de0..1e1708f 100755
--- a/scripts/download-env.sh
+++ b/scripts/download-env.sh
@@ -643,11 +643,6 @@ echo "Installing requests (python module)"
 pip install requests
 check_import requests
 
-echo
-echo "Installing netifaces (python module)"
-pip install netifaces
-check_import netifaces
-
 
 echo
 echo "Installing robotframework (python module)"
diff --git a/scripts/enter-env.sh b/scripts/enter-env.sh
index 17d41c9..3b5cd36 100755
--- a/scripts/enter-env.sh
+++ b/scripts/enter-env.sh
@@ -547,8 +547,6 @@ check_import yaml || return 1
 
 check_import requests || return 1
 
-check_import netifaces || return 1
-
 check_import robot || return 1
 
 check_import pythondata_software_compiler_rt || return 1

And then ran bash scripts/bootstrap.sh instead of the curl of it, but that seems to get a fresh copy and starting to use that. Running scripts/download-env.sh directly after applying the above diff do complete just fine.

[nickoe]

@mateusz-holenko Any tips on how this should properly be fixed?

[mateusz-holenko]

@nickoe Do I understand correctly that the above change solved the problem with netifaces and allowed to further use litex-buildenv successfully (apart from the bootstrap phase)?

As you said, scripts/bootstrap.sh clones a fresh copy of the repository, so local changes to scripts won't have any effect.
I guess that the proper solution is to merge the fix into master. Could you prepare a PR?