ARM architecture support

[nik123]

Recently I have tried to install DVC on Nvidia Jetson TX2 without any success (I have tried pip and deb packages). It seems that amd64 is the only supported architecture by now. Are there any plans for ARM-based devices support (Nvidia Jetson, Raspberry Pi, etc.)?

Requirements:

1. Hardware: Jetson TX2.
2. OS: Ubuntu 18.04.3.
3. Python version: 3.6.9 (inside virtual environment).

Steps to reproduce:

pip install dvc[all]

Result:

Installation fails with error message:

  CMake Error at cmake_modules/FindNumPy.cmake:62 (message):
    NumPy import failure:
  
    Traceback (most recent call last):
  
      File "<string>", line 1, in <module>
  
    ModuleNotFoundError: No module named 'numpy'
  
  Call Stack (most recent call first):
    CMakeLists.txt:189 (find_package)
  
  
  -- Configuring incomplete, errors occurred!
  See also "/tmp/pip-install-xrq0e8dy/pyarrow/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeOutput.log".
  See also "/tmp/pip-install-xrq0e8dy/pyarrow/build/temp.linux-aarch64-3.6/CMakeFiles/CMakeError.log".
  error: command 'cmake' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly

P.S.:
You can find full log for pip install dvc[all] here: https://gist.github.com/nik123/f12c7e26c1005d12bbccff3e97ae586c

@nik123 , have you encountered any specific error?

[efiop]

@nik123 To be honest, we haven't ever looked into ARM support 🙂 Indeed, deb package is a binary package, which is only meant for x86_64. Have you tried installing from conda or pip?

@efiop , from his comment:

I have tried pip and deb packages

[efiop]

@MrOutis Oops, sorry, long day 🙂 Yeah, would love to see the pip error then. I imagine the issue there might be that some of our dependencies don't have binary wheels for arm and so pip tries to compile and fails.

[nik123]

@efiop I have tried to install via pip too. Right now I don't have access to the arm device. But as soon as I will have access to it I will try to install dvc again and post result here.

[nik123]

@MrOutis , @efiop : I have updated description with error message and also attached full log for pip install dvc[all]

P.S.:
It seems if I install dvc itself (pip install dvc) it works fine. However if I try pip install dvc[all] it fails with error.

[efiop]

@nik123 Ok, looks like pyarrow is the cause there. Do you need support for hdfs? If not, just use whichever options you need instead of all, except for hdfs. E.g. pip install '[gs,s3,azure,oss,ssh,gdrive]' to install everything except for hdfs support. Please try it out and let us know how if it works or not. 🙂 Honestly looks promising so far 🙂

[nik123]

@efiop pip install 'dvc[gs,s3,azure,oss,ssh,gdrive]' ended with success.

[efiop]

@nik123 That is great! 🙂 Does dvc also work as expected after installation?

Btw, could you show the error you were getting when installing deb package? It seems to have the platform info in it, so i expect it to handle incorrect destination platform gracefully.

I suppose pip package suits you in that form? Or do you need deb package?

In terms of action points here, so far we may be able to put some env marker check (e.g. https://github.com/vlachoudis/bCNC/blob/master/setup.py#L26 or maybe just if in setup.py would work, unsure) to not include pyarrow into all target on ARM. Though, I can't really guarantee that some another dependency won't break, as we admittedly don't support ARM "officially".

This is the first time someone reached out about using dvc on ARM, so I am wondering how many ARM users we have and if it would be reasonable to add an arm machine on travis to check that...

[Suor]

Setting aside the question whether we need to support ARM, it's a pyarrow issue, i.e. pip install pyarrow will fail as well. It makes sense opening issue there too.