[Python] Difficulty running tests after conda install

[asfimport]

I install PyArrow from conda-forge, and then try running tests (or import generally)

conda create -n test-arrow pytest ipython pandas nomkl pyarrow -c conda-forge 

mrocklin@carbon:~/workspace/arrow/python$ py.test pyarrow/tests/test_parquet.py 
Traceback (most recent call last):
File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/_pytest/config.py", line 328, in _getconftestmodules
return self._path2confmods[path]
KeyError: local('/home/mrocklin/workspace/arrow/python/pyarrow/tests/test_parquet.py')During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/_pytest/config.py", line 328, in _getconftestmodules
return self._path2confmods[path]
KeyError: local('/home/mrocklin/workspace/arrow/python/pyarrow/tests')During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/_pytest/config.py", line 359, in _importconftest
return self._conftestpath2mod[conftestpath]
KeyError: local('/home/mrocklin/workspace/arrow/python/pyarrow/tests/conftest.py')During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/_pytest/config.py", line 365, in _importconftest
mod = conftestpath.pyimport()
File "/home/mrocklin/Software/anaconda/lib/python3.6/site-packages/py/_path/local.py", line 668, in pyimport
__import__(modname)
File "/home/mrocklin/workspace/arrow/python/pyarrow/__init__.py", line 54, in <module>
from pyarrow.lib import cpu_count, set_cpu_count
ModuleNotFoundError: No module named 'pyarrow.lib'
ERROR: could not load /home/mrocklin/workspace/arrow/python/pyarrow/tests/conftest.py

Probably this is something wrong with my environment, but I thought I'd report it as a usability bug

Environment: conda create -n test-arrow pytest ipython pandas nomkl pyarrow -c conda-forge

Ubuntu 16.04
Reporter: Matthew Rocklin / @mrocklin

_{Note: This issue was originally created as ARROW-3280. Please see the migration documentation for further details.}

[asfimport]

Matthew Rocklin / @mrocklin:
Actually, let me back up.  I wrote a test and wanted to run it.  I tried running py.test from that directory and got the error that I list above.  That's probably reasonable, given that Python is probably confused about paths given that I'm in a directory named arrow.  

I googled online for arrow developer notes, and eventually found that I was supposed to look in the directory for a README file.  That file didn't have anything about testing in it explicitly.  

I see now that it has a "Build from source" section that links to external docs.  I'll go and try that and see what happens.

[asfimport]

Matthew Rocklin / @mrocklin:
I got as far as the following.  My apologies for the rambling issue.  Feel free to close.  I'm also happy to reopen something else a bit cleaner.

mrocklin@carbon:~/workspace/arrow/python$ python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
> --with-parquet --with-plasma --inplace
your setuptools is too old (<12)
setuptools_scm functionality is degraded
Traceback (most recent call last):
File "setup.py", line 589, in <module>
url="https://arrow.apache.org/"
File "/home/mrocklin/Software/anaconda/envs/pyarrow-dev/lib/python3.6/site-packages/setuptools/__init__.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/home/mrocklin/Software/anaconda/envs/pyarrow-dev/lib/python3.6/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/home/mrocklin/Software/anaconda/envs/pyarrow-dev/lib/python3.6/site-packages/setuptools/dist.py", line 370, in __init__
k: v for k, v in attrs.items()
File "/home/mrocklin/Software/anaconda/envs/pyarrow-dev/lib/python3.6/distutils/dist.py", line 281, in __init__
self.finalize_options()
File "/home/mrocklin/Software/anaconda/envs/pyarrow-dev/lib/python3.6/site-packages/setuptools/dist.py", line 529, in finalize_options
ep.load()(self, ep.name, value)
File "/home/mrocklin/workspace/arrow/python/.eggs/setuptools_scm-1.15.1rc1-py3.6.egg/setuptools_scm/integration.py", line 19, in version_keyword
File "/home/mrocklin/workspace/arrow/python/.eggs/setuptools_scm-1.15.1rc1-py3.6.egg/setuptools_scm/__init__.py", line 117, in get_version
File "/home/mrocklin/workspace/arrow/python/.eggs/setuptools_scm-1.15.1rc1-py3.6.egg/setuptools_scm/__init__.py", line 69, in _do_parse
File "setup.py", line 519, in parse_git
return parse(root, **kwargs)
File "/home/mrocklin/workspace/arrow/python/.eggs/setuptools_scm-1.15.1rc1-py3.6.egg/setuptools_scm/git.py", line 99, in parse
ValueError: invalid literal for int() with base 10: 'ge2c4b09d'

[asfimport]

Wes McKinney / @wesm:
This appears to be a setuptools_scm issue – what version of setuptools are you using?

[asfimport]

Matthew Rocklin / @mrocklin:
Yeah, given the error I thought that as well, but updating didn't seem to do much good.

mrocklin@carbon:~/workspace/arrow/python$ python -c "import setuptools; print(setuptools.__version__)"
40.4.1

[asfimport]

Uwe Korn / @xhochy:
@mrocklin Please delete your /home/mrocklin/workspace/arrow/python/.eggs folder. It contains an old setuptools_scm version that probably messes things up here; and then install setuptools_scm using pip/conda into the environment.