bumpy missing as pip dependency

[LorenzMeier]

Startup will fail if numpy not installed on the system

[stephendade]

Is this when using MAVProxy direct from source, or via pip or the windows setup exe?

[monkeypants]

Is there a reason not to use requirements.txt (as in pip -r requirements)?

[stephendade]

A requirements.txt would be a good idea!

[monkeypants]

setup.py contains this comment:

  # note that we do not include all the real dependencies here (like matplotlib etc)          
  # as that breaks the pip install. It seems that pip is not smart enough to                  
  # use the system versions of these dependencies, so it tries to download and install        
  # large numbers of modules like numpy etc which may be already installed                    

Then it goes on to specify:

  install_requires=['pymavlink>=1.1.50',
                    'pyserial'],

Which, as Lorenz points out, does not include numpy (or, as the comment says, matplotlib "etc"). I'm not sure how to test that a requirements file does not pull down files it shouldn't (like the comment complains about), but the following goes some way towards testing that pip does install what it should.

1. check you have python "virtualenv" and "pip" installed, with virtualenv --help and pip --help. If you don't, install them.
2. cd into the base of your repository, then create a new virtualenv (without any site-packages, with virtualenv .venv (or whatever you want to call it). Might want to add .venv to your gitignore too, wouldn't want to accidentally check a whole python environment in!
3. activate the virtualenv with source .venv/bin/activate. Now, when you call python, you will be using this local 'empty' virtualenv. You can switch back to your machine's normal site python at any time with the deactivate command. (You can also completely remove the virtualenv from your system by deleting the .venv folder, or whatever you called it).
4. type pip install -r requirements.txt and wait for it to finish. This will pull everything in requirements.txt into the clean environment, which will need them because (by default) virtualenv created an interpreter with none of the local site-packages installed.
5. try to setup MavProxy in your virtualenv (e.g. python setup.py build; python setup.py install), then try to use it. When this doesn't work due to missing dependencies, add more lines to requirements.txt specifying the packages you need, then run pip -r requirements again. It should be faster next time, will only download the things it doesn't already have.

This approach should result in a working virtualenv assuming two things. One; that requirements.txt explicitly includes all the required dependencies, and two; that all the non-python dependencies (libraries etc) are met. If it makes a working virtualenv, it should of course also make a working python site package if you run pip install -r requirements.txt without an active virtualenv.

I made a branch called "requirements-txt-issue159" in my fork at https://github.com/monkeypants/MavProxy that contains the requirements.txt and .gitignore changes. I added numpy and matplotlib, but of course there will be more requirements that are not met yet. @stephendade, can you please do a pip freeze on your system to list your packages, I'm on a new machine and haven't got a working setup at the moment. If you give me your freeze list, I'll pick through and figure out what I need to get everything working.

BTW: Starting on a fresh ubuntu 14.04 machine, I had to install sudo apt-get install libpng12-dev libfreetype6-dev to be able to install matplotlib via pip.

[stephendade]

So running pip freeze on my laptop gives:

Chameleon==2.6.1
Jinja2==2.7.2
M2Crypto==0.21.1
MAVProxy==1.4.19
Mako==0.9.1
MarkupSafe==0.18
PAM==0.4.2
Pillow==2.3.0
PyYAML==3.10
Pygments==1.6
SOAPpy==0.12.0
SecretStorage==2.0.0
Twisted-Core==13.2.0
Twisted-Names==13.2.0
Twisted-Web==13.2.0
VTK==5.8.0
adium-theme-ubuntu==0.3.4
ansible==1.5.4
apt-xapian-index==0.45
argparse==1.2.1
backports.ssl-match-hostname==3.4.0.2
bzr==2.7.0dev1
bzr-builddeb==2.8.6
catkin-pkg==0.2.7
ccsm==0.9.11.3
certifi==14.05.14
chardet==2.0.1
colorama==0.2.5
command-not-found==0.3
compizconfig-python==0.9.11.3
configglue==1.1.2
configobj==4.7.2
debtagshw==0.1
decorator==3.4.0
defer==1.0.6
dirspec==13.10
distro-info==0.12
docutils==0.12
droneapi==1.1.1
duplicity==0.6.23
empy==3.1
fpconst==0.7.2
gbp==0.6.9
html5lib==0.999
httplib2==0.8
intelhex==1.5
keyring==3.5
launchpadlib==1.10.2
lazr.restfulclient==0.13.3
lazr.uri==1.0.3
lockfile==0.8
lptools==0.2.0
lxml==3.3.3
matplotlib==1.3.1
numpy==1.8.2
oauth==1.0.1
oauthlib==0.6.1
oneconf==0.3.7
osc==0.142git
paramiko==1.10.1
pexpect==3.1
piston-mini-client==0.7.5
protobuf==2.6.1
pyOpenSSL==0.13
pycrypto==2.6.1
pycups==1.9.66
pycurl==7.19.3
pydns==2.3.6
pygame==1.9.1release
pyglet==1.1.4
pygobject==3.12.0
pygpgme==0.3
pyinotify==0.9.4
pylibftdi==0.14.2
pymavlink==1.1.42
pyparsing==1.5.7
pyserial==2.7
pysmbc==1.0.14.1
python-apt==0.9.3.5ubuntu1
python-daemon==1.5.5
python-dateutil==2.4.2
python-debian==0.1.21-nmu2ubuntu2
python-debianbts==1.11
python-espeak==0.5
pytz==2012c
pyxdg==0.25
reportbug==6.5.0
reportlab==3.0
requests==2.2.1
scipy==0.13.3
sessioninstaller==0.0.0
simplejson==3.3.1
six==1.5.2
software-center-aptd-plugins==0.0.0
ssh-import-id==3.21
system-service==0.1.6
tornado==4.1
traits==4.1.0
ubuntu-dev-tools==0.153
unity-lens-photos==1.0
urlgrabber==3.9.1
urllib3==1.7.1
virtualenv==1.11.4
wadllib==1.3.2
wheel==0.24.0
wsgiref==0.1.2
wxPython==2.8.12.1
wxPython-common==2.8.12.1
xdiagnose==3.6.3build2
zope.interface==4.0.5


[mrpollo]

pip freeze would only yield relevant results when run inside a virtualenv, I created #160 to move the discussion of requirements.txt and virtualenv support there

[monkeypants]

Running it in my virtualenv would only show matplotlib, numpy and their dependencies. I asked for Steve's freeze basically as a bunch of hints to make it easier for me to build up my requirements.txt from scratch.

[monkeypants]

How are you installing opencv? I think I used the Debian/Ubuntu packages previously, but if we are going to bootstrap a clean environment it might be better to build it out?

[stephendade]

In terms of OpenCV, according to http://tridge.github.io/MAVProxy/, use an apt-get install python-opencv

[monkeypants]

oh right, I forgot about that page. Might be good if the README linked to it :)

[stephendade]

It will be, once #151 is merged.