Merge pull request #355 from yarikoptic/enh-test-contributing

DOC+DOCTEST:  more exhaustive list of dependencees

CONTRIBUTING.md

@@ -119,6 +119,11 @@ we outline the workflow used by the developers:
 (If any of the above seems like magic to you, then look up the
 [Git documentation](http://git-scm.com/documentation) on the web.)
 
+Development environment
+-----------------------
+
+See [README.md:Dependencies](README.md#Dependencies).
+
 Additional Hints
 ----------------
 
@@ -182,6 +187,9 @@ deactivate
 Alternatively, or complimentary to that, you can use `tox` -- there is a `tox.ini`
 file which sets up a few virtual environments for testing locally, which you can
 later reuse like any other regular virtualenv for troubleshooting.
+Additionally, [tools/testing/test_README_in_docker](tools/testing/test_README_in_docker) script can
+be used to establish a clean docker environment (based on any NeuroDebian-supported
+release of Debian or Ubuntu) with all dependencies listed in README.md pre-installed.
 
 
 ### Coverage
@@ -236,3 +244,16 @@ tracker.  Resolving these issues allows you to start contributing to the project
 without much prior knowledge.  Your assistance in this area will be greatly
 appreciated by the more experienced developers as it helps free up their time to
 concentrate on other issues.
+
+Various hints for developers
+----------------------------
+
+### Useful tools
+
+- while performing IO/net heavy operations use [dstat](http://dag.wieers.com/home-made/dstat)
+  for quick logging of various health stats in a separate terminal window:
+
+        dstat -c --top-cpu -d --top-bio --top-latency --net
+
+- to monitor speed of any data pipelining [pv](http://www.ivarch.com/programs/pv.shtml) is really handy,
+  just plug it in the middle of your pipe

README.md

@@ -32,16 +32,30 @@ internals and/or contributing to the project.
 # Dependencies
 
 Although we now support Python 3 (>= 3.3), primarily we still use Python 2.7
-and thus instructions below are for python 2.7 deployments. 
-On Debian-based systems we recommend to enable
-[NeuroDebian](http://neuro.debian.net) since we use it to provide
-backports of recent fixed external modules we depend upon:
+and thus instructions below are for python 2.7 deployments.  Replace `python-{` 
+with `python{,3}-{` to also install dependencies for Python 3 (e.g., if you would
+like to develop and test through tox).
+
+On Debian-based systems we recommend to enable [NeuroDebian](http://neuro.debian.net)
+since we use it to provide backports of recent fixed external modules we depend upon:
+
+```sh
+apt-get install -y -q git git-annex-standalone
+apt-get install -y -q patool python-scrapy python-{appdirs,argcomplete,git,humanize,keyring,lxml,msgpack,mock,progressbar,rdflib,setuptools,six,sparqlwrapper}
+```
+
+or additionally, if you would like to develop and run our tests battery as
+described in [CONTRIBUTING.md](CONTRIBUTING.md) and possibly use tox and new
+versions of dependencies from pypy:
 
 ```sh
-apt-get install patool python-bs4 python-git python-testtools python-mock python-nose git-annex-standalone
+apt-get install -y -q python-{dev,httpretty,testtools,nose,pip,vcr,virtualenv} python-tox
+# Some libraries which might be needed for installing via pip
+apt-get install -y -q lib{ffi,ssl,curl4-openssl,xml2,xslt1}-dev
 ```
 
-or otherwise you can use pip to install Python modules
+or use pip to install Python modules (prior installation of those libraries listed above
+might be necessary)
 
 ```sh
 pip install -r requirements.txt

datalad/downloaders/tests/test_http.py

@@ -25,13 +25,15 @@
 from ...tests.utils import with_fake_cookies_db
 
 # BTW -- mock_open is not in mock on wheezy (Debian 7.x)
-if PY3:
+try:
+    if PY3:
+        raise ImportError("Not yet ready apparently: https://travis-ci.org/datalad/datalad/jobs/111659666")
+    import httpretty
+except ImportError:
     class NoHTTPPretty(object):
        __bool__ = __nonzero__ = lambda s: False
        activate = lambda s, t: t
     httpretty = NoHTTPPretty()
-else:
-    import httpretty
 
 from mock import patch
 from ...tests.utils import assert_in

datalad/support/network.py

@@ -24,8 +24,6 @@
 # TODO not sure what needs to use `six` here yet
 import requests
 
-from bs4 import BeautifulSoup
-
 import logging
 lgr = logging.getLogger('datalad.network')
 

datalad/tests/test_tests_utils.py

@@ -13,7 +13,12 @@
 import random
 import traceback
 
-from bs4 import BeautifulSoup
+try:
+    # optional direct dependency we might want to kick out
+    import bs4
+except ImportError:
+    bs4 = None
+
 from glob import glob
 from os.path import exists, join as opj, basename
 
@@ -358,7 +363,7 @@ def test_path_and_url(path, url):
         u = urlopen(url)
         assert_true(u.getcode() == 200)
         html = u.read()
-        soup = BeautifulSoup(html, "html.parser")
+        soup = bs4.BeautifulSoup(html, "html.parser")
         href_links = [txt.get('href') for txt in soup.find_all('a')]
         assert_true(len(href_links) == 1)
 
@@ -367,6 +372,8 @@ def test_path_and_url(path, url):
         html = u.read().decode()
         assert(test_txt == html)
 
+    if bs4 is None:
+        raise SkipTest("bs4 is absent")
     test_path_and_url()
 
 

requirements.txt

@@ -35,10 +35,7 @@ beautifulsoup4
 nose>=1.3.4
 testtools
 vcrpy
-#httpretty >= 0.8.13 may be even more since python 3.4,5 is not supported
-# https://github.com/gabrielfalcao/HTTPretty/archive/master.zip
-# until https://github.com/gabrielfalcao/HTTPretty/pull/279 gets a resolution
-https://github.com/yarikoptic/HTTPretty/archive/bf-3.x.zip
+httpretty>=0.8.14
 msgpack-python
 
 # -- Optional for Testing

tools/testing/.gitignore

@@ -0,0 +1 @@
+test_README_in_docker-Dockerfile

tools/testing/conf/etc/apt/apt.conf.d/99apt-cacher

@@ -0,0 +1,2 @@
+#Acquire::http { Proxy "http://172.17.42.1:3142"; };
+#Acquire::http { Proxy "http://smaug.datalad.org:3142"; };

tools/testing/test_README_in_docker

@@ -0,0 +1,60 @@
+#!/bin/bash
+#emacs: -*- mode: shell-script; c-basic-offset: 4; tab-width: 4; indent-tabs-mode: t -*- 
+#ex: set sts=4 ts=4 sw=4 noet:
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+#
+#   See COPYING file distributed along with the datalad package for the
+#   copyright and license terms.
+#
+# ## ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ##
+#
+# Helper to generate a Docker instance mapping user uder docker into your USER/UID/GID
+# and allowing to run tox within that clean automatically generated according to
+# README.md's apt-get lines environment
+#
+set -e
+#set -x
+set -u
+
+DL_DIST=$1
+
+topdir=$(realpath `dirname $0`)
+dockerfile=$topdir/test_README_in_docker-Dockerfile
+# echo "D: $DL_APT"
+sed -e "s,DL_DIST,$DL_DIST,g" \
+    -e "s,DL_USER,$USER,g" \
+    -e "s,DL_UID,`id -u`,g" \
+    -e "s,DL_GID,`id -g`,g" \
+    -e "s,DL_GIT_USER_EMAIL,`git config --get user.email`,g" \
+    -e "s,DL_GIT_USER_NAME,`git config --get user.name`,g" \
+    $dockerfile.in >| $dockerfile
+
+#DL_APT=$(grep '^\(apt-get\|pip\)' ./../../README.md)
+
+grep '^apt-get ' ./../../README.md | sed -e 's|python-{|python{,3}-{|g' \
+| while read aptline; do
+    sed -i -e "s|\(\(.*\)DL_APT\(.*\)\)|\2$aptline\3\n\1|g" $dockerfile
+    :
+done
+sed -e '/DL_APT/d' -i $dockerfile
+
+tag=datalad:test_README_${USER}_$DL_DIST
+echo "I: tag $tag"
+if docker images | grep -q datalad.*test_README.*$DL_DIST; then
+    echo "I: tag already exists -- skipping rebuilding"
+else
+    docker build -t $tag -f $dockerfile . #&& rm Dockerfile
+    #docker build --no-cache=True -t $tag -f $dockerfile . #&& rm Dockerfile
+fi
+
+topgitdir=`realpath ${topdir}/../..`
+echo "I: top git dir $topgitdir"
+
+tox="$topgitdir/.tox"
+if [ -e "$tox" ]; then
+    echo "I: removing existing tox under $tox"
+    rm -r $tox
+fi
+
+echo "I: running tox within docker"
+docker run -it --rm=true -v $topgitdir:/home/$USER/datalad $tag tox --sitepackages

tools/testing/test_README_in_docker-Dockerfile.in

@@ -0,0 +1,41 @@
+FROM neurodebian:DL_DIST
+MAINTAINER Yaroslav Halchenko <yoh@datalad.org>
+
+USER root
+
+# Speed up installation using our apt cacher
+#RUN mkdir /etc/apt/apt.conf.d/
+COPY conf/etc/apt/apt.conf.d/99apt-cacher /etc/apt/apt.conf.d/99apt-cacher
+RUN chmod a+r /etc/apt/apt.conf.d/99apt-cacher
+
+# Make deb-src avail
+RUN sed -i -e 's,^deb\(.*\),deb\1\ndeb-src\1,g' /etc/apt/sources.list.d/neurodebian.sources.list /etc/apt/sources.list
+
+# Assure popcon doesn't kick in
+RUN bash -c "echo 'debconf debconf/frontend select noninteractive' | debconf-set-selections -"
+
+RUN apt-get update
+# Use bash for extended syntax
+RUN bash -c "DL_APT"
+# Some rudimentary tools if we need to do anything within docker
+RUN bash -c "apt-get install -y -q vim less"
+RUN apt-get clean
+
+# Let's setup user matching user
+RUN groupadd --gid DL_GID -r DL_USER && useradd -m --uid DL_UID -g DL_USER DL_USER
+
+USER DL_USER
+
+WORKDIR /home/DL_USER
+# Prepare system for working with Git
+RUN git config --global user.email "DL_GIT_USER_EMAIL"
+RUN git config --global user.name "DL_GIT_USER_NAME"
+# RUN git clone git://github.com/datalad/datalad
+RUN mkdir datalad
+
+WORKDIR /home/DL_USER/datalad
+
+# run outside so we could re-enter etc
+# RUN tox
+# Fall back to root, so we have full control happen we want to reuse it?
+# USER root