Fix/install requirements #1

isislovecruft · 2013-04-19T19:11:43Z

These commits fix a broken requirements.txt file (it produced the following reproduceable error):

(oonib)~/code/ooni-backend $ pip install -r requirements.txt   
Exception:
Traceback (most recent call last):
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", line 240, in run
    for req in parse_requirements(filename, finder=finder, options=options):
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1356, in parse_requirements
    req = InstallRequirement.from_line(line, comes_from)
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 118, in from_line
    return cls(req, comes_from, url=url)
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 43, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/home/isis/.virtualenvs/oonib/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2515, in parse
    raise ValueError("No requirements found", s)
ValueError: ('No requirements found', '')

Storing complete log in /home/isis/.pip/pip.log

The above error was due to requirements.txt containing the lines:

# Mirrored on Tor's webserver:
https://people.torproject.org/~ioerror/src/mirrors/ooniprobe/scapy-02-25-2013-tip.zip#egg=

The #egg= at the end should be #egg=scapy instead.

I have also updated README.md with a set of commands for getting ooni-backend installed on a system, including updating pip to a version (>=1.3.0) which includes the recent patches to force HTTPS for packages downloads and refuse installation if there is a hashsum mismatch. The commands also include installation and setup of virtualenv and virtualenvwrapper, creation of a virtualenv for ooni-backend, and lastly, the installation of ooni-backend without sudo inside the virtualenv.

* Add set of commands for getting oonib up and running, with a safe version of pip, in a virtualenv.

* Change commands to explicitly require Python2.7, because we can guarantee that version works.

…file.

…ld use.

aagbsn · 2013-04-19T22:26:06Z

requirements.txt

 storm>=0.19
-transaction>=1.3.0


transaction, txtorcon, and probably the zope.* packages are required by oonib

Yeah, I added transaction back in. It doesn't seem to be imported or used anywhere though -- is there something strange going on with that package?

And you're correct, we need zope.component: https://travis-ci.org/isislovecruft/ooni-backend/jobs/6485396

I've added the three zope modules back into requirements.txt.

…README.md.

* travis-lint reports: (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ travis-lint Hooray, /home/isis/code/torproject/ooni-backend/.travis.yml seems to be solid! (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ date Fri Apr 19 23:06:31 UTC 2013

… pip1.3.1.

* Change pip's package index command to be in the requirements.txt file.

isislovecruft · 2013-04-19T23:56:00Z

We're not going to be able to support Python3.x for this without a bit of work, I think: https://travis-ci.org/isislovecruft/ooni-backend/jobs/6485395

( cyclone dependency has old except Exception, e: syntax)

isislovecruft · 2013-04-20T00:12:54Z

Where is oonib.conf? It's hardcoded, and doesn't exist in the repo.

isislovecruft · 2013-04-20T00:52:29Z

And we need openssl: https://travis-ci.org/isislovecruft/ooni-backend/jobs/6485817
Not to mention pyOpenSSL. https://travis-ci.org/isislovecruft/ooni-backend/jobs/6486107

aagbsn · 2013-04-20T03:17:19Z

https://github.com/TheTorProject/ooni-backend/blob/master/oonib.conf.example

* (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ date Sun Apr 21 03:18:17 UTC 2013 (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ travis-lint Hooray, /home/isis/code/torproject/ooni-backend/.travis.yml seems to be solid!

…nts.travis". * (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ date Sun Apr 21 03:29:49 UTC 2013 (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ travis-lint Hooray, /home/isis/code/torproject/ooni-backend/.travis.yml seems to be solid!

…config lines.

* Fixed bugs due to several log functions attempting to use "config.advanced.debug" and "config.basic.logfile". These are both located under the config.main section. * Add LICENSE header and encoding specification to log.py. (Yes, we need these.) * Add log.warn() function back in.

…not set.

* txtorcon will handle finding the tor binary for us.

* ∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ date Sun Apr 21 07:25:24 UTC 2013 ∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ travis-lint Hooray, /home/isis/code/torproject/ooni-backend/.travis.yml seems to be solid!

* travis-lint checks out okay: (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ date Sun Apr 21 07:55:19 UTC 2013 (oonib)∃!isisⒶwintermute:~/code/torproject/ooni-backend ∴ travis-lint Hooray, /home/isis/code/torproject/ooni-backend/.travis.yml seems to be solid!

isislovecruft · 2013-04-21T09:46:17Z

Awww yeah. Best commit message evar. :D

chhhhshssshsh
Roger, this is Houston. All lights go, you're good for take off. Out.
chhhshshssshhh

isislovecruft · 2013-05-04T01:33:43Z

bump. this is ready for merge.

could someone please pretty please with cherries on top review it?

isislovecruft · 2013-05-04T12:13:18Z

.requirements.travis

@@ -0,0 +1,25 @@
+## we do not want to override the PI URI for TravisCI:
+#-i https://pypi.python.org/packages


These lines which are commented out are because I discovered that pip-1.3.0 does not actually use SSL all of the time like it claims it does. What it does do is go to https://pypi.python.org/simple/ and select the package name, and then crawl all the links from the package's page, HTTPS or otherwise. So, for PyYAML, it would do go to https://pypi.python.org/simple/PyYAML/ where it sees:
PyYAML-3.10.win32-py3.0.exe
PyYAML-3.10.tar.gz
PyYAML-3.10.zip
3.08 home_page
3.08 download_url
3.09 home_page
3.09 download_url
3.05 home_page
3.05 download_url
3.06 home_page
3.06 download_url
3.04 home_page
3.04 download_url
3.03 home_page
3.03 download_url
3.01 home_page

Instead, if we force pip to use https://pypi.python.org/packages/source/ as its package index, it will use SSL, all the time. However, we cannot simply use pip's options pip --no-index --index-url https://pypi.python.org/packages/source to do this, because then it tries to get everything from there, making it break on Windows and making it break when the packages source is elsewhere (like on the homepage of whatever project). Not to mention that specifying urls like this is subject to change without notice, and so we would have to deal with this shit breaking nonstop.

Therefore, these links are commented out. They might be useful, and they took me a while to find and confirm that they work, so I left them in. But we should just use pip-1.3.0 the normal way, and wait patiently while they slowly upgrade all their links to use SSL, which it appears they are doing.

https://pypi.python.org/packages/2.7/T/Twisted/

hellais · 2013-05-04T14:42:37Z

Thanks for this. Merged!

peer locator: save reported peers to file

isislovecruft added 13 commits April 19, 2013 18:57

Fix missing "#egg=scapy" specification in requirements.txt.

309d1b8

Update install documentation.

1eb5b25

* Add set of commands for getting oonib up and running, with a safe version of pip, in a virtualenv.

Add python2.7-dev to the commands for installing dependencies.

42c8bdf

* Change commands to explicitly require Python2.7, because we can guarantee that version works.

Fix typo in README.md installation commands.

62711b5

Force usage of newer pip even if a distro version is installed.

a3c9707

Force pip to upgrade virtualenv if it was already installed.

75d3ea7

Remove outdated .gitmodules file.

764b4ae

Remove ooni-probe specific (and outdated) documentation from HACKING …

82c5050

…file.

Remove unused txtorcon import statement from /oonib/oonibackend.py.

48172b9

Remove unused modules from requirements.txt.

c93088a

Specify minimum version of txtorcon in requirements.txt.

aabb0d0

Fuck you, pip.

140e801

Force pip to forget whatever builtin PI it thought it thought it shou…

2e62b04

…ld use.

aagbsn reviewed Apr 19, 2013
View reviewed changes

isislovecruft added 7 commits April 19, 2013 23:03

Remove git, twisted-names, and cyclone from the dependencies list in …

972cd19

…README.md.

Add transaction and TwistedNames to requirements.txt.

ad572b1

Apparently, travis-lint doesn't find missing quotes. And TravisCI has…

12d6970

… pip1.3.1.

And that should be a lowercase "i".

35a786c

Install tor keyring in .travis.yml and use deb.tpo for Tor install.

19d00c2

* Change pip's package index command to be in the requirements.txt file.

Removing faulting setup.py code.

4af3f84

Testing pip1.3.1 without explicit https per-package PI URIs.

06611c7

isislovecruft added 3 commits April 20, 2013 00:14

Presuming missing file 'oonib.conf' to be 'ooniprobe.conf.example'.

d6fcb5c

Copy oonib.conf.example, not ooniprobe.conf.example.

fd3ffa8

Remove CI tests for python3.x, add openssl to requirements.

58d6b47

isislovecruft added 2 commits April 20, 2013 00:59

Add pyOpenSSL dependency back into requirements.txt.

b87b8f2

Add openssl self-signed cert generation commands to .travis.yml.

9dbedbe

isislovecruft added 17 commits April 21, 2013 03:18

Update TravisCI config with unattended SSL cert generation commands.

5808bfd

Fix the "got non-string for process arg" errors resulting from blank …

66aa12e

…config lines.

Add file headers and cleanup module docstring in oonib/runner.py.

e77aad6

Cleanup log/print statements and docstrings in oonib/runner.py.

568d05b

Create a tempdir at the top of the repo directory for tor_datadir if …

ad74b06

…not set.

Use the socks_port option from oonib.conf when starting Tor.

be1d323

Don't pass the tor_binary option to txtorcon if not set.

5a45d4a

* txtorcon will handle finding the tor binary for us.

Update the .gitignore.

851e91b

Only import txtorcon classes which are in use.

0fe8088

Fix the "directory not empty" error when removing the tmp tor_datadir.

0a99413

Everything exits 0, and then Travis exits with 1. #FML.

7b9bed9

Only kill it if we didn't already killitwithfire().

aae5648

isislovecruft added 2 commits April 21, 2013 10:30

Updated the README with better instructions. [ci skip]

d676445

Testing the ooni-backend continuous integration buildbot notifications.

75c95e5

isislovecruft reviewed May 4, 2013
View reviewed changes

hellais merged commit 75c95e5 into ooni:master May 4, 2013

isislovecruft mentioned this pull request Jul 8, 2013

pip should not execute arbitrary code from the Internet pypa/pip#425

Closed

ivilata pushed a commit to equalitie/ooni-backend that referenced this pull request Jul 13, 2017

Merge pull request ooni#1 from ivilata/peer-locator-fix

7acb830

peer locator: save reported peers to file

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/install requirements #1

Fix/install requirements #1

isislovecruft commented Apr 19, 2013

aagbsn Apr 19, 2013

isislovecruft Apr 19, 2013

isislovecruft Apr 19, 2013

isislovecruft commented Apr 19, 2013

isislovecruft commented Apr 20, 2013

isislovecruft commented Apr 20, 2013

aagbsn commented Apr 20, 2013

isislovecruft commented Apr 21, 2013

isislovecruft commented May 4, 2013

isislovecruft May 4, 2013

hellais commented May 4, 2013

		@@ -0,0 +1,25 @@
		## we do not want to override the PI URI for TravisCI:
		#-i https://pypi.python.org/packages

		storm>=0.19
		transaction>=1.3.0

Fix/install requirements #1

Fix/install requirements #1

Conversation

isislovecruft commented Apr 19, 2013

aagbsn Apr 19, 2013

Choose a reason for hiding this comment

isislovecruft Apr 19, 2013

Choose a reason for hiding this comment

isislovecruft Apr 19, 2013

Choose a reason for hiding this comment

isislovecruft commented Apr 19, 2013

isislovecruft commented Apr 20, 2013

isislovecruft commented Apr 20, 2013

aagbsn commented Apr 20, 2013

isislovecruft commented Apr 21, 2013

isislovecruft commented May 4, 2013

isislovecruft May 4, 2013

Choose a reason for hiding this comment

hellais commented May 4, 2013