eoinof / stem Public

 https://trac.torproject.org/projects/tor/ticket/5810

 Note: this breaks quite a few, arguably broken, unit tests!

1) Added a 'slightly hacky' python-rsa lib implementation of the verification code.
2) Added an 'even hackier' python-crypto implementation of the verification code in
   cases where python-rsa is not installed.
3) Added some logging code.
4) Refactored the digest() function to create the correct digest.

  https://trac.torproject.org/projects/tor/ticket/5810

1)Replaced invalid Relay Descriptor info data with valid data.
   [This breaks quite a few tests that explicitly check the data!]

2)Small change in how mocking._get_descriptor_content(..) expands the templates.
   Previously it was adding a space at the end of the
   signing-key, onion-key & router-signature lines which was wrong.
   Now it does not add a space.

Our enumerations were only defined in our 'Module Overview' section which sucks
for a few reasons...

1. We can't interlink with the enum definitions.
2. Blocks lack any formatting that could improve readablity.
3. In the long term I'd like to replace the 'Module Overview' blocks with
   something better.

Sphinx's python domain directives include a 'data' type, used for global
values. That's exactly what our enums are so using the directive, with tables
to list our enum values.

Oops, not sure how I got this wrong. We want to link to our repository, not the
list of projects.

Moving the exceptions to stem/__init__.py meant that they were no longer
covered my any of our automodule declarations. Adding a special 'Exceptions'
section to the controller's api page.

Fixing #6357

The prior spelling fixes condensed a comment that we had about the error
message tor will provide when we use the wrong controller password. The change
makes sense, but these lines are here to tell us the exact error message tor
provides (after which it's hopefully reasonably obvious why we check for the
"Password did not match HashedControlPassword" part of it).

…fix.

Ok, this puzzled me for a while. Our stem.process integ tests make
'/tmp/stem_integ' to be its temporary data directory for the run, then cleans
it up afterward.

This is great, except that the RUN_SOCKET causes us to run with the following
in its torrc...

ControlSocket /tmp/stem_integ/socket

It's a coincidence that both use the same tmp directory, but it's not
necessarily bad. What was bad, though, is that...

* the stem.process tests broke since the mkdir call for stem_integ failed
* if that hadn't failed our tearDown would have deleted stem_integ, and with it
  our control socket causing all further tests to fail

We should probably use the tempfile module to make our stem.process data
directory, but for now just being lazy and having it account for the RUN_SOCKET
target.

Caught by Ravi on...
https://trac.torproject.org/7284

Logo for the stem project, curtsey of the WP Clipart project.

Adding a link to my contact page. I'd prefer for people to contact tor-dev@,
but I've learned from experience that I'll get quite a bit more input by
providing a text field that doesn't require subscription.

Noting where the site images come from. Almost all of them are from the
collection I've gathered of free image resources
(http://www.atagar.com/freeMedia/).

I've tried pretty hard to avoid images with unknown licenses but there is one
exception: the anvil schematic for the API link. This is a fantastic image
and I've searched quite a bit to figure out where it originally came from
(found it on a blog but doubt that's the source). Pity...

Adding a link to our front page for a bug tracker page on trac that's specific
to our project.

…_file()

When calling the Controller's get_network_statuses() method the last line would
cause an exception because we tried to parse an empty string as a router status
entry. Aborting when we get to the end instead.

…uit.

Attempting to fix https://trac.torproject.org/projects/tor/ticket/7259 by
using a newly created circuit instead of parsing the GETINFO circuit-status
output.

We called os.walk() when determining the files for which we want to check
whitespace. However, when presented with a file rather than a directory this
causes us to not check anything...

>>> list(os.walk("/tmp/foo"))
[]

This broke our attempts to check 'run_tests.py', and let a couple whitespace
issues slip in. Fixing get_issues()'s handling for individual files.

Issue caught by Eoin on...
https://trac.torproject.org/7263

We were using relative paths for our whitespace checks, which caused varying
behavior based on our cwd...

atagar@morrigan:~/Desktop/stem$ ./run_tests.py --unit
...
TESTING PASSED (7 seconds)

atagar@morrigan:~/Desktop/stem$ cd ..
atagar@morrigan:~/Desktop$ stem/run_tests.py --unit
...
WHITESPACE ISSUES
* stem/example.py
  line 18   - indentation should match surrounding content (2 spaces)
  line 19   - missing 'with' import (from __future__ import with_statement)
  line 23   - indentation should match surrounding content (2 or 8 spaces)
  line 35   - indentation should match surrounding content (4 spaces)
  line 72   - line has trailing whitespace
  line 76   - indentation should match surrounding content (0 spaces)
  line 77   - indentation should match surrounding content (0 spaces)

* stem/run_tests.py
  line 289  - indentation should match surrounding content (2 spaces)
  line 486  - line has trailing whitespace

TESTING PASSED (19 seconds)

Note that 'example.py' isn't part of stem. It's an untracked file that I have
in the stem directory. The reason that it's being included in the whitespace
check is that we're grabbing all python files under 'stem' which, now that
we're one level up, is the whole project.

Using absolute paths that are relative of run_tests.py so we get consistent
results.

The process module's integ tests start and stop tor instances. However, if
you're already running tor then the DataDirectory of these instances will
conflict with what you're already running...

Oct 29 19:18:11.532 [notice] Tor v0.2.1.30. This is experimental software. Do not rely on it for strong anonymity. (Running on Linux i686)
Oct 29 19:18:11.540 [warn] ControlPort is open, but no authentication method has been configured.  This means that any program on your computer can reconfigure your Tor.  That's bad!  You should upgrade your Tor controller as soon as possible.
Oct 29 19:18:11.541 [notice] Initialized libevent version 1.4.13-stable using method epoll. Good.
Oct 29 19:18:11.542 [notice] Opening Socks listener on 127.0.0.1:2777
Oct 29 19:18:11.543 [notice] Opening Control listener on 127.0.0.1:2778
Oct 29 19:18:11.543 [warn] It looks like another Tor process is running with the same data directory.  Waiting 5 seconds to see if it goes away.
Oct 29 19:18:16.546 [err] No, it's still there.  Exiting.

Issue spotted by gsathya on...
https://trac.torproject.org/7251

The test_cached_microdesc_consensus test is missing a return after its skip,
causing us to fail if...

1. We don't have a microdescriptor consensus.
2. We're running python 2.5 or 2.6

Spotted by gsathya on...
https://trac.torproject.org/7250

The 'Mirror Mirror on the Wall' tutorial had some repetition due to having both
an example for getting descriptors from the disk and through the control port.
Moving that code to a helper function so I can drop the common parts from the
second example.

The writeheader() method of the DictWriter class was added in python 2.7,
breaking our 2.5 and 2.6 compatability...

======================================================================
ERROR: test_multiple_descriptor_types
----------------------------------------------------------------------
Traceback:
  File "/home/atagar/Desktop/stem/test/unit/descriptor/export.py", line 91, in test_multiple_descriptor_types
    self.assertRaises(ValueError, export_csv,)
  File "/usr/lib/python2.6/unittest.py", line 336, in failUnlessRaises
    callableObj(*args, **kwargs)
  File "/home/atagar/Desktop/stem/stem/descriptor/export.py", line 39, in export_csv
    export_csv_file(output_buffer, descriptors, included_fields, excluded_fields, header)
  File "/home/atagar/Desktop/stem/stem/descriptor/export.py", line 91, in export_csv_file
    writer.writeheader()
AttributeError: DictWriter instance has no attribute 'writeheader'

Noting the prereq in our pydocs and ignoring the 'header' flag if we can't
support it.

In d30a628 I simplified the _get_descriptor_components() by using the
collections.OrderedDict builtin. Unforutunately OrderedDict was introduced in
python 2.7, breaking our 2.5 and 2.6 compatability...

======================================================================
ERROR: test_voting_delay
----------------------------------------------------------------------
Traceback:
  File "/home/atagar/Desktop/stem/test/unit/descriptor/networkstatus/document_v3.py", line 359, in test_voting_delay
    document = get_network_status_document_v3({"voting-delay": "12 345"})
  File "/home/atagar/Desktop/stem/test/mocking.py", line 782, in get_network_status_document_v3
    return stem.descriptor.networkstatus.NetworkStatusDocumentV3(desc_content, validate = True)
  File "/home/atagar/Desktop/stem/stem/descriptor/networkstatus.py", line 458, in __init__
    self._header = _DocumentHeader(document_file, validate, default_params)
  File "/home/atagar/Desktop/stem/stem/descriptor/networkstatus.py", line 541, in __init__
    entries = stem.descriptor._get_descriptor_components(content, validate)
  File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 289, in _get_descriptor_components
    entries = collections.OrderedDict()
AttributeError: 'module' object has no attribute 'OrderedDict'

Caught by eoinof on...

https://trac.torproject.org/7244

I'm fixing this by adding an OrderedDict substitute that's under the MIT
license...

http://pypi.python.org/pypi/ordereddict

Spelling mistake caught by eoinof...
https://trac.torproject.org/7245

This count is then displayed at the end of the test run as
skipped tests may not otherwise be noticed!

Stem's API docs were a monolithic dump of our pydocs. This was utterly
unreadable so breaking it down into individual module pages. I also applied
some uniform formatting changes to all of our pydocs to make them more reader
friendly.

We have a null logging handler to avoid having the logging module give a
warning about no handlers being present. There's no reason for stem's users to
care about it so making it private.

Spell checked all of the stem/* files and the test utilites. Quite a few
mistakes...

We were using definition entries for the AuthMethod enums which was nice, but
gave them more emphasis than they deserved. Users won't usually touch a
PROTOCOLINFO response directly so the enumeration really isn't that important.

I disliked having a '(browse)' link for a couple reasons...

1. people might copy it as part of the clone command if they're new to git
2. it was pretty small considering its relevant importance

Instead linking the clone command itself and the git image.

This is a little different from the other modules in that we're including
stem.response.__init__ as an automodule then autoclasses for all of the
contents. The response classes each have precious little documentation to
having them each on their own page would be pointless.

Also moving the AuthMethod enum to the connection so we can drop the
protocolinfo module documentation (it wouldn't be included since we're using
autoclass).