Tests dir is missing files in sdist/pip tarballs

[yurivict]

I'm getting these failures in the FreeBSD port:

$ cd /usr/ports/security/vanguards-tor make test
...skipped...
===>  Testing for vanguards-tor-0.1.1
===>   vanguards-tor-0.1.1 depends on package: py27-pytest>0 - found
===>   vanguards-tor-0.1.1 depends on file: /usr/local/bin/python2.7 - found
============================= test session starts ==============================
platform freebsd11 -- Python 2.7.15, pytest-3.4.2, py-1.5.3, pluggy-0.6.0
rootdir: /usr/ports/security/vanguards-tor/work/vanguards-0.1.1, inifile:
plugins: hypothesis-3.56.10
collected 9 items

tests/test_bandguards.py ..                                              [ 22%]
tests/test_cbtverify.py .                                                [ 33%]
tests/test_main.py FFF                                                   [ 66%]
tests/test_rendguard.py F                                                [ 77%]
tests/test_vanguards.py FF                                               [100%]

=================================== FAILURES ===================================
__________________________________ test_main ___________________________________

    def test_main():
      sys.argv = ["test_main"]
>     vanguards.main.main()

tests/test_main.py:86: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../stage/usr/local/lib/python2.7/site-packages/vanguards/main.py:57: in main
    state.new_consensus_event(controller, None)
../stage/usr/local/lib/python2.7/site-packages/vanguards/vanguards.py:80: in new_consensus_event
    routers = controller.get_network_statuses()
tests/test_main.py:43: in get_network_statuses
    stem.descriptor.DocumentHandler.ENTRIES))
/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:199: in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

descriptor_file = 'tests/cached-microdesc-consensus'
args = (None, False, 'ENTRIES'), kwargs = {}

    def _parse_file_for_path(descriptor_file, *args, **kwargs):
>     with open(descriptor_file, 'rb') as desc_file:
E     IOError: [Errno 2] No such file or directory: 'tests/cached-microdesc-consensus'

/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:264: IOError
_________________________________ test_configs _________________________________

    def test_configs():
      global GOT_SOCKET
      sys.argv = ["test_main", "--control_socket", "arg.sock" ]
>     vanguards.main.main()

tests/test_main.py:97: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../stage/usr/local/lib/python2.7/site-packages/vanguards/main.py:57: in main
    state.new_consensus_event(controller, None)
../stage/usr/local/lib/python2.7/site-packages/vanguards/vanguards.py:80: in new_consensus_event
    routers = controller.get_network_statuses()
tests/test_main.py:43: in get_network_statuses
    stem.descriptor.DocumentHandler.ENTRIES))
/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:199: in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

descriptor_file = 'tests/cached-microdesc-consensus'
args = (None, False, 'ENTRIES'), kwargs = {}

    def _parse_file_for_path(descriptor_file, *args, **kwargs):
>     with open(descriptor_file, 'rb') as desc_file:
E     IOError: [Errno 2] No such file or directory: 'tests/cached-microdesc-consensus'

/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:264: IOError
________________________________ test_failures _________________________________

    def test_failures():
      global THROW_SOCKET,THROW_AUTH,DATA_DIR,NO_HSLAYER
      global TOR_VERSION
      # Test lack of failures
      vanguards.config.apply_config(DEFAULT_CONFIG)
      sys.argv = ["test_main" ]
      try:
>       vanguards.main.main()

tests/test_main.py:127: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../stage/usr/local/lib/python2.7/site-packages/vanguards/main.py:57: in main
    state.new_consensus_event(controller, None)
../stage/usr/local/lib/python2.7/site-packages/vanguards/vanguards.py:80: in new_consensus_event
    routers = controller.get_network_statuses()
tests/test_main.py:43: in get_network_statuses
    stem.descriptor.DocumentHandler.ENTRIES))
/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:199: in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

descriptor_file = 'tests/cached-microdesc-consensus'
args = (None, False, 'ENTRIES'), kwargs = {}

    def _parse_file_for_path(descriptor_file, *args, **kwargs):
>     with open(descriptor_file, 'rb') as desc_file:
E     IOError: [Errno 2] No such file or directory: 'tests/cached-microdesc-consensus'

/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:264: IOError
________________________________ test_usecounts ________________________________

    def test_usecounts():
      rg = RendGuard()
      c = MockController()
    
      i = 0
      while i < REND_USE_GLOBAL_START_COUNT:
        rg.circ_event(c, rend_circ(i))
        assert c.closed_circ == None
        i += 1
        # Verify we're getting the right rend node
        assert rg.use_counts["7791CA6B67303ACE46C2B6F5211206B765948147"].used == i
    
      # Test circuit closing functionality
      c.closed_circ = None
      rg.circ_event(c, rend_circ(i))
      assert c.closed_circ == str(i)
    
      i += 1
      vanguards.rendguard.REND_USE_CLOSE_CIRCUITS_ON_OVERUSE = False
      c.closed_circ = None
      rg.circ_event(c, rend_circ(i))
      assert c.closed_circ == None
      vanguards.rendguard.REND_USE_CLOSE_CIRCUITS_ON_OVERUSE = True
    
      # Test scaling
      state = VanguardState("tests/junk")
      state.rendguard = rg
      i += 1
      while i < REND_USE_SCALE_AT_COUNT*1.5:
        rg.circ_event(c, rend_circ2(i))
        i += 1
    
      assert rg.total_use_counts == REND_USE_SCALE_AT_COUNT*1.5
      assert rg.use_counts["7791CA6B67303ACE46C2B6F5211206B765948147"].used == REND_USE_GLOBAL_START_COUNT+2
    
>     state.new_consensus_event(c, None)

tests/test_rendguard.py:90: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../stage/usr/local/lib/python2.7/site-packages/vanguards/vanguards.py:80: in new_consensus_event
    routers = controller.get_network_statuses()
tests/test_rendguard.py:28: in get_network_statuses
    stem.descriptor.DocumentHandler.ENTRIES))
/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:199: in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

descriptor_file = 'tests/cached-microdesc-consensus'
args = (None, False, 'ENTRIES'), kwargs = {}

    def _parse_file_for_path(descriptor_file, *args, **kwargs):
>     with open(descriptor_file, 'rb') as desc_file:
E     IOError: [Errno 2] No such file or directory: 'tests/cached-microdesc-consensus'

/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:264: IOError
----------------------------- Captured stdout call -----------------------------
WARNING[Wed Aug 08 16:40:39 2018]: Relay 7791CA6B67303ACE46C2B6F5211206B765948147 used 101 times out of 101. This is above its weight of 0
...skipped...
WARNING[Wed Aug 08 16:40:40 2018]: Relay 8C730EAF14903803BA1055202BE65C54105E5C4F used 1398 times out of 1500. This is above its weight of 0
------------------------------ Captured log call -------------------------------
logger.py                   55 WARNING  Relay 7791CA6B67303ACE46C2B6F5211206B765948147 used 101 times out of 101. This is above its weight of 0
...skipped...
logger.py                   55 WARNING  Relay 8C730EAF14903803BA1055202BE65C54105E5C4F used 1398 times out of 1500. This is above its weight of 0
______________________________ test_new_vanguards ______________________________

    def test_new_vanguards():
      state = VanguardState("tests/state.mock2")
    
      # - Load a routerlist using stem
      routers = list(stem.descriptor.parse_file("tests/cached-microdesc-consensus",
                     document_handler =
>                       stem.descriptor.DocumentHandler.ENTRIES))

tests/test_vanguards.py:100: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:199: in parse_file
    for desc in handler(descriptor_file, descriptor_type, validate, document_handler, **kwargs):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

descriptor_file = 'tests/cached-microdesc-consensus'
args = (None, False, 'ENTRIES'), kwargs = {}

    def _parse_file_for_path(descriptor_file, *args, **kwargs):
>     with open(descriptor_file, 'rb') as desc_file:
E     IOError: [Errno 2] No such file or directory: 'tests/cached-microdesc-consensus'

/usr/local/lib/python2.7/site-packages/stem/descriptor/__init__.py:264: IOError
____________________________ test_update_vanguards _____________________________

    def test_update_vanguards():
      controller = MockController()
      vanguards.vanguards.LAYER1_LIFETIME_DAYS = 30
      state = VanguardState.read_from_file("tests/state.mock")
>     sanity_check(state)

tests/test_vanguards.py:117: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

state = <vanguards.vanguards.VanguardState instance at 0x806870200>

    def sanity_check(state):
>     assert len(state.layer2) == NUM_LAYER2_GUARDS
E     assert 4 == 3
E      +  where 4 = len([<vanguards.vanguards.GuardNode instance at 0x806870fc8>, <vanguards.vanguards.GuardNode instance at 0x806870248>, <vanguards.vanguards.GuardNode instance at 0x806870cb0>, <vanguards.vanguards.GuardNode instance at 0x806870128>])
E      +    where [<vanguards.vanguards.GuardNode instance at 0x806870fc8>, <vanguards.vanguards.GuardNode instance at 0x806870248>, <vanguards.vanguards.GuardNode instance at 0x806870cb0>, <vanguards.vanguards.GuardNode instance at 0x806870128>] = <vanguards.vanguards.VanguardState instance at 0x806870200>.layer2

tests/test_vanguards.py:62: AssertionError
====================== 6 failed, 3 passed in 2.72 seconds ======================

[mikeperry-tor]

These are caused by various files missing from the tests subdir in the ports package.

What did you use to make the ports package file list? Did that come from MANIFEST.in?

[yurivict]

What did you use to make the ports package file list?

The --record feature of distutils.

[mikeperry-tor]

Can you tell me the exact commands you ran to get to this test failure, including build and install commands?

I did a --record on an install and I think that might be a different thing. It does not list any test files.

[mikeperry-tor]

(Note I do not have FreeBSD so I can't do the ports thing.. but this is not a FreeBSD-specific issue.. it is an issue with however the list of files was generated in the port tree).

[yurivict]

IMO, you should publish how to run testcases. Please keep in mind that it should run testcases without the package being installed.

[mikeperry-tor]

I run test cases in 'tox', which sets up virtualenvs for python3, python2.7, and pypy and runs the test cases in their own isolated environment, after building pip packages for them.

This is why I am confused and wondering how you built the package and got the file list. The tox system should be building standard pip bdist and sdist packages and running tests on them, in a clean way.

[mikeperry-tor]

(To run tox, install the tox package for your distribution and run 'tox' from the toplevel git checkout.)

[mikeperry-tor]

Downside of tox: it explicitly ignores distribution installations of stem and ipaddress, and uses pinned pip version.

So it still would be useful to know what you used for the ports test that failed, so we can get that working in a way that tests the currently install distribution libraries rather than the pinned pip ones.

[mikeperry-tor]

An alternate way to run tests outside of tox, btw, is something like:

python -m pytest tests/
python3 -m pytest tests/
pypy -m pytest tests/

But again, for that to work, the tests dir hs to be complete. Your tests failed because it was missing things.

[mikeperry-tor]

https://github.com/mikeperry-tor/vanguards/blob/master/README_TESTS.md

[mikeperry-tor]

I am going to close this as notabug, since I believe it to be due to missing test files in the ports tree.

Please let me know if you still have issues after reviewing https://github.com/mikeperry-tor/vanguards/blob/master/README_TESTS.md

[yurivict]

I'm still seeing this problem with version 0.2.2:

$ make test
===>  Testing for vanguards-tor-0.2.2
===>   vanguards-tor-0.2.2 depends on package: py27-pytest>0 - found
===>   vanguards-tor-0.2.2 depends on file: /usr/local/bin/python2.7 - found
=========================================================================== test session starts ============================================================================
platform freebsd11 -- Python 2.7.15, pytest-3.4.2, py-1.5.3, pluggy-0.6.0
rootdir: /usr/ports/security/vanguards-tor/work/vanguards-0.2.2, inifile:
plugins: hypothesis-3.56.10
collected 13 items                                                                                                                                                         

tests/test_bandguards.py ...                                                                                                                                         [ 23%]
tests/test_cbtverify.py .                                                                                                                                            [ 30%]
tests/test_main.py .FFF                                                                                                                                              [ 61%]
tests/test_rendguard.py .                                                                                                                                            [ 69%]
tests/test_vanguards.py ....                                                                                                                                         [100%]

================================================================================= FAILURES =================================================================================
_______________________________________________________________________________ test_configs _______________________________________________________________________________

    def test_configs():
      global GOT_SOCKET
      shutil.copy("tests/state.mock", "tests/state.mock.test")
      sys.argv = ["test_main", "--control_socket", "arg.sock" ]
      vanguards.main.main()
      assert GOT_SOCKET == "arg.sock"
    
>     vanguards.config.apply_config(DEFAULT_CONFIG)

tests/test_main.py:116: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config_file = 'tests/default.conf'

    def apply_config(config_file):
      config = SafeConfigParser(allow_no_value=True)
    
>     config.readfp(open(config_file, "r"))
E     IOError: [Errno 2] No such file or directory: 'tests/default.conf'

src/vanguards/config.py:185: IOError
______________________________________________________________________________ test_failures _______________________________________________________________________________

    def test_failures():
      global THROW_SOCKET,THROW_AUTH,DATA_DIR,NO_HSLAYER
      global TOR_VERSION
      shutil.copy("tests/state.mock", "tests/state.mock.test")
      # Test lack of failures
>     vanguards.config.apply_config(DEFAULT_CONFIG)

tests/test_main.py:142: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config_file = 'tests/default.conf'

    def apply_config(config_file):
      config = SafeConfigParser(allow_no_value=True)
    
>     config.readfp(open(config_file, "r"))
E     IOError: [Errno 2] No such file or directory: 'tests/default.conf'

src/vanguards/config.py:185: IOError
_______________________________________________________________________________ test_oneshot _______________________________________________________________________________

    def test_oneshot():
      global GOT_SAVE_CONF, FAIL_SAVE_CONF
      # Test no saveconf, no exit by default
>     vanguards.config.apply_config(DEFAULT_CONFIG)

tests/test_main.py:289: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config_file = 'tests/default.conf'

    def apply_config(config_file):
      config = SafeConfigParser(allow_no_value=True)
    
>     config.readfp(open(config_file, "r"))
E     IOError: [Errno 2] No such file or directory: 'tests/default.conf'

src/vanguards/config.py:185: IOError
=================================================================== 3 failed, 10 passed in 38.84 seconds ===================================================================
*** Error code 1

The tests are run with this command from the source directory:

cd ${WRKSRC} && PYTHONPATH=${WRKSRC}/src python -m pytest -rs tests

What is wrong?

(You said that there are "missing files in the ports tree". This is run in the source directory, ports tree doesn't have to do with this.)

[mikeperry-tor]

Where did your ${WRKSRC} directory come from?

It looks like tests/default.conf is missing from ${WRKSRC}/tests. That file is in the git repository right here: https://github.com/mikeperry-tor/vanguards/blob/master/tests/default.conf

Do you not have a complete git checkout for some reason? Did git crash or run out of disk space?

[mikeperry-tor]

Ok, I reproduced this by directly downloading vanguards-0.2.2.tar.gz from sdist/pip and then running the tests on that tarball. Looks like that file is not present there.

Adding tests/*.conf to MANIFEST.in seems to fix this for me. Pushing a commit for that.

[mikeperry-tor]

Fix is in git master. Leaving this open until 0.2.3 is tagged.

[yurivict]

Tests work now, thanks!

[mikeperry-tor]

Fixed in 0.2.3 and 0.3.0