python2 setup.py test : AttributeError: 'module' object has no attribute 'test_models'

[Dieterbe]

When I run python2 setup.py test on the latest master git checkout (but I've actually been having this problem for months) I get this error.
AttributeError: 'module' object has no attribute 'test_models'

Below is the complete output.

running test
running egg_info
creating src/gensim.egg-info
writing requirements to src/gensim.egg-info/requires.txt
writing src/gensim.egg-info/PKG-INFO
writing top-level names to src/gensim.egg-info/top_level.txt
writing dependency_links to src/gensim.egg-info/dependency_links.txt
writing manifest file 'src/gensim.egg-info/SOURCES.txt'
reading manifest file 'src/gensim.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'src/gensim.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 76, in <module>
    include_package_data = True,
  File "/usr/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/site-packages/setuptools/command/test.py", line 137, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/usr/lib/python2.7/site-packages/setuptools/command/test.py", line 117, in with_project_on_sys_path
    func()
  File "/usr/lib/python2.7/site-packages/setuptools/command/test.py", line 146, in run_tests
    testLoader = loader_class()
  File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
  File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
  File "/usr/lib/python2.7/unittest/loader.py", line 128, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.7/unittest/loader.py", line 103, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python2.7/site-packages/setuptools/command/test.py", line 34, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib/python2.7/unittest/loader.py", line 100, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_models'

For more info, see also http://groups.google.com/group/gensim/browse_thread/thread/80866ad542a56c38

[piskvorky]

This usually means the module cannot be imported; what does from gensim.test import test_models from a python shell give you? (using the same PYTHONPATH etc. as when you run the python setup.py test.

[Dieterbe]

>>> from gensim.test import test_models
>>> 

[piskvorky]

Hmm I just checked the unittest source code (py2.5): that line really just constructs the module, per path component. So it must be some path problem.

Does the same thing happen in the current develop? If yes, try importing test_models in test/__init__.py. Or failing that, get dirty and see what the heck the obj and part objects are in your case.

Btw does this mean you were never able to run the unit tests in recent versions?

[Dieterbe]

I don't recall being able to run the test suite, unless maybe a long time ago(?). that is correct.

However, now I seem to be having some luck.
my master -> test suite runs
my develop -> test suite runs
upstream/master -> same AttributeError
upstream/develop -> test suite runs fine
(upstream means yours)
this is interesting. git diff upstream/master upstream/develop reveals some changes wrt modules etc.
To be sure, I removed python3 from my system, but the problem persists (in your master), so we can be sure it's using python2 correctly. The import test_models in src/gensim/test/__init__.py did not help.
The following code:

        obj = module
        print "obj classname:", obj.__class__.__name__
        print "str(obj):", str(obj)
        print "obj.__dict__:", obj.__dict__
        print "parts:", parts
        for part in parts:
            parent, obj = obj, getattr(obj, part)

reveals:

obj classname: module
str(obj): <module 'gensim' from '/home/dieter/code/gensim/src/gensim/__init__.pyc'>
obj.__dict__: {'__builtins__': {'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'help': Type help() for interactive help, or help(object) for help about object., 'vars': <built-in function vars>, 'SyntaxError': <type 'exceptions.SyntaxError'>, 'unicode': <type 'unicode'>, 'UnicodeDecodeError': <type 'exceptions.UnicodeDecodeError'>, 'memoryview': <type 'memoryview'>, 'isinstance': <built-in function isinstance>, 'copyright': Copyright (c) 2001-2010 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved., 'NameError': <type 'exceptions.NameError'>, 'BytesWarning': <type 'exceptions.BytesWarning'>, 'dict': <type 'dict'>, 'input': <built-in function input>, 'oct': <built-in function oct>, 'bin': <built-in function bin>, 'SystemExit': <type 'exceptions.SystemExit'>, 'StandardError': <type 'exceptions.StandardError'>, 'format': <built-in function format>, 'repr': <built-in function repr>, 'sorted': <built-in function sorted>, 'False': False, 'RuntimeWarning': <type 'exceptions.RuntimeWarning'>, 'list': <type 'list'>, 'iter': <built-in function iter>, 'reload': <built-in function reload>, 'Warning': <type 'exceptions.Warning'>, '__package__': None, 'round': <built-in function round>, 'dir': <built-in function dir>, 'cmp': <built-in function cmp>, 'set': <type 'set'>, 'bytes': <type 'str'>, 'reduce': <built-in function reduce>, 'intern': <built-in function intern>, 'issubclass': <built-in function issubclass>, 'Ellipsis': Ellipsis, 'EOFError': <type 'exceptions.EOFError'>, 'locals': <built-in function locals>, 'BufferError': <type 'exceptions.BufferError'>, 'slice': <type 'slice'>, 'FloatingPointError': <type 'exceptions.FloatingPointError'>, 'sum': <built-in function sum>, 'getattr': <built-in function getattr>, 'abs': <built-in function abs>, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'print': <built-in function print>, 'True': True, 'FutureWarning': <type 'exceptions.FutureWarning'>, 'ImportWarning': <type 'exceptions.ImportWarning'>, 'None': None, 'hash': <built-in function hash>, 'ReferenceError': <type 'exceptions.ReferenceError'>, 'len': <built-in function len>, 'credits':     Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
    for supporting Python development.  See www.python.org for more information., 'frozenset': <type 'frozenset'>, '__name__': '__builtin__', 'ord': <built-in function ord>, 'super': <type 'super'>, 'TypeError': <type 'exceptions.TypeError'>, 'license': Type license() to see the full license text, 'KeyboardInterrupt': <type 'exceptions.KeyboardInterrupt'>, 'UserWarning': <type 'exceptions.UserWarning'>, 'filter': <built-in function filter>, 'range': <built-in function range>, 'staticmethod': <type 'staticmethod'>, 'SystemError': <type 'exceptions.SystemError'>, 'BaseException': <type 'exceptions.BaseException'>, 'pow': <built-in function pow>, 'RuntimeError': <type 'exceptions.RuntimeError'>, 'float': <type 'float'>, 'MemoryError': <type 'exceptions.MemoryError'>, 'StopIteration': <type 'exceptions.StopIteration'>, 'globals': <built-in function globals>, 'divmod': <built-in function divmod>, 'enumerate': <type 'enumerate'>, 'apply': <built-in function apply>, 'LookupError': <type 'exceptions.LookupError'>, 'open': <built-in function open>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'basestring': <type 'basestring'>, 'UnicodeError': <type 'exceptions.UnicodeError'>, 'zip': <built-in function zip>, 'hex': <built-in function hex>, 'long': <type 'long'>, 'next': <built-in function next>, 'ImportError': <type 'exceptions.ImportError'>, 'chr': <built-in function chr>, 'xrange': <type 'xrange'>, 'type': <type 'type'>, '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", 'Exception': <type 'exceptions.Exception'>, 'tuple': <type 'tuple'>, 'UnicodeTranslateError': <type 'exceptions.UnicodeTranslateError'>, 'reversed': <type 'reversed'>, 'UnicodeEncodeError': <type 'exceptions.UnicodeEncodeError'>, 'IOError': <type 'exceptions.IOError'>, 'hasattr': <built-in function hasattr>, 'delattr': <built-in function delattr>, 'setattr': <built-in function setattr>, 'raw_input': <built-in function raw_input>, 'SyntaxWarning': <type 'exceptions.SyntaxWarning'>, 'compile': <built-in function compile>, 'ArithmeticError': <type 'exceptions.ArithmeticError'>, 'str': <type 'str'>, 'property': <type 'property'>, 'GeneratorExit': <type 'exceptions.GeneratorExit'>, 'int': <type 'int'>, '__import__': <built-in function __import__>, 'KeyError': <type 'exceptions.KeyError'>, 'coerce': <built-in function coerce>, 'PendingDeprecationWarning': <type 'exceptions.PendingDeprecationWarning'>, 'file': <type 'file'>, 'EnvironmentError': <type 'exceptions.EnvironmentError'>, 'unichr': <built-in function unichr>, 'id': <built-in function id>, 'OSError': <type 'exceptions.OSError'>, 'DeprecationWarning': <type 'exceptions.DeprecationWarning'>, 'min': <built-in function min>, 'UnicodeWarning': <type 'exceptions.UnicodeWarning'>, 'execfile': <built-in function execfile>, 'any': <built-in function any>, 'complex': <type 'complex'>, 'bool': <type 'bool'>, 'ValueError': <type 'exceptions.ValueError'>, 'NotImplemented': NotImplemented, 'map': <built-in function map>, 'buffer': <type 'buffer'>, 'max': <built-in function max>, 'object': <type 'object'>, 'TabError': <type 'exceptions.TabError'>, 'callable': <built-in function callable>, 'ZeroDivisionError': <type 'exceptions.ZeroDivisionError'>, 'eval': <built-in function eval>, '__debug__': True, 'IndentationError': <type 'exceptions.IndentationError'>, 'AssertionError': <type 'exceptions.AssertionError'>, 'classmethod': <type 'classmethod'>, 'UnboundLocalError': <type 'exceptions.UnboundLocalError'>, 'NotImplementedError': <type 'exceptions.NotImplementedError'>, 'AttributeError': <type 'exceptions.AttributeError'>, 'OverflowError': <type 'exceptions.OverflowError'>}, 'similarities': <module 'gensim.similarities' from '/home/dieter/code/gensim/src/gensim/similarities/__init__.pyc'>, '__file__': '/home/dieter/code/gensim/src/gensim/__init__.pyc', 'corpora': <module 'gensim.corpora' from '/home/dieter/code/gensim/src/gensim/corpora/__init__.pyc'>, 'matutils': <module 'gensim.matutils' from '/home/dieter/code/gensim/src/gensim/matutils.pyc'>, '__package__': 'gensim', '__path__': ['/home/dieter/code/gensim/src/gensim'], 'models': <module 'gensim.models' from '/home/dieter/code/gensim/src/gensim/models/__init__.pyc'>, 'interfaces': <module 'gensim.interfaces' from '/home/dieter/code/gensim/src/gensim/interfaces.pyc'>, '__name__': 'gensim', '__doc__': '\nThis package contains interfaces and functionality to compute pair-wise document\nsimilarities within a corpus of documents.\n', 'utils': <module 'gensim.utils' from '/home/dieter/code/gensim/src/gensim/utils.pyc'>}
parts: ['test']

I could also git bisect this issue to find out the exact cause/fix, OTOH, develop is more important anyway...

[piskvorky]

Ok, cool! So the new develop works.

Still, if you get round to finding the real error, let me know. In case it was some nasty hidden bug that's still sitting there, silently. For example, now according to your ouput it looks like it fails on loading test (from parts), not test_models like before.

[Dieterbe]

Closing this, master and develop both work fine now. Technically the issue could still be sleeping somewhere in the code, but as long as it doesn't show up I don't think we should spend time on it.