git lookup fails against packed-refs, in a downstream project

[warner]

I noticed a strange test failure in my magic-wormhole project yesterday. The exact message was:

[ERROR]
Traceback (most recent call last):
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/trial/runner.py", line 508, in loadPackage
    module = modinfo.load()
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/modules.py", line 392, in load
    return self.pathEntry.pythonPath.moduleLoader(self.name)
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/reflect.py", line 301, in namedAny
    topLevelPackage = _importAndCheckStack(trialname)
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/reflect.py", line 240, in _importAndCheckStack
    return __import__(importName)
  File "/home/warner/stuff/tahoe/magic-wormhole/src/wormhole/test/test_tor_manager.py", line 8, in <module>
    from ..tor_manager import TorManager, DEFAULT_VALUE
  File "/home/warner/stuff/tahoe/magic-wormhole/src/wormhole/tor_manager.py", line 9, in <module>
    from txtorcon import (TorConfig, launch_tor, build_tor_connection,
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/txtorcon/__init__.py", line 16, in <module>
    from txtorcon.controller import connect
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/txtorcon/controller.py", line 34, in <module>
    from txtorcon.torconfig import TorConfig
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/txtorcon/torconfig.py", line 34, in <module>
    stdout=None, stderr=None):
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 293, in deprecationDecorator
    function, version, None, replacement)
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 238, in getDeprecationWarningString
    _fullyQualifiedName(callableThing), version, format, replacement)
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 198, in _getDeprecationWarningString
    'version': getVersionString(version)}
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/incremental/__init__.py", line 518, in getVersionString
    result = '%s %s' % (version.package, version.short())
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/incremental/__init__.py", line 221, in short
    gitver = self._getGitVersion()
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/incremental/__init__.py", line 422, in _getGitVersion
    return self._parseGitDir(os.path.join(upOneMore, '.git'))
  File "/home/warner/stuff/tahoe/magic-wormhole/.tox/py27/local/lib/python2.7/site-packages/incremental/__init__.py", line 395, in _parseGitDir
    headContent.split(" ")[1]))) as f:
exceptions.IOError: [Errno 2] No such file or directory: '/home/warner/stuff/tahoe/magic-wormhole/.git/refs/heads/master'

wormhole.test.test_tor_manager
===============================================================================

magic-wormhole uses @meejah's txtorcon, which has a (twisted.python.deprecate.)@deprecated wrapper around one function that we call. This wrapper uses an incremental.Version object. When Twisted pulls the version string out of the Version object, it calls an Incremental function which then uses _getGitVersion() to find the exact SHA1 string.

Incremental is assuming that Git repositories always store their references in discrete files, in .git/refs/heads/$NAME. But if you have several branches, and then do a git gc, Git may decide to combine everything under .git/refs/heads/* into a single file (with one ref per line) in .git/packed-refs. The Incremental function fails here.

It's also not clear to me that Incremental should be touching Git at all, in most situations. The @deprecate(Version(..)) wrapper that txtorcon uses is referring to a txtorcon version, but the git tree that is being scanned here is for magic-wormhole (my Tox run is using whatever txtorcon release is most current and meets my setup.py requirements, in this case 0.19.1, installed from a tarball or wheel that tox/pip downloaded from pypi). The txtorcon git tree is long gone.

Versioneer deals with this by modifying _version.py when the project is "built", which includes (among other things) creating a wheel or sdist tarball, so the version string is frozen at the same time the code is frozen. The only time Git is touched is when you're running or testing from the git checkout (for the top-level project), or when you build a sdist/wheel from a checkout. And Versioneer runs git directly, to avoid making assumptions about the internal layout of a .git directory (which might not even be in the same directory as the source tree, if the right environment variables are set).

I'm not sure what the best fix for this would be.. at the very least, the search for .git/refs/heads/NAME needs to be enhanced to read .git/packed-refs too. But I suspect that the entire notion of reading a version from Git needs to be revisited.

[warner]

Oh, I should mention this tox environment was using Incremental-16.10.1 and Twisted-17.1.0, under python-2.7

[warner]

For other folks who are hitting this problem: a temporary workaround is to change your master branch to something else, then change it back. E.g. commit an arbitrary change, then revert it. Or do a reset to some earlier commit, then reset back. This will create a new .git/refs/heads/master, and then Incremental can find the discrete file again.