using versioneer in multiple setuptools dependant projects leads to interchanged version numbers

[marscher]

I know only distutils is supported. But still I wanted to report this (maybe to avoid others run into this). I will try to figure out whats going on.

Lets say A depends B (both using versioneer). B is in on pypi with a source tarball, so have a static _version.py file.

A drags in B during setup phase, and version of B gets replaced by version of A.
Can be observed here: https://travis-ci.org/markovmodel/PyEMMA/builds/40644627#L623

Maybe this also occurs if A and B are using distutils (have not tested).

[marscher]

since the dependency resolution takes place in the same python process, my guess is that the same versionfile_source is being taken into account in setup of B.

[marscher]

forced reloading of versioneer module in setup.py of B seems to help 👍 Maybe this should be default behaviour of versioneer (have an import hook, which reloads)?

import imp
fp, pathname, description = imp.find_module('versioneer')
versioneer = imp.load_module('versioneer', fp, pathname, description)

[warner]

Oh, that's tricky.. so the setup.py from both projects is being evaluated in the same python process? And the setup.py's import versioneer is picking up the wrong version of versioneer? And by "wrong", I guess we mean "whichever versioneer.py got imported first is used for all other builds".

Hm. I see the problem, and it's ugly, but I'm hesitant to dive into force-reloads and module-loader internals. Is it possible to have setup.py say from . import versioneer, or is that illegal when we're running it as python setup.py? Hrm, I wish there were a cleaner solution..

[marscher]

If versioneer is on same level as setup.py it should work as expected.

[warner]

Hm. Maybe we could do:

• setup.py defines a dict of config data, and doesn't set versioneer.stuff
• cmdclass = versioneer.get_commands(configdict)

Then the config info could get curried into the command classes, and there wouldn't be any global state to get confused between separate uses of the same module.

[warner]

If we do it that way, we'd also need version=versioneer.get_version(configdict) in the args to setup().

What if we put a .versioneer.ini file in the top directory? The setup.py changes would then be just:

import versioneer
cmdclass=versioneer.cmdclass()
setup(... version=versioneer.get_version(), cmdclass=cmdclass, ...)

We can probably make that backwards-compatible too, by requiring those values to be set if a .ini file isn't found. Or add a [versioneer] section to an existing setup.cfg file, instead of using a new file.

[marscher]

I vote for the [versioneer] section in setup.cfg. This seems to be the right solution!

[warner]

FYI, #74 is where this is happening. Give it a whirl and see if it does what you need. (might still be a bit touchy, let me know if you run into problems)

[warner]

Ok, I just landed the setup.cfg change (1fe079a). I think that should fix this issue. Could you take a look and close it if so?

[warner]

I've been doing a deep-dive of distutils/setuptools internals, and this problem has arisen again. The setup.cfg helps, although I think the current code is looking for setup.cfg relative to the versioneer.py, which is all wrong (I think it needs to look for it in the current directory, and we need to assume/require that you run setup.py locally instead of e.g. python path/to/distant/setup.py install, etc).

Once we fix that, the remaining problem is if A and B are using different versions of Versioneer.

The specific problem has to do with the way that setuptools' setup.py install and setup.py develop commands work. They both build the install_requires= dependencies in the same process. For each dependency, it finds/fetches a tarball, unpacks it into a tempdir, cds into the tempdir, then exec()s the setup.py . B's setup.py does import versioneer, but because it's the same process (and same modules table) as A (which has already imported versioneer), B gets A's versioneer.

This will be troublesome when we make the next release, because it switches to using setup.cfg . So if B is expecting versioneer-0.14 (and configuring it with versioneer.tag_prefix=, etc), but A is expecting the upcoming versioneer-0.15 (and using setup.cfg), then B's setup is going to look for a missing setup.cfg, and fail (or, worse, fall back to something that doesn't work).

I can think of a few options, and I don't like any of them:

1. versioneer install should compute HASH=sha256(versioneer.py) and install it as versioneer_HASH.py instead of plain old versioneer.py. Then the instructions will tell you to edit setup.py to say import versioneer_HASH as versioneer. That way each version of versioneer will be distinct. We probably only need 7 or 8 hex digits to be unique for the foreseeable future.
2. instead of import versioneer, we could recommend the imp.load_module() clause described above, to bypass the module table
3. we could recommend a similar clause that deletes versioneer from the module table just before a normal import. (this might suffer from problems on way back up the dependency tree, where the top-level setup() call gets control back after building all the dependencies, at which point it will be using the wrong versioneer.. I'm not sure if it builds the top-level package first or last).
4. versioneer install could inline the entirety of versioneer.py into your setup.py, instead of writing it to an external file that can be imported
5. we could recommend a setup.py clause that exec()s the nearby versioneer.py, instead of import

Does anyone have any bright ideas that might let us avoid these (ugly) choices? Or any preferences among them?

[warner]

I landed the use-cwd-for-root (instead of relative-to-versioneer.py) change, so that problem is fixed.

I still need to come up with an answer for the second problem.. I'm leaning towards versioneer_HASH.py, but I'm all ears if someone has a better idea.

[marscher]

why is a clean uncached import so inconvenient in your opinion? Having different names for versioneer.py looks ugly

[warner]

I guess I didn't want to ask devs to put a weird-looking imp.load_module() clause in their setup.py.. that's a surprising bit of magic, there.

Fortunately, I lucked upon a better approach. It turns out that setuptools has a sandbox of sorts: when it builds dependencies (as part of setup.py install, setup.py develop, or setup.py easy_install .), it wraps the dependency's setup.py call with code that stores the contents of sys.modules beforehand, and restores it afterwards. Which means that anything imported by the dependency's setup.py won't affect the main/top project.

So we just have to keep the top project's import from affecting the dependency, and we can do that by just del sys.modules["versioneer"] from inside versioneer.py (like when get_versions() or get_cmdclass() are called). The top level setup.py already has its versioneer symbol by this point, so removing it from sys.modules doesn't affect its behavior, but it means the dependency can import versioneer and gets its own version.

The other reason this solution is great is because we can tolerate a dependency that is using an older Versioneer (which doesn't have this trick). If we instead made a new version of Versioneer that somehow auto-reloaded itself correctly, that would help the case where the dependency was using the new version (even if the top-level project were not), but it wouldn't help when the dependency was using an old version (it'd be hopelessly stuck with the pre-imported versioneer.py). That would leave developers in a bad spot: they couldn't use the new Versioneer until all of their dependencies had upgraded first.

This trick seems to decouple the two. Thank goodness setuptools had that sandbox in place already: even though it's only protecting us in one direction, if it wasn't there, we'd still have that troublesome upgrade path.

[marscher]

Thanks, sounds like a practical solution. Does it also work for distutils based projects, where there is no sandbox? I mean I'm happy with this approach, but it may exclude distutils people, which should then at least be documented somewhere.

[warner]

Yeah, it should cover distutils too, at least in any case where it makes a difference. If A depends upon B, then A must be using setuptools, because there's no way to express the install_requires= dependency without setuptools (i.e. distutils.setup() throws an error if you try to pass the install_requires= argument).

It doesn't matter whether B is using distutils or setuptools because the commands that can provoke a subdependency build (like easy_install) forcibly inject setuptools into setup.py's environment whether they want it or not. The sandbox is created before executing B's setup.py, but also setuptools will monkeypatch the distutils command classes so B will use the setuptools versions even if it only imports distutils.