using a git project with a submodule

[chrisdrackett]

We have a git project that has a submodule. At this point the git project is installed correctly, but the submodule is not updated. It would be nice if pip would grab submodules when installing a git project.

[carljm]

My initial thought is that this is too VCS-specific to have support written into pip. In general, we try to support the lowest common denominator across VCSes but not get into VCS-specific details.

If another maintainer has another opinion, I don't feel strongly about it.

[jezdez]

Yeah, I feel the same about that. In case you're using fabric or similar you can easily script the submodule update in the src/<package> dir.

[chrisdrackett]

what about adding the --recursive flag to clone? that way it'll get the submodule without adding a lot of overhead

[pnasrat]

In my opinion if the repository requires a user to do extra stuff to make it work that should be encapsulated in setup.py.

recursive may be an option to consider, but we'd probably need to do client version checks - it's been available since

http://git.kernel.org/?p=git/git.git;a=commit;h=adc542353199bdc0b9b1ed5d03cf881b7efaff6e

[hltbra]

In my day-to-day work it is very usual to have submodules in projects, and I think we should consider --recursive or alike when cloning a git repo.

A project like Pinax could be made of some git submodules, and that's why I think we should support submodules inititialization (even if it is needed to call update --init).

[chrisdrackett]

Yeah, I've been setting up some new developers and this is the number 1 issue that causes problems. I suppose that we could move the process into setup.py, but it just seems to me that it should be part of pip vs. us having to edit many setup.py files.

[carljm]

I still don't think this is pip's responsibility to a degree that we ought to add additional complication to the git backend for it; the git backend is troublesome enough as-is. If we could just add --recursive to the clone command and be done, I'd be open to that, but if --recursive was just introduced in 2009 I'm not sure we can just drop support for git versions without it. And adding version checks gets back into "more complications" territory; then we have packages installing differently depending on which version of git the user has installed.

@chrisdrackett - I'm not sure I understand why you'd have to edit "many setup.py files" - do you have many Python packages whose repo uses submodules (if so, that seems questionable to me; I'd look at re-thinking that)? Or are you imagining that you'd have to handle it in the setup.py for any package dependent on the one that uses a submodule? I think you should be able to just handle it in the setup.py for the one package using a submodule; have setup.py check if its inside a git repo, and if so run the appropriate submodule update before running an "install" or "develop".

[chrisdrackett]

Ok, so the big example for me at the moment is the javascript framework we're working on has a wrapper app built in django. The framework itself (a javascript repo) is a submodule of the django/python helper app. I plan on doing the same thing with common CSS/sass files that live in a submodule, etc.

So the problem isn't so much with python requirements, those are handled fine by pip, its the non-python files that cause issues.

I'm guessing that this is more rare of an issue for me as a "technical designer" but its a hard point for the developers I'm working with (and deployment) regardless.

[kmmbvnr]

So, what is the correct solution, if submodules will not be handled by pip? Embedding VCS-specific logic into setup.py scripts seems really bad idea.

[kmmbvnr]

If someone stucked with git submodules problem, there is ugly solution:

In setup.py

class build_with_submodules(build):
    def run(self):
        if path.exists('.git'):
            check_call(['git', 'submodule', 'init'])
            check_call(['git', 'submodule', 'update'])
        build.run(self)

setup(
    cmdclass={"build": build_with_submodules},
    ....

[chrisdrackett]

this is still a terrible pain point for us, especially as we're not deploying to heroku

[carljm]

This was fixed in merging #577.