feat(git): initialise submodules when cloning repos

[JamieMagee]

See https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---recurse-submodulesltpathspec

Fixes #1356

[rarkins]

Question: Don't we need to prevent Renovate from scanning files within the submodules and attempting to update them?

[JamieMagee]

Good question, it's not a scenario I've considered. I'll test later.

[viceice]

Maybe the submodule checkout should be an option, because you have to add an exclude rule for the submodule folders to renovate.

Maybe we can check for a .gitmodules file and extract the folders to exclude from there?

[rarkins]

I think the submodules are only required for artifact generation (lock files, etc), so potentially we can lazily fetch them as-needed. But if we fetch them prior to scanning/extracting then we ideally need a way to automatically exclude them. .gitmodules could be that way.

[rarkins]

There's one possibility I can think of where submodules might be useful during extraction. It's possible for some managers like Maven that a package could point to a "parent" package and that parent package includes information such as the default registry to use. In this case if we don't have submodules resolved then the package package would look like a broken local link. It's an edge case but maybe a valid one.

However, preventing problems during extraction could be pretty simple - we already call getFileList() from within gitfs, so it could pretty easily filter out submodule files at the same time.

[rarkins]

Make sure that the getFileList() function doesn't return submodule files to the upper layer which extracts packages

[JamieMagee]

Sure, I'll see what the best way to do this is, with an eye on implementing #2502 next.

[JamieMagee]

Make sure that the getFileList() function doesn't return submodule files to the upper layer which extracts packages

I've added the check to getFileList

Maybe the submodule checkout should be an option, because you have to add an exclude rule for the submodule folders to renovate.

I'm more of a fan of enabling it by default. I think it's a sensible default, and makes it easier for users to onboard to renovate.

[JamieMagee]

@rarkins This is ready for review

[rarkins]

@JamieMagee hadn't you added code to exclude submodules files from the getFileList()? I don't see it now

[JamieMagee]

Yeah, I even added tests. Looks like the branch I rebased was out of date with origin, so I pushed an old commit.

[rarkins]

Thanks. FYI, my position on rebasing is:

• You don't need to do it, because we always merge squash to master anyway
• I prefer you don't do it, because I have seen too many human errors because of it and I like to keep the history within the PR
• But if you really prefer it, I don't object

[JamieMagee]

Sure, I understand that the merging strategy is mostly a personal preference.

[renovate-bot]

🎉 This PR is included in version 19.52.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[ajcann]

@rarkins Are we able to exclude the submodule recurse from happening? Currently this breaks renovate for us as our git submodules belong to private repositories.

[JamieMagee]

@ajcann Yeah, this should be fairly easy to add a config flag for.