Support git LFS

[oschwald]

What would you like Renovate to be able to do?

Use renovate with files stored in git lfs.

We use Yarn's offline-mirror feature to vendor the packages. The vendored packages are stored in Git using LFS. When Renovate tries to push the changes, it fails with the message below.

Relevant debug logs

Slightly sanitized error message.

DEBUG: 35 file(s) to commit (repository=group/repo, branch=renovate/js)
DEBUG: Committing files to branch renovate/js (repository=group/repo, branch=renovate/js)
DEBUG: Error commiting files (repository=group/repo, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n", "stack": "Error: Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n\n    at GitExecutorChain.onFatalException (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)" }
DEBUG: Passing repository-changed error up (repository=group/repo, branch=renovate/js)
 INFO: Repository has changed during renovation - aborting (repository=group/repo)

[viceice]

For this we would need to make the renovate git layer git-lfs aware.

[viceice]

@oschwald Can you prepare a minimal public reproduction repo? So we have something to test against it.

[oschwald]

Thanks. I made an example repo at https://github.com/oschwald/renovate-lfs-issue.

With this repo, the exact failure is:

DEBUG: Error commiting files (repository=oschwald/renovate-lfs-issue, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n",
         "stack": "Error: Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n\n    at GitExecutorChain.onFatalException (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }

[viceice]

I think this needs a new repo config options to configure files (minimatch / wildcard) to checkout with git lfs.
By default git lfs should be disabled.

[github-actions]

Thank you for providing a reproduction! 🎉 🚀

The Renovate team will take a look at the reproduction repository. Once we confirm the provided repository reproduces the problem, the label will be changed to reproduction:confirmed.

[sgtsquiggs]

git lfs uses a pre-push git hook, which is disabled by renovate using --no-verify #6521

[amacneil]

Hi friends,

What would it take to get git lfs pull support over the finish line? Can we contribute anything to help?

This is the one feature preventing us from using renovate, and like many others we are on yarn 2 with the yarn binary stored in lfs.

Unfortunately, dependabot doesn’t support yarn 2, and renovate doesn’t support git lfs, so we’re in a bit of a rock and hard place.

[jtbandes]

Hi @rarkins – any chance you can comment on the above? 💕

[rarkins]

@fgblomqvist @viceice could either of you summarize the current status?

[viceice]

We still need to define if it will be always enabled? That way i assume it will always automatically checkout all lfs files. So we don't like to have that.

I think we need to add lfs install to buildpack without full enable it. So install binaries only.

Then we need a configurable list of lfs files, which renovate should checkout from / commit to branch.

[amacneil]

Does anyone need this feature for anything other than yarn 2?

I think it would be sufficient for 99% of users to checkout a default list of files, e.g. git lfs pull --include .yarn yarn.lock **/package-lock.json.

If anyone arrives after that with a request for different files, then it could be made configurable (or just add to the lfs checkout list). I haven't heard any use case for making it configurable yet, but many people (like me) are stuck because of yarn 2.

[ekristen]

@cyrilc-pro that's a fair point, but if it is store in git-lfs today, things will not work properly so it just might have to be a known limitation in the documentation.

[AIGeneratedUsername]

FYI, one issue with Git LFS is that Renovate is running commands like git show origin/master:package-lock.json. This command is OK as long as package-lock.json is not stored using Git LFS.

@cyrilc-pro Could you please clarify what exactly will be "not OK" if a file is stored using Git LFS? Yes, the mentioned command will not display a file content, but what exactly will go wrong in Renovate?

[cyrilc-pro]

@ekristen I'm OK with this being a limitation. In my case, I have a package-lock.json that is almost 1.5MB, which is why our company policy requires it to be stored with Git LFS.

@AIGeneratedUsername The symptom is that the package-lock.json is not updated along with the package.json. The Renovate logs contain a warning about the package-lock.json not being valid JSON:

WARN: Error parsing npm lock file (repository=<redacted>, branch=<redacted>-npm-minor)
      "err": {
        "message": "Unexpected token v in JSON at position 0",
        "stack": "SyntaxError: Unexpected token v in JSON at position 0\n    at JSON.parse (<anonymous>)\n    at parseLockFile (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/utils.ts:10:27)\n    at writeExistingFiles (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/post-update/index.ts:161:24)\n    at getAdditionalFiles (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/post-update/index.ts:473:3)\n    at processBranch (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/update/branch/index.ts:471:31)\n    at writeUpdates (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/process/write.ts:159:17)\n    at update (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/process/extract-update.ts:214:11)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/index.ts:76:19)\n    at attributes.repository (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/renovate.ts:18:22"
      }

The ultimate result is the failure of the npm install command on the branch that Renovate created.

[natan500]

Hi, I have the same issue as @cyrilc-pro mentioned, is there any solution for that?

[ekristen]

At this point if package-lock.json or any file is in git-lfs the git show will not work, but as it stands right now, because they are in git-lfs, and since renovate doesn't support git-lfs, renovate shouldn't be working at all anyways.

I propose we get git-lfs support merged and then we can look to improve the git show command, theoretically you can do git show origin/master:npm-package.json | git lfs smudge

So the code could be changed to do git show and look if version is present with git-lfs header and if so run it through git lfs smudge to get the actual file contents.

[ekristen]

Any thoughts on pushing this forward, I host documentation with my source code as such I have some jpegs and the like stored in git-lfs, this is breaking golang dependencies because the hashes don't match. In my experience supporting git-lfs is fairly trivial. Thanks all!

[rarkins]

I've lost track of what the next steps are

[viceice]

we don't support it because we blocked git hooks for security reasons.

if you self-host, then you can configure to allow hooks and you need to install git-lfs

here's how to do in official renovate images:
https://github.com/containerbase/base/blob/821a5931113d622de322683957f0c435352f703c/test/latest/Dockerfile#L170

[ekristen]

@viceice so is this something that's not going to be supported in the hosted version?

[viceice]

• feat: add git lfs admin option #10748

this was a starter to fully support lfs by renovate. feel free to take and complete it 🤗

[rarkins]

Not in the foreseeable future for security reasons

[viceice]

@viceice so is this something that's not going to be supported in the hosted version?

probably yes, until someone finds a way without git hooks.

[ekristen]

Thanks @rarkins and @viceice -- I'm pretty familiar with git-lfs but I went and did a bit of reading. You said not going to be allowed due to hooks being not supported for security reasons, but as far as I know git-lfs doesn't actually use any hooks, at least not the traditional .git/hooks.

If we were to install git-lfs for the user running renovate and more specifically the git commands, the git lfs install simply adds a filter stanza to the users ~/.gitconfig which is then used during clone and checkout to read the .gitattributes and look for matching filters to do the git-lfs things which to my knowledge filters is in internal git supported mechanism.

I've checked my configuration, and I'm looking at some additional things right now but I can't find any use of hooks or reference to hooks. Am I missing something?

[viceice]

it's using hooks somewhere to push the lfs files, see comments above. that's why you can allow hooks for self-host

[ekristen]

@viceice ah, that's the missing link, for pushing, I see a post-checkout too (now that I'm looking a bit deeper), wonder if they are used for initial cloning though.

What if we supported git-lfs for cloning but not for pushing, make that a limitation. Then for cloning it would work for go mods and others that have hashes of their contents but wouldn't support renovating any files that are actually in git-lfs?

Another approach could be to take the direnv.net approach and allow only hooks that meet a specific hash. For example, we could allow the 3 hook files that get created so long as their hash matches.

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting '.git/hooks/pre-push'.\n"; exit 2; }
git lfs pre-push "$@"

This is the pre-push in a cloned repo that has git-lfs file, renovate could allow this file to exist in the repo as long as the file matches 911a6ef2ce82426869f449da86b8ca5f7cb6205dfd8236b35ef277b26aef9fed if it doesn't match it's removed?

Thoughts?

[viceice]

@rarkins i think we need to revert git-lfs install. otherwise the hosted app will start downloading all lfs files on checkout.

we need a very defensive setting first: #10748