Support yarn offline mirror

[jasonbarry]

My project uses yarn's offline mirror feature so that we deploy from local tarballs instead of straight from npm. This means that when packages are added/removed, their tarballs should be included in the commit. Even though I've properly set up my .yarnrc file to support this, the pull requests from Renovate don't include the tarballs.

Is this something that could be supported?

[rarkins]

I assume you're talking about your own self-hosted version of Renovate? e.g. for internal enterprise use?

I haven't used yarn's offline mirror before so don't have personal familiarity with it, but I don't see anything about it's description that would stop me accepting ideas or PRs for it. Could you describe in more detail about how you expect it to work from a Renovate perspective?

[jasonbarry]

Sure. I'm using the Renovate GitHub app (not self-hosted).

Here's my .yarnrc file:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


lastUpdateCheck 1515463913082
yarn-offline-mirror "./.yarn"
yarn-offline-mirror-pruning true

Let's say I'm upgrading mapbox-gl from 0.43 to 0.44. On my own, I'd do:

yarn upgrade mapbox-gl --latest

Which results in my git stage to look like

deleted:    .yarn/lodash._baseisequal-3.0.7.tgz
deleted:    .yarn/lodash.isequal-3.0.4.tgz
deleted:    .yarn/lodash.istypedarray-3.0.6.tgz
deleted:    .yarn/mapbox-gl-0.43.0.tgz
new file:   .yarn/mapbox-gl-0.44.0.tgz
new file:   .yarn/source-map-0.7.0.tgz
modified:   package.json
modified:   yarn.lock

Notice the new and deleted .tgz files of mapbox-gl and its dependencies. The diff from Renovate bot, however, looks like this:

modified:   package.json
modified:   yarn.lock

If I merge that in, my deployment will fail, because during deployment I use the yarn install --offline command after I check out my git repo, and the correct versions of the .tgz files aren't present. I'd expect the pull request from Renovate to include the .tgz files.

[jasonbarry]

I've also noticed in the docs the yarnrc config key. Is this supposed to be set to the path of your .yarnrc file? I currently do not include the yarnrc config key in my renovate.json file.

[rarkins]

ok thanks for the detailed description. Just to make sure I'm following, you mean that you are essentially committing all dependencies and sub=dependencies into the repo?

There's nothing "wrong" with the idea, but the bad news is: this is not an easy addition right now, for a couple of reasons:

1. We currently don't git clone anything as we can easily cherry pick which files we need via the GitHub API. Doing this would require at least a shallow clone and then logic added to to detect change and upload them. Otherwise we'd need to GET every single file individually via the API

2. From the app's perspective, it needs more security/thought than if you're just running a bot on repositories of your own that you trust. e.g. if git cloning then need to make sure it's not some "exploding" git repository, etc

Based on my current plans, I'd think this is as late as a 2H2018 type of thing, as I plan to add sure git cloning for other reasons too.

Is this supposed to be set to the path of your .yarnrc file?

No. That's just a way to supply .yarnrc programmatically in case it's not committed to the repo. More common for .npmrc than .yarnrc, because many use the former for auth.

[rarkins]

Do you know of any articles written about this type of use? I went through pages of search results discussing offline mirror without seeing one that talks about committing to the repository itself. What’s the advantage vs just committing node_modules?

[jasonbarry]

@rarkins See the sections You can check in “Offline mirror” into git or mercurial repository and Let’s compare checking in node_modules to checking in “Offline mirror” here: https://yarnpkg.com/blog/2016/11/24/offline-mirror/

Mainly the benefits are fewer files and less noisy diffs.

[rarkins]

@jasonbarry thanks for the pointer - makes sense. I think I found that article but stopped reading too early!

[mrtnzlml]

Hello guys, was there any progress in the last year? This is one of the 2 issues blocking us from adopting Renovate bot in our monorepo at Kiwi.com... :)

[rarkins]

Actually this should be achievable now, although it yet implemented. We support go module vendoring already for example. I’ll take another look soon.

[daveparslow]

I would love to help out with this one, where is the module vendoring code?

[rarkins]

@daveparslow that would be awesome if you have the time to dig into it, even if just to move it along a bit. I'll document requirements here regardless as it's good to have.

So far, the only manager supporting vendored modules is Go. The logic is quite simple - after running "go mod vendor", we then use git to detect all added, modified and deleted files, and make sure they're returned:

renovate/lib/manager/gomod/artifacts.js

Lines 166 to 185 in 956445e

	const status = await platform.getRepoStatus();
	for (const f of status.modified.concat(status.not_added)) {
	if (f.startsWith(vendorDir)) {
	const localModified = upath.join(config.localDir, f);
	res.push({
	file: {
	name: f,
	contents: await fs.readFile(localModified, 'utf8'),
	},
	});
	}
	}
	for (const f of status.deleted || []) {
	res.push({
	file: {
	name: '|delete|',
	contents: f,
	},
	});
	}

Meanwhile, here is where Renovate currently detects/pushes the updated yarn.lock:

renovate/lib/manager/npm/post-update/index.js

Lines 476 to 478 in 956445e

	updatedArtifacts.push({
	name: lockFileName,
	contents: res.lockFile,

Based on the docs, it looks like the offline mirror configuration is in the .yarnrc file. I would guess that right now Yarn will already update these offline modules, so Renovate just needs to (a) detect which cases it should check for them, and (b) add them all to the updated list.

A useful first step would be a simple repo to reproduce it, and a confirmation of the config/yarnrc logic necessary.

[mrtnzlml]

Hello, guys. It's been quite some time so I just wanted to ask for some updates. How is it looking? Is it on your roadmap or we should not wait for it? Thank you very much @rarkins. :)

[rarkins]

@nihal9ns let's first create a test repo as per Yarn's description here: https://yarnpkg.com/blog/2016/11/24/offline-mirror/

[nihalwasheretoo]

@rarkins The test repo lies here : https://github.com/nihal9ns/yarn-offline-mirror

[rarkins]

@nihal9ns thanks. Can you merge that Configure Renovate PR and run again so we see the two upgrade PRs?

This should be our (non-working) baseline and then I'll see to fix them so that we get the full upgrade (including offline module update)

[turnrye]

Hey @rarkins and @nihal9ns, how can I help with this? We're running it on-prem currently and are manually pushing commits up to update the offline mirror today. I'd be interested in helping with this change if you can help set me in the right direction.

[renovate-bot]

🎉 This issue has been resolved in version 19.2.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[rarkins]

@turnrye thanks for the offer, but it should be working now. Please confirm!

[turnrye]

@rarkins wow, thanks! We've updated and are now using it. Will let you know.

Edit: this is working nicely for us now. Thanks for the great help here.

[kanjieater]

This doesn't seem to be working, for me at least.

I added an offline mirror after the initial onboarding, which I don't think matters, but I'm not certain.

project/.yarnrc:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
lastUpdateCheck 1518388841556
yarn-offline-mirror "./.yarn"
yarn-offline-mirror-pruning true

This is located at the same level as the yarn.lock in the project (project/yarn.lock)

My project/renovate.json looks like this:

{
  "extends": [
    "config:base"
  ],
  "automerge": true,
  "major": {
    "automerge": false
  },
  "rebaseWhen": "behind-base-branch",
  "hostRules": [
    {
      "domainName": "github.com",
      "encrypted": {
        "token": "some token, removed"
      }
    }
  ]
}

I'm not getting tar's committed to the repo by renovate. As you can see in the image below, renovate will prune, but it's not adding the updated tar's. The last commit was me manually calling yarn install & then pushing them to the repo.

It's a private project on gitlab, but let me know how I can help.

[rarkins]

@kanjieater please don't comment on long-closed issues, especially ones with quite a few people subscribed. Post in https://github.com/renovatebot/config-help first, provide a reproduction repo, and if it's a bug then we will transfer it into this repository as a new issue.