New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Rushjs monorepo support #3681
Comments
Are you able to describe the commands you use if you were to manually make the updates? Ie so I can try to replicate them with Renovate. For lerna for example if we detect a package file is part of the monorepo then we make all changes and then run “lerna bootstrap” after installing the version of lerna specified |
Hello. Sorry, here's a more complete example:
|
I am in a similar situation at https://github.com/neo-one-suite/neo-one, we recently made an effort to migrate to rushjs. Renovate does all of the package.json updates appropriately but we need to be able to apply the changes from |
@danwbyrne note that you actually just need @rarkins The main caveat here, is the lockfile. This will cause a file conflict in every renovate PR as soon as one is merged; effectively being limited to 1 PR concurrency (in our testing using the above command as postUpgradeTask) |
@jlsjonas thanks for the info.
We would also need to detect/know that inside Renovate because we'd make sure we install the right manager and version prior to running
Is this as an alternative to
I think "might" or "often" is more correct than "will"? i.e. if it's npm or yarn lockfiles then |
This can quite easily be detected based on the
As an recommended alternative to being installed globally. npx is not an option as it would introduce a phantom node_modules folder (see https://rushjs.io/pages/maintainer/enabling_ci_builds/)
It doesn't increase chances of conflicts indeed, however what we're experiencing (at least on Bitbucket Server/Stash) is that PR's already receive a conflict stamp (at least on the lockfile) if it's changed, even if there's no line-level conflicts. This might be an issue with BBS more than anything though. |
What's the fallback if none of the lines are uncommented? Or do you think it's a reasonable requirement that Renovate users must specify one? BTW is it mandatory that rush projects include
Sounds like it could be. Ideally it would only conflict if there are line-level conflicts. |
From their getting started:
The rush.json file is generated as part of the
Again from their docs:
So, yes.
Definitely |
@rarkins was there any movement on this? Would the team be open to a PR adding basic rush support? I'd be happy to chip in, although I don't have a good sense of how big a lift it would be 🙃 My take on pnpm vs npm vs yarn with rush: I think it would be fine to only support rush_x_pnpm at least initially. The default rush setup is with pnpm, and it looks to be the rush development team's preferred package manager. So I suspect the majority of rush monorepos are using pnpm, and there'd be significant value in having those ones play nice with renovate. In any case, pnpm would make sense to tackle first. Some more observations that may be relevant to a renovate-rush integration:
import {RushConfiguration} from '@microsoft/rush-lib'
const rushConfig = RushConfiguration.loadFromDefaultLocation()
rushConfig.projects.forEach(p => console.log(p.packageName, p.projectFolder)
CC @octogonz who might correct me on some of the above points. Those are things I've noticed using rush on a public repo: https://github.com/mmkal/ts.
|
I would definitely be open to a PR, as we haven't moved this forward yet. Feel free to ping us here in this issue if/when you uncover any unforeseen decisions needing made. The big challenge is that npm support is already a bit messy due to npm, yarn, pnpm and lerna all essentially being "package managers" for Would rush be like |
Having native support would be great! I've set up Renovate with Rush+pnpm with a config similar to: "postUpgradeTasks": {
"commands": [
// ignore post-install scripts that might not work in the Renovate runner
"echo 'ignore-scripts=true' >> common/config/rush/.npmrc",
// Playwright caches browsers in the home directory, which doesn't work on Renovate
"env PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 node common/scripts/install-run-rush.js update"
],
"fileFilters": [
"common/config/rush/browser-approved-packages.json",
"common/config/rush/nonbrowser-approved-packages.json",
"common/config/rush/repo-state.json",
"common/config/rush/pnpm-lock.yaml"
]
} and it seems to work so far. The ignore-scripts and Playwright config is specific to our setup, but I thought I'd document here in case it's useful to others. |
@Stuk this is great, thank you! I didn't know about |
Renovate searches for all |
@rarkins @viceice do I ask because I'm seeing renovate PRs getting errors along the lines of |
They don't run in the hosted app |
This is probably okay. It's not the Rush way, but it's probably close enough to be accurate.
This is a reasonable bet to make. It might not matter too much however, since the
This doesn't make sense to me. It would work for Lerna folder layouts that have been converted to Rush, but it probably would not work for a typical Rush folder layout.
This is the main point. For a single repo, your algorithm might be like this:
Whereas for a Rush repo, invoking
|
In rush is there the concept of "internal" packages that Renovate should ignore by default? And if so then is it like Lerna and Yarn Workspaces where we should just ignore any internal package dependency where we find the same name in a package.json? |
Architecturally, the design is prescriptive: the Rush projects are enumerated explicitly in Rush's config files (mainly rush.json) and all other package.json files can be assumed to be test assets, or deadwood files from a previous build, or some other role that should probably be ignored. Rush never uses wildcards to discover projects because scanning folders is inefficient, plus a CI machine will often preserve leftover files from a previous build so that caches can be reused. But this is a technical detail. Renovate would probably do fine to use a simple glob to find package.json files and a simple .gitignore type filter to allow users to manually exclude files that got processed incorrectly. It's not ideal, but it's a much simpler contract to support, particularly if you want to be compatible with many different versions of Rush. |
Ohhh wait a sec... I think you're actually asking about local project references (where the a dependency gets symlinked to another project in the repo, rather than being installed from NPM). Yes, Rush does this. There are two models. The installation model we're moving towards is PNPM workspaces model, which use the libraries/stream-collator/package.json . . .
"devDependencies": {
"@microsoft/rush-stack-compiler-3.5": "workspace:*",
"@types/heft-jest": "1.0.1",
"@types/node": "10.17.13",
"@rushstack/eslint-config": "workspace:*",
"@rushstack/heft": "workspace:*"
}
. . . These are easy to detect and ignore. This model is opt-in as an experiment currently, and will become the default in Rush 6. The longstanding older model used normal version ranges, like this example: "dependencies": {
"@microsoft/tsdoc": "0.12.21", // <--- reference to local project
"ajv": "~6.12.3",
"jju": "~1.4.0",
"resolve": "~1.12.0"
}, This is harder to detect. The dependency is considered a local reference (i.e. symlinked instead of installed) only if all of these things are true:
You can use the |
Thank you everyone for the interest and expert details on Rush. File matchingWe'll need to decide between the "traditional" npm manager approach of matching Manager conflictsAssuming we define Obviously we could ask users to manually disable
Do you think (2) is a valid use case, or is (1) simplest and best? Ignoring internal packagesGenerally, if Renovate finds a dependency with a valid version then it tries to update it. For Lerna and Yarn Workspaces, we do a post-processing step to collect all package.json>name fields and then ignore any references to those packages in any other package files. We don't have any fancy logic to check if they're within the valid range or not, etc. If Rush is moving towards the Artifact updatingRenovate uses the term "artifact updating" to mean any updating required after we've patched package files. e.g. may be lock files, checksums, vendored dependencies, or whatever Rush needs. For most package managers we perform the artifact updating immediately after updating a dependency. i.e.
For current npm managers, we delay artifact updating until after all dependency updating / package file patching has been done. Which best describes Rush?
Remember that if we run the Rush command after each patch, it can mean that the same package in multiple package files can have inconsistent versions. e.g. if we want to update a dependency X from 1.0.0 to 1.0.1 in 5 different files then the first four times would mean both 1.0.0 and 1.0.1 are present while only after the 5th time would it be consistently 1.0.1. Installing tools before updating artifactsDo I understand correctly that there's no need to do something like Commit file patternsCan we have pre-defined patterns of which "rush files" to look for modifications for? Although the equivalent of |
My as-a-user, not-a-maintainer views (since I said I'd be willing to open a PR - that's looking ambitious now, but can try to help with hopefully accurate info):
(1) looks simplest and best to me. I suspect the vast majority of rush monorepos just... use rush. If not, the first and best choice for those repos would be to fix that and not have secret projects that their monorepo manager isn't told about. The second choice would probably be to open an issue with rush asking them to fix whatever issue is preventing users from wanting to declare their packages in rush.json. The third option would be to create a new issue here saying "please support my hybrid rush use case" and see how many 👍 s it gets.
For this one, it's pretty important that
No need to
In the repo I've been testing on it's Again, take what I say with a pinch of salt, the above is just based on what I've seen as a (fairly recent) user. |
I agree with everything @mmkal said.
It would be safe to match any file under Rush supports many other advanced version-related features that we could throw at you (e.g. autoinstallers, installation variants, allowedAlternativeVersions, preferredVersions, etc.). But 95% of your users won't use those features, or would be fine if Renovate Bot occasionally makes a PR that needs manual assistance to merge. Better to start simple and see what people actually ask about. |
@rarkins This has the help wanted tag but hasn't been updated in a few months. Is there anything outstanding that needs help specifically? ie, how can we help move this along? This would be very helpful for me to have at the moment. |
Next step is someone to summarise exactly what need to be done and how that fits into the Renovate general flow. Eg is it a new manager, what external commands need running, etc. |
What would that look like? For reference, this is about one of the simplest possible rush repos out there: https://github.com/mscharley/node-presets
How this fits internally into the Renovate lifecycle internally though I can't speak to, I've only used Renovate as a user. I just used this workflow now to bump TypeScript on the above repo and came up with the following PR for a TypeScript library bump: |
So in that case:
Regarding the |
So, the Other than adding cruft to a repo that doesn't use them, I don't believe they're actively harmful. A configuration option would be good. In lieu of adding a a configuration option, I would definitely run it. Adding it won't hurt people who aren't using the automatic changelog features but not having it will be a no-go for people who are as they would have to manually add those files to each PR, rendering the PRs very hard to use. If you do add it as a configuration option, it's safe to default to
|
Wait, I take that back after re-reading the config option, |
Multiple runs after adding all the updates should be idempotent and do nothing though obviously it's going to be a hit to runtime. I haven't tried, but running after each update will likely fail in many cases due to conflicting version specifications across the whole repository. |
So multiple PRs will definitely cause conflicts. |
Rush will install its own copy of /**
* If the "(p)npm-local" symlink hasn't been set up yet, this creates it, installing the
* specified (P)npm version in the user's home directory if needed.
*/
public static async ensureLocalPackageManager( |
@octogonz I've set up Renovate with Rush with a config like below: "postUpgradeTasks": {
"commands": [
"node common/scripts/install-run-rush.js update",
"node common/scripts/install-run-rush.js change --bulk --message \"{{{prTitle}}}\" --bump-type patch"
],
"fileFilters": [
"common/config/rush/browser-approved-packages.json",
"common/config/rush/nonbrowser-approved-packages.json",
"common/config/rush/repo-state.json",
"common/config/rush/pnpm-lock.yaml",
"common/changes/**/*.json"
],
"executionMode": "branch"
}
is there any solution to this problem? |
@rarkins Another problem: why
|
@F3n67u can you try using |
Issue I see in some projects is SOC compliance, making an equivalent of Renovate would require to create a branch with a PR with that change. you would have to create a CommandLineParser and CommandLineAction that does Git things, probably also an HTTP client to create a PR. That's what I will have to do. |
Using this configuration we get a "Artifact update problem" with the following error:
Any ideas what this pattern refers to? |
@omairvaiyani you need add "node common/scripts/install-and-run-rush.js" to https://docs.renovatebot.com/self-hosted-configuration/#allowedpostupgradecommands |
You need to be running self-hosted. |
What's the status for non-self-hosted? I went thru a couple of comments, and I'm unsure if I can or not use Renovate with my rush mono repo. |
|
This comment was marked as spam.
This comment was marked as spam.
This is a working config for self-hosted:
and in
We should commit changes before |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Thanks for posting that config wtuminski 🙏 Here's a tweaked version that handles mutations in .npmrc files when using
|
What would you like Renovate to be able to do?
We have recently moved to a monorepo managed by Rushjs. Rush is similar to yarn workspaces, and projects are defined by entries in a top-level rush.json file. Currently it seems that there's some manual work to update rush lockfiles, and it would be great if renovate did this out of the box.
Describe the solution you'd like
Enable renovate to work with rushjs monorepos. Perhaps custom lockfile logic could occur when detecting a project with a top-level rush.json file and a common/ folder.
Describe alternatives you've considered
We might be able to handle lockfile updating ourselves, with a custom CI runner that builds and updates the lockfile.
Additional context
We experienced this with the Fusion.js codebase after we migrated to a monorepo.
We saw that renovate was opening up Pull Requests, but not updating the common lockfile.
The text was updated successfully, but these errors were encountered: