netlify / build-image Public
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for monorepos, including lerna or yarn workspaces #196
Comments
related internal issue: netlify/buildbot#163 |
For my use-case (yarn workspaces) I have been able to work around this issue by using: netlify.toml [build]
command = "npm i yarn@1.10.1 && yarn && yarn build" Of course, this is not ideal as it takes additional time to run npm install and yarn install again, however, the cache dir which the npm installed version of yarn uses is the same so it should be all ready for the next time a build is triggered. Bottom line, it has increased building times by about 1 minute for me. Hopefully, we will be able to revert to the expected behaviour soon, though :) Btw, thanks for the help @fool PS. Perhaps it's good practice to call yarn install with |
I had a frustrating afternoon playing guess-and-check with netlify builds on my yarn workspaces + lerna repo. I'm 99% sure its todo with netlify's caching scheme for node_modules since my build goes from failing to working if i just clear the cache. I'd don't want to have to do this everytime tho... looking for a config option or ENV var to the effect of @zanona you can set a build env var I think why its working for you is actually the UPDATE: can confirm. I had a good deploy. Then I did a no code change (updated a README). Build failed. Redeployed with the "clear cache" btn and it worked. For now, |
Strange. I have a but i changed my
The issue I'm seeing in my dead builds:
was resolved by yarnpkg/yarn#5114 and should work since yarn v1.4
Why do i not have the yarn version i think i should have? UPDATE: @zanona's workaround solved for now. I added a [build]
command = "npm i -gs yarn && yarn build" and my pkgJson#scripts:
|
I expect Netlify to be able to look for different Netlify.toml files: So for deploying a monorepo with multiple sites I would create multiple Netlify sites each of them with a different Netlify.toml. |
Just curious if anyone has [since] had luck with caching non-root |
I am in the same boat, even though I try to bootstrap lerna before the build command it still not able to... |
I believe this is the way to implement the pre-build changes needed by many issues with the current build process. It allows a bit more flexibility for advanced users and does not break things for existing users. Here are some examples: ## Skipping unnecessary language installs (netlify#141) One can create `.netlify/remove_languages.sh` with contents: ```bash rm Pipfile rm Pipfile.lock rm requirements.txt rm runtime.txt ``` ## Specify Composer file (netlify#237) One can create `.netlify/patch_composer.sh` with contents: ```bash mv -f composer-netlify.json composer.json ``` ## Deploying multiple sites from the same repo (netlify#196) Add enviroment vairable $NETLIFY_SITE to project Create a `.netlify/netfily-site.sh` with contents: ```bash mv -f .netlify/$NETLIFY_SITE netlify.yaml ```
+1 |
I also would really like to drop a netlify.toml file into my base dir, and have multiple of them for my monorepo. I was a bit surprised it didn't work like this already. :( |
Any news on this please @fool? In the meantime, please could you update the documentation to make it clear you can only have one The big thing missing for me is the greater redirect control one gets with the I don't really even need to share anything, so presumably you could make each site build only aware of a given sub-dir, as if that's the root of the whole thing - then everything else would work as-is? |
I’m working on a PR that builds both a static documentation Middleman site and an Ember UI. Part of the script involves checking what branch is being deployed and using a different But the caching is my concern, it doesn’t seem like it can be hacked around in this manner |
Netlify now supports netlify.toml files per subdirectory in a monorepo. But still, I am not able to set it up that the npm/yarn cache is working properly. |
Congrats on netlify.toml support. I am having an issue where, when using yarn workspaces, yarn installs all of the monorepo packages. This is problematic because yarn also ends up running package-specific postinstall scripts that break my build. I would like to do something like this to take out the offending package:
But Netlify does not seem to be honoring the rm -rf command. |
for monorepo support where there's a single root yarn.lock, we're currently making our site-specific it might be easy to let us also specify the [build]
install = "cd ../; yarn" or if we could have access to some site-specific preset env variables (or the site name) in [build]
command = "yarn build-$SITE_NAME"
publish = "apps/$SITE_NAME/"
[context.master]
command = "yarn build-$SITE_NAME-dev" or something like [sites.foo]
command = "yarn build-foo"
publish = "apps/foo/"
[sites.foo.context.master]
command = "yarn build-foo-dev" |
I experimented with Yarn Workspaces and Netlify's recently released better support for monorepos (netlify.toml per site in subdirectory). I discovered a setup that might solve some of the above mentioned problems. Assumptions:
So, here's the fun part. Based on facts that:
You can set up your project like this:
And Netlify will pick up your Unfortunately, this does not eliminate the problem with proper caching of I hope this will help someone :-) |
neat trick! trying it now and will update when i get it working! |
FWIW I managed to properly cache lerna packages between the builds. I created a minimal example here: https://github.com/slawekkolodziej/netlify-lerna-caching-demo As for Netlify Build Plugins, it seems that they are executed AFTER the yarn install, plugin won't help here (at least with currently available API) |
Please note a workaround for some of the problems in this issue is ongoing at #440. It allows detecting Yarn using the environment variable |
FWIW, we tried endless ideas to solve this and even tried @slawekkolodziej's super clever idea but couldn't get that to work either as no folder was able to be cache manually by us. We have switched over to doing our builds on Github Actions via Netlify Actions and have cut our build times in half (via being able to cache properly) and are deploying the build immediately to Netlify via the Netlify CLI. Wasn't hard to setup either and saves us a lot of money! |
For anyone else frustrated with the lack of support for caching yarn workspaces, I also went @dwilt's route and implemented a GH action to set env variables based on what branch name is building. This is similar to how you can define variables for different contexts in netlify.toml. Check it out here: production-environment-variables. |
How were you able to cache properly via this action? |
Hey folks Since this ended up working as a "catch-all" issue for many different problems with monorepo support on our build system over time, I wanted to clarify what is supported already and what's missing. Supported:
Missing:
Let me know if I missed anything in here, else I believe we can close this issue and move any remaining discussions around |
@JGAntunes Is there currently a full guide about using monorepos with Netlify? Our case is a bit extreme because we have dependencies that need to be compiled and it takes a lot of time. Each git push leads to recompiling the exact same dependencies 4 - 5 times (once per site) which is very slow and costly. So we are not sure of what's the correct way to go about it with the current capabilities of Netlify. |
@JGAntunes thank you for the clarification! I need to test out the following (haven’t yet because I should be working). It could be a promising pattern (assuming it works) for deploying monorepo projects containing multiple Netlify sites with shared local dependencies. It assumes the use of npm v7 workspaces. Say we have a monorepo project using npm v7 workspaces. It contains multiple Netlify sites (each with its respective subfolder and Say we also have some local “lib” packages within the monorepo. The apps both depend on these libs. However, the apps themselves are independent from one another. The top-level package.json looks like this: {
"name": "some-monorepo",
"private": true,
"workspaces": [
"apps/*",
"libs/*"
]
} The folder structure looks something like this:
In this scenario, the expectation is that upon deployment of either Netlify site, Here’s what # apps/cms/netlify.toml
[build]
# set the repo root as the base folder so that `npm install` runs in that context
# we want npm v7 workspaces install behavior and this would theoretically do that
base = "../../"
# since the base folder is the root of the repo, we first need to change back down into `apps/cms` before building the app
command = "cd apps/cms/ && npm run build"
# since the www and cms apps are independent of one another, we don’t need to build the cms app just because some stuff changed in `apps/www`
# if _only_ something inside apps/www changed, we skip the build! wowzers!
# if there are changes elsewhere (e.g. in the cms package, or in a shared package) the build runs per usual
# the following git diff asks, “what changed in this latest commit, ignoring the apps/www folder”
ignore = "git diff HEAD^ HEAD ':(exclude)apps/www'" TIL about the Anyhoo, I’m supposed to be working, so I’ll come back and test this type of configuration later if I remember. |
@AoDev we've got some base documentation here - https://docs.netlify.com/configure-builds/common-configurations/#monorepos. We have something more thorough being worked on. Also, not sure if it helps, but while working on this I've setup a couple of example test cases in this repo - https://github.com/JGAntunes/gatsby-yarn-netlify-test/blob/multiple-netlify-sties/packages/blog-1/netlify.toml - under different branches. Feel free to take a look and see if something could be of use for you.
Assuming you're using
@tylerbrostrom can't say for sure that will work, would need to test it, but could be interesting |
@tylerbrostrom just tried a similar setup, although using yarn workspaces and no dice - netlify build is throwing an error regarding the base directory -
still looking for a solution to this |
Exciting update about Yarn workspaces: we’ve made improvements to how we cache dependencies for these projects. You can try out our new caching by creating an environment variable called We want your feedback before we roll this out more widely- please let us know here or in the forum what your experience is like! You can also read more about this change here: https://answers.netlify.com/t/improved-caching-for-yarn-workspaces/36066 |
@kaganjd Hello :) I'm trying to use |
Hey @gigamesh, |
@johncastorina were you able to get it working? I’m facing the same error as you |
Update: the caching improvements we shipped for Yarn workspaces have rolled out to all users, so the environment variable Adding a suggestion from that thread for folks who are running into issues: try adding |
Chiming in here for future reference: Our docs now have a more complete section on monorepos and the ignore command for controlling site builds: |
For others finding this issue in the future: This PR just landed adding caching for npm workspaces (I think this would work with Lerna bootstrap too YMMV) if you enable the |
Currently, you're only caching the
node_modules
at the root level.With lerna or yarn workspaces, there are sub-folders with
node_modules
in them that are not cached. This causes a failure to build. It then builds correctly if it's retried with a clean cache.The text was updated successfully, but these errors were encountered: