Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workspaces: lock file per workspace #5428

Open
netanelgilad opened this issue Feb 28, 2018 · 27 comments
Open

Workspaces: lock file per workspace #5428

netanelgilad opened this issue Feb 28, 2018 · 27 comments
Assignees

Comments

@netanelgilad
Copy link

@netanelgilad netanelgilad commented Feb 28, 2018

Do you want to request a feature or report a bug?
feature

What is the current behavior?
Using yarn workspaces for a monorepo which includes a top level node module creates only a single yarn.lock at the root of the monorepo, with no yarn.lock that is specific for the top level node module.

What is the expected behavior?
I want to use yarn workspaces to manage a monorepo that includes both apps (top level node modules) and libraries. But having only a single yarn.lock file at the root of the monorepo prevents me from packaging my app into a docker image for example. I would love a way to get a yarn.lock file for chosen workspaces that need to have one of their own, because that workspace may later be used outside of the monorepo.

An example:
If I have a monorepo with 2 workspaces: workspace-a and workspace-b. workspace-a uses some of the exported modules from workspace-b. If I want to package workspace-a into a docker image (or any other way of packaging that workspace by itself, without the whole monorepo), I don't have a yarn.lock for it. That means that when I'll move the files of workspace-a into a different environment apart from the monorepo, I'll be missing a yarn.lock file and when installing dependencies, I'll lose all the advantages of a lock file (knowing I'm installing the same dependencies used in development).

I'm quite surprised I couldn't find an issue about this. Am I the only one who wants to work with monorepos that way? Maybe I'm missing something? My current workaround is using lerna without hoisting at all, so I'll have a lock file per package.
The recently released nohoist feature also doesn't seem to help (though I hoped), as it doesn't create a different yarn.lock per workspace.
This issue is somewhat related to this comment on another issue. Thought that it may deserve an issue of its own.

Please mention your node.js, yarn and operating system version.
node 8.9.3, yarn 1.5.1, OSX 10.13.3

@connectdotz

This comment has been minimized.

Copy link
Contributor

@connectdotz connectdotz commented Mar 3, 2018

based on yarn blog:

When you publish a package that contains a yarn.lock, any user of that library will not be affected by it. When you install dependencies in your application or library, only your own yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

it doesn't seem necessary to bundle yarn.lock when publishing the individual package... it is more a development artifact for the whole repo, therefore, no need to put it in each package.

@malbertSC

This comment has been minimized.

Copy link

@malbertSC malbertSC commented Mar 3, 2018

@connectdotz it may not be needed for a library or published package but for building a docker container you're going to want to deploy somewhere it definitely would be.

@connectdotz

This comment has been minimized.

Copy link
Contributor

@connectdotz connectdotz commented Mar 4, 2018

sure... but wouldn't the development docker container just have the whole repo and therefore the yarn.lock anyway? I can see we use docker containers to test our monorepo project for different OS or platforms, in which case we just deploy the whole repo and its yarn.lock. Can you give me an example use case that you need to deploy individual packages from the monorepo project into docker containers during the development cycle, so I can get a more concrete understanding...

@netanelgilad

This comment has been minimized.

Copy link
Author

@netanelgilad netanelgilad commented Mar 4, 2018

So for us, we don't want to package the whole monorepo into the resulting docker container. We are using docker in production and those images should be as light as possible. Our mono repo is quite big and contains multiple microservices that share code between them using library packages (and some of the libraries are relevant for some of the microservices, but not all). So when we package a microservice, we want the image to contain the files of that microservice and any other dependencies as proper dependencies - downloaded from our private registry, and built for the arch of the docker image.

So I think the main consideration here is to keep our docker images as light as possible, and packaging the whole monorepo doesn't fit our needs. Also, when we run "yarn" inside the image of the microservice, we don't want to have symlinks there, just a normal dependency.

The solution here doesn't have to be creating a yarn.lock file per workspace, it could also be a yarn command, that helps in the process of packaging a given workspace, generating a yarn.lock file for a workspace on demand, etc..

Hope it helped clearify the use case.. 🍻

@connectdotz

This comment has been minimized.

Copy link
Contributor

@connectdotz connectdotz commented Mar 6, 2018

@netanelgilad thanks for details, it does help to clarify that your use case is more about publishing individual packages, for production or development, to docker containers. Please join the discussion in #4521 so we can start to consolidate them.

@Pajn

This comment has been minimized.

Copy link

@Pajn Pajn commented Mar 11, 2018

While I can se the use of individual lock files, they are not necessary. If you run docker from the root of the repo with the -f flag pointing to the individual files you'll have the whole repo as context and can copy in the package.json and yarn.lock from the root.

You only need the package.json for the packages you will build in the image and yarn will only install packages for those package.json files you have copied in even thou the yarn.lock includes much more.

EDIT: With that said. it causes docker cache to not be used for package changes in any package even though it is not included in the build

@constfun

This comment has been minimized.

Copy link

@constfun constfun commented Apr 23, 2018

#4206 is related/duplicate, and the use-case described there is exactly the problem we're facing:

Let's say we have ten different packages. We want all of them to live in their own repository so that people can work on them independently if they want, but we also want to be able to link them together if we need to. To do this, we have a mega repository with a submodule for each package, and a package.json that references each of these submodule as a workspace.

@the-spyke

This comment has been minimized.

Copy link
Contributor

@the-spyke the-spyke commented Apr 30, 2018

I have similar issues with workspaces. My project is a web-app which depends on many local packages:

web-app/
|--node_modules/
|--packages/
|  |--controls/
|  |  |--src/
|  |  |--package.json
|  |--utils/
|     |--src/
|     |--package.json
|--src/
|--package.json
|--yarn.lock

Workspace packages controls and utils aren't published and are used by paths. The issue is that I need to release controls package (yarn pack) and I wan't to build/test it on its own. That mean's I want to do yarn install inside web-app/packages/constols/. With workspaces it will use top-level web-app/yarn.lock file together with top-level web-app/node-modules/. So, It installs all packages instead of a subset, specified in web-app/packages/controls/package.json. But I need to check that my package has all required dependencies in it's own package.json and doesn't work by luck of filling missing deps from other workspaces.

There are 2 possible solutions:

  1. If it isn't root, use root's yarn.lock, but install only packages specified in local package.json.
  2. Do not search for top-level configs, but .yarnrc/.npmrc.
@intellix

This comment has been minimized.

Copy link

@intellix intellix commented May 16, 2018

Also struggling with this. We've got an Angular CLI project alongside our API so they're in the same repository and trying to push the frontend to Heroku.

We're using a buildpack which tells Heroku to jump up to the frontend repository first: https://github.com/lstoll/heroku-buildpack-monorepo

Problem is, there's no yarn.lock inside that nohoist package so Heroku just installs with npm and we end up with all new packages rather than the locked ones

@johannes-scharlach

This comment has been minimized.

Copy link

@johannes-scharlach johannes-scharlach commented Jul 10, 2018

You can just use the global yarn.lock file with the individual packages. I've recently approached my Dockerfile like this:

WORKDIR /app
ENV NODE_ENV=production

ADD yarn.lock /app/
ADD package.json /app/

# Only copy the packages that I need
ADD packages/my-first-package /app/packages/my-first-package
ADD packages/my-second-package /app/packages/my-second-package

RUN cd /app && yarn install --frozen-lockfile

This will install only dependencies that are actually in use by the two packages I copied and not by anyone else.

I have a build process where first I'd like to create a release artifact from one package and then not have any of its dependencies installed. This is fairly easy with Docker's multistage build

  1. Add only yarn.lock, package.json and the UI package to docker
  2. run yarn install --frozen-lockfile
  3. run the build process
  4. start a new stage and add yarn.lock, package.json and the necessary runtime packages/workspace folders
  5. Do a COPY --from=<stage> for the built artifact
  6. run yarn install --frozen-lockfile and expose a RUN command.

And you'll end up with a small container that only contains the dependencies specified in your yarn.lock file and needed in production.

@netanelgilad

This comment has been minimized.

Copy link
Author

@netanelgilad netanelgilad commented Jul 19, 2018

@johannes-scharlach
I've ended up using an almost identical method to what you are describing. multistage build is a good tip :).

@connectdotz I think that from my end, this issue could be closed and we can keep working on issue #4521. Since the main yarn.lock file could work with a subset of the packages, it seems like a yarn.lock per workspace is not necessary (even though I still think this could be a better dev workflow 😉 ). But the issue in #4521 is still important, because in the solution we got to here, we need to mention every dependency workspace in the Dockerfile even though yarn should know the interdependencies and how "vendor" a given workspace.

@borekb

This comment has been minimized.

Copy link

@borekb borekb commented Oct 22, 2018

I spent the last couple of days trying to convert our monorepo to first Lerna and then Yarn workspaces. Yarn worked generally more reliably and it's really close to what we need, especially with the recent introduction of yarn workspaces run <script> and other nice things like wsrun.

However, the single yarn.lock is a pain point:

  • I'm not sure how to correctly migrate our existing lockfiles to a single one, see #6563. We have tens of thousands of lines there and adding existing packages as workspaces introduced many subtle versioning issues.
  • Installing dependencies in specific package only ("de-hoisting" / "vendoring") Dockerized build is not well supported, see above (#5428 (comment)) or #4521.

What would you think about Yarn workspaces being just a tiny core – a declaration of packages without any specific functionality. For example:

  • If you wanted to run some script across your workspaces, you'd do yarn workspaces run <script>.
  • If you wanted a single lock file and hoisting (are the two necessarily tied together?), this would be your root package.json:
    "workspaces": {
        "packages": ["packages/*"],
        "hoistDependencies": true
    }
  • If you wanted to migrate your current lockfiles to a hoisted structure, you'd run yarn workspaces hoist-dependencies.

Etc. These are just examples and in practice, some features would probably be opt-out instead of opt-in (for example, people expect a single yarn.lock and hoisting by now) but the general idea is that workspaces would be a lightweight foundation for repo-wide tasks.

What do you think?

@Gudahtt

This comment has been minimized.

Copy link
Contributor

@Gudahtt Gudahtt commented Oct 22, 2018

I believe the problem this feature request is addressing is the same as in #4521 . A command to do essentially what @johannes-scharlach describes would certainly be more feasible than a lockfile per workspace.

There is also an RFC open right now for nested workspaces, which sounds similar to this feature request though I believe it's solving a different problem.

@arcanis

This comment has been minimized.

Copy link
Member

@arcanis arcanis commented Oct 23, 2018

What would you think about Yarn workspaces being just a tiny core – a declaration of packages without any specific functionality

Workspaces won't drastically change, I think we're satisfied with their current interface.

If you wanted to run some script across your workspaces, you'd do yarn workspaces run <script>

That's already possible (v1.10, #6244).

If you wanted to migrate your current lockfiles to a hoisted structure, you'd run yarn workspaces hoist-dependencies.

Since we won't change the workspace interface it would be the opposite (dehoistDependencies).

What I don't like about this is that it takes a technical behavior (hoisting) and tries to turn it into a semantical behavior. You should focus on the user story and then figure out the implementation rather than the opposite.

In this case, I think your use case ("Installing dependencies in specific package only") would be better solved by extending yarn --focus.

@borekb

This comment has been minimized.

Copy link

@borekb borekb commented Oct 23, 2018

I guess the core question is whether hoisting and a single yarn.lock file are strictly necessary for workspaces. I mean, is is what truly defines them or is it "just" the first feature they historically got?

For example, in our use case, the best hypothetical behavior of workspaces would be:

  • Hoist node_modules at development time for efficiency.
  • Keep local yarn.lock files for build (we build specific packages in Docker, something that other people mentioned in this thread as well) and also so that packages can lock their specific versions. See also #6563.
  • Run scripts via yarn workspaces run <script> even if you don't need (or must avoid) hoisting.

Hoisting can be disabled with nohoist, run can be "disabled" by just not using the command but it's not possible to "disable" the single yarn.lock file, and I'm not sure if it's such a core feature that it cannot be disabled or if it just hasn't been requested enough yet :)

@spion-h4

This comment has been minimized.

Copy link

@spion-h4 spion-h4 commented Oct 31, 2018

I think the best way to solve this would be to have yarn install --app-mode package@version

That way, you could simply copy the workspace lockfiles when publishing your app at a certain version, and install in app-mode will respect the bundled lockfile.

Yarn doesn't have to install the entire lockfile; it should easily be able to extract only the part of the dependency graph relevant to that package.

@spion-h4

This comment has been minimized.

Copy link

@spion-h4 spion-h4 commented Oct 31, 2018

In fact this might be fairly easy to do manually even now:

  • download the package zip from the registry directly (yarn has no equivalent, npm does: npm pack package@version)
  • extract the gzip into node_modules/package
  • cd into node_modules/package
  • run yarn install --production from there (it will respect the bundled lockfile)

edit: unfortunately this is all wrong, as the workspace lockfile does not include the versions of packages within the workspace, which might be dependencies of the app package. There would need to be something more involved than copying when creating app lockfiles from workspace lockfiles.

@samuela

This comment has been minimized.

Copy link

@samuela samuela commented Feb 12, 2019

I'm not exactly sure if separate lockfiles is the answer, but I have a similar problem. I have a monorepo set up with a CLI and a backend. The CLI requires a few packages that are platform-specific and only work on desktop machines with a particular setup. On the other hand I need to be able to build my api into a docker image, which is fundamentally impossible in the current implementation of workspaces.

@kristian

This comment has been minimized.

Copy link

@kristian kristian commented Mar 13, 2019

Very similar use case than @samuela here! This one would be massively helpful!

@loopmode

This comment has been minimized.

Copy link

@loopmode loopmode commented May 20, 2019

My use-case might seem laughable compared to the other, "real" ones. But I have a monorepo for some utils - in ths case react hooks - inside packages/*.

I have a second workspace next to packages/*, and that is local/*. This is actually on gitignore, and the idea is that developers in the company may do whatever they like in there, for example put create-react-app apps in there and test the hooks during development.

Now, although the local/* packages are on gitignore, the root yarn.lock is simply bloated and polluted - and checked into git - because of the local workspaces.

What I would wish for is a way to specify that some workspaces shall use some specific lockfiles, e.g. some mapping like so:

  "workspaces": {
    "packages": [
      "packages/*",
      "local/*"
    ],
    "lockfiles": {
      "local/*": "./local.yarn.lock"
    }
  }

Or even a way to specify "do not put anything from this workspace into the lockfile at all".

But yeah, mine is not a serious use-case in the first place :)

@ux-engineer

This comment has been minimized.

Copy link

@ux-engineer ux-engineer commented Jun 6, 2019

I'm not exactly sure if separate lockfiles is the answer, but I have a similar problem. I have a monorepo set up with a CLI and a backend. The CLI requires a few packages that are platform-specific and only work on desktop machines with a particular setup. On the other hand I need to be able to build my api into a docker image, which is fundamentally impossible in the current implementation of workspaces.

You nailed it – as I see it, one of the very core benefits of yarn.lock file is for creating frozen-dep production builds! Did the creators of Yarn forget that?

@ovidiubute

This comment has been minimized.

Copy link

@ovidiubute ovidiubute commented Jun 23, 2019

Another argument for solving the single lockfile problem is code ownership. If you have a monorepo that's using something like the GitHub CODEOWNERS feature, it's not possible to give complete ownership over a package to a group of developers. That's because if they install something in their own workspace, they will invariably change the root level lockfile. This change will need to be approved by the code owners of the lockfile, which, given a monorepo of sufficient scale, will be different than the owners of the original workspace.

@davidwkeith

This comment has been minimized.

Copy link

@davidwkeith davidwkeith commented Aug 19, 2019

Yet another reason to have an option to generate per workspace lockfiles: Google App Engine refuses to launch a Node service without a lock file (NPM/Yarn). This is excellent devops on their part, but a pain for us. So far the options we have are:

  • Deploy all with env vars indicating which service we mean and modify our yarn start (the only supported entry point) to branch based on env vars
  • Have the build script copy the main lockfile to each workspace and deploy just the service we are interested in. (thanks @johannes-scharlach)

Ultimately I think a yarn install --workspace-lockfile command that generated per workspace lockfiles would be the best solution.

@LeeCheneler

This comment has been minimized.

Copy link

@LeeCheneler LeeCheneler commented Aug 21, 2019

Having an option for package level lock files would help us too. Our use case is a little different, we're trialing a new way to manage local dependencies.

So we have some mono repos already, and we have some repos that only contain a single package. These are all published and so can be used together, however there are a lot of times when having them locally and symlinked is super useful.

But some devs have a hard time managing symlinks etc and so we're trying out a standard empty yarn workspace mono repo that we all clone to our machines, then we clone out package repos into that local mono repo. Some of us might just have one packages clones, some might have 5. This is super convenient and makes local, cross repo, cross dependency development an absolute breeze.

But we've come across one problem we can't solve, editing dependencies doesn't update the local yarn lock file, it always updates the root one for the empty mono repo we don't update update dependencies on (it has everything under /packages gitignored.

Having an option to not hoist lock file writes to the mono repo root would be great and have them write out at the package level.

As a note I've also come across the deployment and build issues around Docker that others have mentioned and this would solve that too!

@0x80

This comment has been minimized.

Copy link

@0x80 0x80 commented Sep 2, 2019

This feature would be very valuable to me too. In my case I have a monorepo with some packages deployed to different platforms. One is a Next.js app deployed with Now.sh and the other is a bunch of cloud-functions deployed to Firebase.

In both of these deployments the source code is bundled and uploaded for install & build in the cloud. Not having a yarn.lock file to go along with the source means that the dependencies are installed using the versions in package.json and no versions are locked.

@kylegwalsh

This comment has been minimized.

Copy link

@kylegwalsh kylegwalsh commented Nov 5, 2019

I would also love to be able to enable yarn.lock files in each workspace.

I realize yarn workspaces are mostly intended for monorepos, but our use case if pretty similar to @LeeCheneler's.

Basically, we have created a React component library that we use as a dependency in different projects (that all have their own repos). By using yarn workspaces we can easily reference the local version of the component library and have changes propagate to the local version of our other projects quickly. We also don't need to modify the package.json when we push to production because the dependency library: "*" works without any changes. Our only issue is that, without yarn locks files, the production versions of each project could wind up using different package versions.

I have to imagine that this issue would common among any package developers that use yarn workspaces.

@migueloller

This comment has been minimized.

Copy link

@migueloller migueloller commented Dec 25, 2019

Another critical issue with a top-level lockfile is that it breaks Docker’s layer caching. One would usually be able to optimize Docker caching by first copying the package.json and yarn.lock. If Docker sees no changes in those files, it will use a previous layer. If that lockfile is a single one for the entire monorepo, though, any change in any package invalidates the cache. For us this results in ridiculously slow CI/CD pipelines where every package is built without the cache. There are other tools, like Lerna, that check for package changes to run certain scripts. This also breaks as a dependency change in the lockfile might not be picked up for being at the top level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.