Support for using a monorepo

[yo3jones]

Polymer serve and build fails when organizing packages in a monorepo. Using lerna, yarn workspaces or even yarn link all fail. The problem seems to be linking dependent packages creates nested node_modules folders which creates resolution conflicts at runtime.

A potential solution would be to add a flag for serve, test and build to ignore nested node_modules directories.

[yo3jones]

Another option would be to support package resolution in parent directories. Node resolution will recursively process parent directories until a package is found.

This would allow hoisting dependencies in the monorepo. I think this also works well with yarn workspace as it would be easier to ensure single versions of dependencies and avoid conflicts.

[AusIV]

I've done a good bit of digging into this problem today, specifically for use with WCT. It looks like the "--npm" flag on WCT and polymer serve pretty much just do s/bower_components/node_modules/g and a couple of other changes to make it work. Unfortunately node module resolution is a lot more complicated than just looking in a folder in a predictable location.

In the case of WCT, I was able to get it to find the browser.js file, but it was also injecting the location of several additional files the browser needed to include, and were assumed to be relative to the project root, even including /node_modules/ in the paths. I'm not familiar enough with the code base to readily chase this down, and while I could probably figure out the problem with those imports I don't know how deep this rabbit hole goes, so I'm probably going to look for an alternative to WCT for testing my LitElement components.

[yo3jones]

I ended up moving away from polymer-cli in favor of webpack. webpack handles monorepos well, and it works well with polymer now that the components have moved to named ES6 module imports.

In terms of unit testing, I am using Karma with webpack. It does the job.

[ZempTime]

Since yarn workspaces provides a flat, deduplicated directory with all the dependencies, perhaps we could accomplish this (at least in polyserve) by retargeting this up two levels?

https://github.com/Polymer/tools/blob/master/packages/polyserve/src/make_app.ts#L47

[AusIV]

That might work for a subset of cases, but it's still going to be fragile. Yarn workspaces allow you to specify where your packages are. Convention dictates that they're in workspace_root/packages/, but you could configure it so that some are at workspace_root/packages/thingA/ and some are at workspace_root/packages/thingB/thingC/*.

Beyond yarn workspaces, it's possible that I have packages I need to refer to in ./node_modules/, others in ../node_modules, others in ../../../../node_modules. Right now I'm using a webpack solution for making sure I grab all the content I need, and I don't think I'd switch away from that to a solution that didn't do node module resolution robustly.

[ZempTime]

All right, so I've done a little digging. This is a dealbreaker issue for my organization. Rather than jump ship to webpack, I wanted to see if an update to polymer/tools to fit our use case was possible.

If this looks like something the polymer team would want to support, I'd be happy to work on a pr for this. I'd also be fine maintaining a private fork of polymer/tools my org would use in the meantime so we could get everything exactly right and up to needed standards, so long as there was support for a pr.

So I wanted to go over:

• what we need
• what supporting that might look like

@justinfagnani @usergenic - sorry to ping you two specifically, but yours are the names I've seen all over the place in todo's in the code. I'm not sure who @ otherwise

Core Issue

My org has switched over to using a lerna-based monorepo with yarn workspaces. Yarn workspaces will resolve our dependency tree and install it up at the root of our monorepo. Deduplicated, flat dependency tree, reminiscent of bower & much as polymer needs.

• monorepo
  • node_modules ← all the dependencies for all the packages and everything are here! all linked up automatically
  • packages
    • our-package-1
    • our-package-2
    • our-app ← polyserve assumes node_modules will be in here

My notion is, what if we could direct our componentRequest's to that top-level node_modules instead of searching for it in our monorepo/packages/our-app?

What might this look like & what would be impacted?

I imagine the most minimal amount of change would look like this. In polymer.json or via the cli, we'd specify that we wanted to useWorkspaces:

  "useWorkspaces": true,

We'd then know that componentDir should resolve to monorepo/node_modules instead of monorepo/packages/our-app/node_modules. I hardcoded this in polymer/tools/polyserve locally, and it seemed to serve up everything fine. Except I didn't figure out how to get babelCompile to treat requests for npm modules at this new root correctly yet.

This maintains consistency with the lerna api. It also maintains the assumption of one, flat, deduplicated place where dependencies are stored. However, what I'm not sure about is if what I've said thus far really makes any sense for the polymer project. It could be the desired use cases are varied and could easily be supported by, for example, allowing polymer serve --component-dir /absolute/nonlocal/path/node_modules or some such. I also don't really know how I feel about even useWorkspaces as a part of the cli options as-is.

I'm not sure what other parts of polymer/tools would be affected, either.

Anyway, I'm brand new to typescript, express, and so forth, so I'm not 100% I'm interpreting the whole situation correctly. I'm gonna have to make a decision for my org fairly soon (there's ~20 devs clogged up behind our bad tooling), so I wanted to give this a shake before going the webpack route. If there's a chance that I can update polymer/tools to our use case, then I'll figure it out.

[ZempTime]

having typed all this and "closed my laptop for the day," I'm now wondering why I haven't tried symlinking monorepo/packages/our-app/node_modules -> monorepo/node_modules

[ZempTime]

oh my 🤦‍♂️ I think this worked

[luwes]

Kind of, if you run yarn install again in the root. It removes all packages in the root node_modules :(

[btopro]

our solution based on this -- https://github.com/elmsln/lrnwebcomponents/blob/master/package.json#L24-L38

pre and post install hooks ensure the symlinks are added and removed correctly in order to do this workflow

[luwes]

This almost does the trick (if you have a pkg called pwa-starter-kit)

    "preinstall": "rm -rf packages/pwa-starter-kit/node_modules",
    "postinstall": "yarn preinstall && ln -s $(pwd)/node_modules $(pwd)/packages/pwa-starter-kit",

Only with removing a package it does not remove the symlink before and clears the root node_modules. (doesn't seem to work with preuninstall)

[ZempTime]

To follow-up on this, we ended up moving to a yarn workspaces/lerna/webpack-based approach

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.