Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Yarn-pnp, esy, and npm-link #3276

Open
jordwalke opened this issue Jan 13, 2019 · 8 comments
Open

Support Yarn-pnp, esy, and npm-link #3276

jordwalke opened this issue Jan 13, 2019 · 8 comments
Labels

Comments

@jordwalke
Copy link

jordwalke commented Jan 13, 2019

The following workflows and package managers don't work when tools build into node_modules directories, or rely on dependencies existing in node_modules.

Because of this, developers building native apps with esy cannot use bs-platform at all. They have very frequently requested the ability to use esy to build bs apps, but there's not much esy can do at this point, and the solution to esy is the same as the solution to yarn pnp and npm link, so might as well solve all three at once.

  1. Yarn Plug'n'play. (Stores dependencies in global cache).
  2. esy used to manage JS dependencies (uses yarn plugin'n'play standard).
  3. Npm link workflows. Two packages can link to the same dependency, with different build flags so putting build artifacts into that dependency's root will have artifacts conflict with each other).

Here's what I believe needs to happen to make bs-platform work with all three of these. The solutions are largely the same for all use cases:

Stop Building Into Dependencies' node_modules

Solves Problems with: Yarn pnp, Esy pnp.

The idea here is that all libraries should be built into the top level project's root directory. Separate Tracking Issue

Don't assume dependencies even exist in node_modules.

Solves Problems with: Yarn pnp, Esy pnp.

Since esy and yarn both never touch the disk if it can read the dependencies from an immutable cache, we can't assume the packages are located at node_modules. Instead yarn pnp generates a .pnp.js and esy generates a ./_esy/default/pnp.js file. That tells you the real locations of dependencies in the immutable cache. Esy uses the exact same pnp standard, but we can make some changes if it makes it easier for you.

Optimization: Yarn pnp requires starting up a JS VM to determine the location of dependencies. I believe we can make esy's pnp.js file carefully constructed so that bsb could just read the locations directly from the text file by parsing simple strings instead of starting a JS VM. It would also work if you started it up with a JS VM but we can overlay an additional protocol on top of yarn pnp for the sake of additional performance.

Avoid postinstall for bs-platform's installation.

Solves Problems with: Yarn pnp.
Esy and npm can still tolerate a postinstall step for bs-platform, but yarn pnp cannot. It's better to avoid it if possible so that even yarn pnp can work well. If not, then at least esy pnp and npm link workflows would still work.

Model compiler explicitly:

This one is only relevant for esy. But if there is an esy.json file alongside package.json for bs-platform, which expresses the dependency on the ocaml compiler it uses, and its version, then esy could be used to install native ocaml ppx's for BuckleScript projects.

@texastoland
Copy link

texastoland commented May 9, 2019

Keeping notes as I look. AFAIK the main change needs to happen in Bsb_pkg. Every other reference to node_modules is either a template or a test. Also relevant is @bsansouci's stale (what happened?) #2514.

@texastoland
Copy link

texastoland commented May 9, 2019

I believe we can make esy's pnp.js file carefully constructed so that bsb could just read the locations directly from the text file by parsing simple strings instead of starting a JS VM.

The manifest file appears resolved by yarnpkg/yarn#6388 (comment).

@huy-nguyen
Copy link

Is there anything a newcomer like me can do to help this gain some traction? Where should I look to get started?

@BlueHotDog
Copy link

Hi, pushing this up. Yarn Berry, the next version of Yarn is moving fast and gaining traction. TS and all major Js tooling support this already. can we push this higher?

@chriswilty
Copy link

It's gone awfully quiet in here.... Has this been dropped, or just greatly deprioritised?

I didn't even know it was a problem until I started a project with Yarn 2, and wondered why nothing was working, so at least add a warning to the Rescript docs.

@fhammerschmidt
Copy link
Contributor

@chriswilty have you seen https://github.com/reason-seoul/yarn-plugin-rescript/ ?

I think it should be mentioned in the docs somewhere, it is still completely new though.

@cometkim
Copy link
Contributor

cometkim commented Oct 6, 2021

I agree to the assumption that node_modules exists is wrong, but this is too pervasive.

So even the Yarn PnP's approach is better than node_modules, in reality there are many problems with adaptation and that why node_modules linker exist.

What I did in yarn-plugin-rescript was to provide a compatibility layer. I'm thinking if it's possible in other places in a similar way.

@ruittenb
Copy link

Is this issue still on the map?

Kingdutch added a commit to zth/rescript-relay-router that referenced this issue Jul 12, 2022
This follows the instructions from https://yarnpkg.com/getting-started/migration

It ensures we keep using node_modules and don't use Plug 'n Play since
ReScript doesn't support PnP yet. See rescript-lang/rescript-compiler#3276

Yarn 3 provides us some better tools for working with monorepo's. For
example it allows us to reference a consistent version throughout the
workspace.
Kingdutch added a commit to zth/rescript-relay-router that referenced this issue Jul 12, 2022
This follows the instructions from https://yarnpkg.com/getting-started/migration

It ensures we keep using node_modules and don't use Plug 'n Play since
ReScript doesn't support PnP yet. See rescript-lang/rescript-compiler#3276

Yarn 3 provides us some better tools for working with monorepo's. For
example it allows us to reference a consistent version throughout the
workspace.
Kingdutch added a commit to zth/rescript-relay-router that referenced this issue Jul 12, 2022
* Upgrade to Yarn 3

This follows the instructions from https://yarnpkg.com/getting-started/migration

It ensures we keep using node_modules and don't use Plug 'n Play since
ReScript doesn't support PnP yet. See rescript-lang/rescript-compiler#3276

Yarn 3 provides us some better tools for working with monorepo's. For
example it allows us to reference a consistent version throughout the
workspace.

* Upgrade Yarn lock file to version 3

* Make `run-s` syntax Yarn 3 compatible

Yarn 3 includes a built in interpreter so we must quote the glob
expression.

* Update from `--frozen-lockfile` to `--immutable`

The flag has been split out into `--immutable`  and `--immutable-cache`
in Yarn 3. We don't commit the caache so we use the former.

* Disable CI install cache for yarn upgrade
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

9 participants