New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yarn's Future - v2 and beyond #6953

Open
arcanis opened this Issue Jan 24, 2019 · 45 comments

Comments

Projects
None yet
@arcanis
Copy link
Member

arcanis commented Jan 24, 2019

Hi everyone,

When the Yarn project started back in 2016, our landscape was very different from what it is now. Package locking was far from being a first class citizen in the Javascript ecosystem, and the time needed to run an install was ... well, it was what it was. Yarn's release shook the status quo and started a movement that ended up being beneficial to everyone, other package managers included. Now is the time to assess where we are, reinforce our strengths and patch our weaknesses.

This thread aims to expose our roadmap for the next major Yarn release, and let you know about significant changes that we plan to make regarding Yarn's design. The codename for these changes is Berry - that's how we'll refer to it during the next few months.

Roadmap

This section lists some of the work we have in progress. It's far from being a comprehensive list, but it gives a good idea of the kind of changes you can expect going forward:

Major changes

  • The lockfile (and configuration) format will become a strict subset of YAML. In the process, the lockfile information will be slightly changed to account for some long-awaited changes (such as #5892).

  • We'll add support for plugins, which will be able to alter various things - from adding new commands to hooking into the resolution / fetching / linking steps to add support for new package sources or install targets.

  • Related to the plugin system, Yarn will become an API as much as a CLI. You can expect to be able to require it and start using its components in your script - no need to parse your package.json anymore, no need to run the resolution .. Yarn will abstract all those tedious tasks away.

  • Support for both Node 4 and Node 6 will be dropped. We don't expect Berry to have stable releases until after Node 6 gets EOL (expected April 2019), by which point it won't matter anymore.

  • The log system will be overhauled - one thing in particular we'll take from Typescript are diagnostic error codes. Each error, warning, and sometimes notice will be given a unique code that will be documented - with explanations to help you understand how to unblock yourself.

  • Some features currently in the core (such as autoclean) will be moved into contrib plugins. They'll still be supported, but might have a different release cycle than the standard bundle.

  • The codebase will be ported from Flow to TypeScript. To understand the rational please continue reading, but a quick summary is that we hope this will help our community ramp up on Yarn, and will help you build awesome new features on top of it.

  • The cache file format will switch from Tar to Zip, which offer better characteristics in terms of random access.

New features

  • Nested workspaces (when one of your workspaces itself defines additional workspaces) will be supported out of the box. This will open up new exciting usages, especially when used together with git submodules.

  • Running yarn run foo:bar (note the :) will run the foo:bar script regardless of which workspace implements it. This will make working with monorepos a lot easier, as you won't have to constantly cd between directories anymore.

  • Running yarn ./packages/my-package add foo will run yarn add foo into my-package. This should also contribute to make monorepos more pleasant, as you won't have to cd to add a new dependency somewhere.

  • A new resolution protocol, workspaces:, allows you to force the package manager to link one of your packages against a workspace. This makes it impossible to accidentally use a version of a package obtained from the remote registry.

  • A new command will appear, yarn constraints. This command will allow you to enforce constraints across workspaces - for example to require all your declared dependencies to use a unique version of a package.

  • A new type of dependencies will be introduced, portals. Portals are like link:, but follow the chain of dependencies (whereas link: dependencies never have any dependencies, but don't require a package.json either).

  • The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).

  • Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

  • Scripts will be able to put their arguments anywhere in the command-line (and repeat them if needed) using $@. Similarly, scripts will have access to $1, $2, etc.

  • The PnP hook will be able to read files directly from the cache Zip archives - not unlike the phar archives in php.

  • Our cache will finally become fully atomic - multiple Yarn instances will be able to run concurrently on the same cache without risking corrupting the data.

And of course, a large amount of bugs are also expected to disappear as we modernize the codebase. A lot of those were caused by intricacies in the various components making up Yarn and, by cleanly separating them into standalone steps, we expect the overall logic to become much clearer and less error prone.

As you might guess, this isn't a comprehensive list - we plan Berry to be the big refactoring that Yarn had needed for a while now so many internal systems will be reworked. After that we expect following majors to become much smaller, as we'll have strong foundation for building new features and reworking the project one component at a time.

Yarn's philosophy

Now that we've discussed about most of the "What", let's discuss about the "Why". First, those are the values we always believed in and around which we continue to design most of our features.

Developing Javascript projects shouldn't leave the door open to surprises

This is our number one motto. We believe that Yarn should have a totally predictable and reproducible behavior. That your package manager has a responsibility to warn you when your project relies on unsafe patterns that could jeopardize your application at the worst possible time.

Developing Javascript projects should be easy

The complexity of the average Javascript project grown during the past years in order to accommodate with the ever increasing number of use cases people found. We believe that the package managers are in a unique position to help decrease some of the cognitive load and make it easy for their users to manage their projects and follow good practices.

Contributing to the Yarn ecosystem should be simple

Because a single project will never be able to satisfy all the needs from all developers counting on us, various projects spawned and grown, reinforcing Yarn rather than competing against it. Some examples are Create-react-app, Lerna, OpenCollective, or patch-package. We like that approach and want to give you the tools to do even more in a more efficient way. Not all features belong to the core, but being able to innovate and experiment is at the core of our philosophy, and we really mean it.

Goals for 2019

Now that our values are clear, what are the next steps we need to do in order to reach our goal? Note that some are long-term, some short-term, and that all are open to discussion (ideally in separate threads, otherwise it might get noisy 🤡).

Yarn will become a development-first tool

We've been saying it for some time now, but now we're ready to start going for it. Package managers are not tools you should ever run on your production servers. The more code run there, the higher are the probabilities that something will eventually go wrong and bring your production systems down. Yarn being development-first means that we'll make it possible for you to reach the state where you clone a repository, and voilà. This includes putting the emphasis on Plug'n'Play, which has been designed with this specific use case in mind.

Yarn will be rewritten in TypeScript

Yarn is currently fully covered by Flow types, and to be honest it works pretty well for us - the key part being “us”. Because we want to make it as easy as possible for third-party contributors to shim in and help us maintain this awesome tool, we'll switch the codebase to TypeScript. We hope this will help making the codebase feel more familiar than the projects your already contribute to.

Yarn will become an API, and its internal components will be split into modular entities

This one is huge. At the moment, when working with Yarn, your only option is the command line interface. We don't offer primitives that allow you to take advantage of the complex logic we've implemented - whether it's the resolvers, linkers, or even accessing the configuration. This will change, and Yarn will be redesign to become an API first and a CLI second.

Yarn will be designed to support different install targets than Node

Package management is a problem that keeps on being reinvented - even Yarn itself did it in the past. We believe the reason for this is that all package managers need to layout the installed packages in slightly different ways in order for them to be read by the host. This unfortunately is rarely possible, and it ends up being easier to rewrite the package manager and discard the already-existing user experience and features. Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Overall compatibility will be preserved when possible

Such semver-major changes are by design backward-incompatible, but we'll make sure to keep them to an acceptable level. In particular, the core command (yarn install, yarn add, yarn remove, yarn run) will keep the same behavior, and your previous yarn.lock will be silently migrated. One important caveat: our installs will now use Plug'n'Play by default. We've proven during the past few months that the approach was sound, worked to trim all potential implementation issues, and finally discussed with project maintainers to figure out whether there was anything we could do to help them make sure everything was ready. Now is the time to take the plunge.

FAQ

When will it be released?

We'll open a new repository early February with the v2, the time to flesh out some remaining interfaces. Starting from then, we'll make Berry accessible simply by running yarn policies set-version berry within your project.

What will happen to the v1?

As mentioned before, Berry won't have a stable release until this summer, and work will continue on the v1 until then. Even after Berry comes live, the v1 branch will still stay well maintained for the foreseeable future, perhaps distributed through a codename. This will ensure that your applications that work now continue to work in the future, until you feel ready to move to the v2.

How can i help?

The biggest way you could help us at the moment would be to enable PnP (yarn --pnp) and report any issue to the maintainers of the packages that don't work yet (ideally with a PR if you have the time). Most of the work they would need to do is to make sure they use the standard require.resolve api, or offer a way to configure a different resolution strategy. At the moment the biggest blocker we have is the tsc default CLI, which doesn't do either (thankfully ts-loader is just fine - we actually use it in order to compile our bundles!).

@heupr heupr bot assigned arcanis Jan 24, 2019

@heupr heupr bot added the triaged label Jan 24, 2019

@Jessidhia

This comment has been minimized.

Copy link

Jessidhia commented Jan 24, 2019

[*]: Lack of tsc support also affects code completion in VS Code, as it uses the same code as tsc for looking them up.

It may appear to work out of the box sometimes but that's only for types published in DefinitelyTyped which are fetched in the background by the IDE.


This small bit of negativity aside, really looking forward to the future, in particular to better workspace workflows 🎉

And, who knows, tsc support for pnp 😇

@felixfbecker

This comment has been minimized.

Copy link

felixfbecker commented Jan 24, 2019

Love the idea of plugins. Use cases I have are filtering out dependencies that should not be installed (by looking at their package.json metadata or files list) and filtering out files that aren't needed and should not be extracted to disk.

@jleclanche

This comment has been minimized.

Copy link

jleclanche commented Jan 24, 2019

Writing posix command lines inside your scripts field will work regardless of the underlying operating system. This is because Berry will ship with a portable posix-like light shell that'll be used by default.

That sounds promising. Does that mean no more need for things like cross-env to be able to do MY_ENV=foo some-command?

@bartlangelaan

This comment has been minimized.

Copy link

bartlangelaan commented Jan 24, 2019

Question!

The lockfile (and configuration) format will become a strict subset of YAML.

Yarn will be designed to support different install targets than Node [...]
Starting from Berry, we made it an explicit goal that each component of our pipeline can be switched to adapt to different install targets. In a way, Yarn will now be a package manager platform as much as a package manager. If you're interested into implementing PHP, Python, Ruby package installers without ever leaving Yarn, please open an issue and we'll help you get started!

Does this also mean that the yarn.lock is going to be optional, and Yarn would be able to (with the help of plugins) read and write to (for example) package-lock.json and in the future composer.lock?

Or will Yarn still depend on the yarn.lock file to save the exact state of the Node / PHP dependency tree?

It would be awesome if Yarn can be used on projects built for NPM.

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 24, 2019

Ah ah, good question - no (mostly), and yes (but not really) 🙂

The new architecture will see its components moved into various packages - for example plugin-http will only contain the code needed to resolve and fetch the packages when referenced through https?: URLs. Those plugins will all use the API interfaces exposed by the core package in order to work properly - and because (at least at the beginning) the core package will handle the generic resolution algorithm, it will be its task to load the lockfile.

The core package will not support anything else than yarn.lock. We have to put the line somewhere, and given that we don't want people to use Yarn and npm on the same projects (use any of them, but stay consistent) we won't make efforts to support this use case. Additionally, the npm lockfile has characteristics that simply don't make it suitable for us - and the other way around. We've actually discussed it before (#5654).

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

@mbeaudru

This comment has been minimized.

Copy link

mbeaudru commented Jan 24, 2019

The yarn link command will now persist its changes into the package.json files (through link: dependencies), and the linked packages will now properly resolve peer dependencies (this feature is brought to you by Plug'n'Play 😉).

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway? Ref: #6888

Does it means it will be required to use Yarn PnP to use this feature ?

Anyway, thanks for the share this list is awesome!

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 24, 2019

So when a package will be linked into a project, even if the package finds it's dependency in it's surrounding before finding it in the project, it will use the peerDependendy provided by the project anyway?

Yep.

Does it means it will be required to use Yarn PnP to use this feature ?

Yes, this is only possible because PnP is aware of the full dependency tree (when it would otherwise be Node that would try to locate the link dependencies, based on the realpath).

@jleclanche

This comment has been minimized.

Copy link

jleclanche commented Jan 24, 2019

Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

@arcanis Stepping away from the "using the same package-lock.json file" question; was it considered to use the same format as npm's package lock? Or a superset of it?

I really like yarn.lock becoming programmatically accessible as YAML, but using the same format would make it easier to read both using the same tooling.

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 24, 2019

was it considered to use the same format as npm's package lock? Or a superset of it?

Yes, but as I mentioned the npm format has (what we believe are) design flaws that make it incompatible with our goals. We've already got this discussion in the linked thread 😉

@gregjw

This comment has been minimized.

Copy link

gregjw commented Jan 24, 2019

Can't wait!

@bartlangelaan

This comment has been minimized.

Copy link

bartlangelaan commented Jan 24, 2019

That being said, I guess you could reimplement the core and read from the package-lock.json file. It's just that it wouldn't map well with the design of the engine, and would (imo) be counter-productive. Again, use a single version of a single package manager in a single project - that's the only way you can ensure reproducible results and get rid of "work for me".

I understand completely that it can lead to unexpected results, but as long as its logic can be overwritten with plugins, it can be easier to migrate and convince others to use Yarn on already existing NPM and Composer projects. But of course, only with the help of plugins; this helps keep the core stable.

@flybayer

This comment has been minimized.

Copy link

flybayer commented Jan 24, 2019

Is there any way to add npm’s audit feature to yarn? IMO, that is a huge step toward better security, and I would love to have inside yarn.

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 24, 2019

Yarn already implements audit since the 1.12 thanks to @rally25rs's efforts 🙂

@kevinSuttle

This comment has been minimized.

Copy link

kevinSuttle commented Jan 24, 2019

Yarn will be designed to support different install targets than Node

Hoping for solid interop with Rust crates!

@michax

This comment has been minimized.

Copy link

michax commented Jan 24, 2019

Yarn will be designed to support different install targets than Node

Really cheering for that one!

@ajbogh

This comment has been minimized.

Copy link

ajbogh commented Jan 24, 2019

Would love a future writeup on the Flow=>Typescript conversion from the perspective of Yarn - difficulties/ease, time involved, costs, benefits, etc.

@bondz

This comment has been minimized.

Copy link

bondz commented Jan 24, 2019

What will the installation story of yarn look like?

If there are different plugins and yarn targeting platforms outside of Node.js.

@erikyuzwa

This comment has been minimized.

Copy link

erikyuzwa commented Jan 24, 2019

totally interested to see where this goes in terms of deployments without running yarn install on production servers...thanks for setting this topic up @arcanis

@jcreamer898

This comment has been minimized.

Copy link

jcreamer898 commented Jan 24, 2019

Yarn will be rewritten in TypeScript

🎉

@soederpop

This comment has been minimized.

Copy link

soederpop commented Jan 25, 2019

This is great.

Yarn will become an API, and its internal components will be split into modular entities

This is by far the best thing IMO especially combined with plugins.

Yarn workspaces transformed how I work with javascript. With an API and the ability to write plugins, I can imagine workspaces being made a lot smarter.

Here is a common problem I run into, and maybe there is a configuration based approach to fixing it but I've always relied on custom scripts

For example, a in workspaces project layout like

├── apps
│   ├── webapp-1/package.json
│   ├── webapp-2/package.json
├── components
│   ├── table/package.json
│   ├── form/package.json
│   ├── field/package.json
├── layouts
│   ├── collapsible-side-menu/package.json
│   ├── long-scrollable/package.json
├── themes
│   ├── light/package.json
│   ├── dark/package.json

If multiple teams or developers are working on this, there are some who are constantly touching layouts, and components, others who primarily work on themes, and others who build apps using these dependencies.

yarn workspace's behavior where it symlinks to the local project instead of the npm dependency is great, however it makes it so if I clone this repo from scratch and want to work on an app, I have to build all the themes, layouts, etc as well

i've worked around this with custom scripts, but they're hard to maintain, feel wrong, and involve a lot of shelling out to yarn to do work.

i think plugins + an api for yarn would make this so much cleaner.

Keep up the great work. This is one of the most exciting roadmaps i've seen in a while for a project.

@3stacks

This comment has been minimized.

Copy link

3stacks commented Jan 25, 2019

Why yaml for the lockfile? Presumably, the v1.x json lockfiles won't be compatible with the v2 yaml ones. Doesn't it make sense for it to be json for the native node support?

Maybe someone can explain the benefit, cause I'm not seeing it.

lockfile just jumped the 🦈

@htoooth

This comment has been minimized.

Copy link

htoooth commented Jan 25, 2019

i hope yarn will add yarn tasks , it can list all tasks in package.json scripts.

@taitung

This comment has been minimized.

Copy link

taitung commented Jan 25, 2019

Just a quick thought
echo Developing Javascript projects should be easy
Setting up test is a bit tedious regardless of it is jest or mocha...
Any ideas or thoughts incorporating test (should be the unit test) but allow flexibility also.

@asn007

This comment has been minimized.

Copy link

asn007 commented Jan 25, 2019

@htoooth if I recall correctly, invoking yarn run without additional arguments serves this purpose

@xiaoxiangmoe

This comment has been minimized.

Copy link

xiaoxiangmoe commented Jan 25, 2019

Yarn will be designed to support different install targets than Node

Will yarn support deno soon?

@GonchuB

This comment has been minimized.

Copy link

GonchuB commented Jan 26, 2019

Love the out of the box improvements for monorepos ♥️

@rally25rs rally25rs added next-major and removed triaged labels Jan 26, 2019

@purpledrgn

This comment has been minimized.

Copy link

purpledrgn commented Jan 28, 2019

One of the biggest flaws with the way node_modules works is that modules like fs and various network native libraries (eg. https) are just open. Problem highlighted by drama in the community last year.

It would be nice if yarn could tackle this and have a "strict" mode that's very security conscious (eg. bubbles to the surface all modules that require disk, network and binaries for strict approval process). Understandably this can be tedious, but personally would find that acceptable.

I understand this potentially can be tackled with your plugin system, but I'm not a fan of something this sensitive (and with this many potential bells and whistles) be done under plugin limitations and as an after thought, rather then be a main line feature (just as a lockfile is).

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 28, 2019

I think the ecosystem will eventually get there. However, to be truly effective, such a system would have to be implemented in Node itself imo - or at least some extra primitives would have to be exposed, such as an actual sandbox (vm isn't secure).

So not right now, but I know that there are efforts underway to make it happen, and from Yarn's perspective we would be open to integrate with them (for example by prompting for permissions at install time, etc).

@karlhorky

This comment was marked as off-topic.

Copy link

karlhorky commented Jan 28, 2019

I know that there are efforts underway to make it happen

Oh wow, do you have links for this @arcanis? Sounds really cool.

Edit: Maybe some of the efforts mentioned are nodejs/node#24908 or nodejs/security-wg#327?

@purpledrgn

This comment was marked as off-topic.

Copy link

purpledrgn commented Jan 28, 2019

@arcanis I'm also not sure if there is a process yet.

From @Trott nodejs/node#24908 (comment)

Node.js does surprisingly little with package.json. Most of the magic with package.json is in npm. There may be resistance to the idea that Node.js should parse and understand more package.json than it already does. A lot of feature requests come in the form of suggesting some package.json configuration and they are (as far as I've seen, anyway) always rejected.

I sincerely hope this doesn't end up in everyone waiting for everyone else.

I've managed to find issues and pull requests of nodejs devs trying out sandbox strategies but while sandboxing is tangentially related, it's functionally somewhat irrelevant (especially for things such as say network requests). Similarly, file system sandboxing may be considered safe with regard to nodejs runtime, but if we're thinking of it as "dependency rights" it doesn't matter if it technically it should-be-secure, from the perspective of an end user it's absolutely nonsensical to be there if it's unneeded. Not to mention that you may have cases where full system access is needed (I would imagine most of them).

To give an example, say there's a dependency that advertises itself as merging two specialized objects but requires file system access (sandboxed, for the sake of argument). Does file system sandboxing make a difference here? Because it doesn't to me, this requirement is too suspicious by itself.

I agree with you that nodejs features play a big part, but I think the registries and package managers are the ones who need to make the push forward and define at least some of what's required from a dependency tree stand point. Going by the comment by @Trott the nodejs team doesn't want to touch dependency tree problems.

azu added a commit to jser/jser.github.io that referenced this issue Jan 29, 2019

2019-01-29のJS: Node.js 11.8.0(Policy機能)、Nuxt.js 2.4.0(TSサポート)、Yarn v2…
…のロードマップ (#582)

* [Node v11.8.0 (Current) | Node.js](https://nodejs.org/en/blog/release/v11.8.0/ "Node v11.8.0 (Current) | Node.js")
* [Yarn's Future - v2 and beyond · Issue #6953 · yarnpkg/yarn](yarnpkg/yarn#6953 "Yarn's Future - v2 and beyond · Issue #6953 · yarnpkg/yarn")
* [Release v2.4.0 · nuxt/nuxt.js](https://github.com/nuxt/nuxt.js/releases/tag/v2.4.0 "Release v2.4.0 · nuxt/nuxt.js")
* [nuxt.js/RELEASE_PLAN.md at dev · nuxt/nuxt.js](https://github.com/nuxt/nuxt.js/blob/dev/RELEASE_PLAN.md "nuxt.js/RELEASE_PLAN.md at dev · nuxt/nuxt.js")
* [Nuxt.js and TypeScript as of Jan 2019 – Hai-iro – Medium](https://medium.com/haiiro-io/nuxt-js-and-typescript-as-of-jan-2019-4d8674a91815 "Nuxt.js and TypeScript as of Jan 2019 – Hai-iro – Medium")
@ghost

This comment was marked as off-topic.

Copy link

ghost commented Jan 29, 2019

Still missing something like npm has audit. there is so many insecure modules around

@lirantal

This comment was marked as off-topic.

Copy link
Contributor

lirantal commented Jan 29, 2019

It would be nice if yarn could tackle this and have a "strict" mode that's very security conscious (eg. bubbles to the surface all modules that require disk, network and binaries for strict approval process). Understandably this can be tedious, but personally would find that acceptable.

@purpledrgn not sure if this is something that a package manager would tackle or closer to Node.js core. Keep in mind that this concern is not only for when you install a package but also for runtime. This topic has been raised a few times already in the issue queue and we have a PR opened for Node.js core to implement access control: nodejs/node#22112

Still missing something like npm has audit. there is so many insecure modules around

@vyteniskuciauskas-home24 checking your dependencies at install time is only a small part. There are more concerns around open source dependencies such as monitoring project deps after they've been deployed, and providing actionable remediation, etc. full disclosure, I work at snyk.io, so I can go on and on about all the benefits that you'd be missing out on :-)

@purpledrgn

This comment was marked as off-topic.

Copy link

purpledrgn commented Jan 29, 2019

@lirantal

@purpledrgn not sure if this is something that a package manager would tackle or closer to Node.js core. Keep in mind that this concern is not only for when you install a package but also for runtime. This topic has been raised a few times already in the issue queue and we have a PR opened for Node.js core to implement access control: nodejs/node#22112

Can you elaborate? I've seen that before. Am I mistaken when I think it's simply a worker thread related feature?

I can see how this is similar to ACLs (which seems to be where it's clout is coming from) but I feel that's a very shaky road to go on: a lot of (potentially more verbose) complexity to get to the same result of telling packages what they can and can not do, with all the same problem areas (eg. a malicious express package would still have access to do malicious things). If the package managers forbid it via static analysis (ie. does it import 'fs' etc) or it's blocked at runtime, whats the difference aside from runtime variant being more costly?

For the idea in the issue to work (even assuming it can work with out separate threads) you would still have to have something akin to pseudo-runtimes for each dependency. And even then it still wouldn't solve any additional problems static analysis approach doesn't solve (assuming no flaws; which both impl can have)

The intent problem is only potentially solved via antivirus like pattern matching search; at least I don't see any other solution having come up in other areas with similar problems.

Personally I feel, given that a lot of packages people use actually do not need network or file access, a split between packages that require filesystem/network and everything else would solve a lot of issues. If you are a potential malware (due to your requirements) have your code signed-off by a trusted third party (or several). It would be interesting what the split is (among very popular packages) on packages that fall in either bucket. A quick peek at some of my own package.json files: database/cache drivers, 3rd party auth systems like auth0, http servers, network request libraries (eg. fetch implementations), some niche utilities related to ips, dev build deps that potentially have compiled or binary bits. Overall not a lot, and most from large groups, some backed or sponsored by large companies. At a glance I'd say no little to lose from a split and more rigorous "trust" process.

@arcanis

This comment was marked as off-topic.

Copy link
Member Author

arcanis commented Jan 29, 2019

Please open a separate thread for focused, more-than-one-answer-expected, discussions 🙏

@adrian-bobev

This comment has been minimized.

Copy link

adrian-bobev commented Jan 30, 2019

Question

Do you have any plans for yarn publish support for mono repos with workspaces?

Currently Lerna should be used to update all the packages versions and update dependencies. It would be great if yarn takes care of this too

@arcanis

This comment has been minimized.

Copy link
Member Author

arcanis commented Jan 30, 2019

Yes and no. It's likely the publish workflow will be kept relatively similar to what currently exists now, but we'll be looking forward to see different implementations of it (through plugins). If an implementation becomes clearly superior, I could see us moving it into the core.

@samsieber

This comment has been minimized.

Copy link

samsieber commented Feb 6, 2019

Does this issue supersede #5773 (🦁 Yarn 2.0 Working Group ) - if so can we close it?

@arcanis arcanis pinned this issue Feb 7, 2019

@arcanis arcanis referenced this issue Feb 7, 2019

Closed

🦁 Yarn 2.0 Working Group #5773

0 of 5 tasks complete

@yarnpkg yarnpkg deleted a comment from 0946752671 Feb 7, 2019

@grantila

This comment has been minimized.

Copy link

grantila commented Feb 12, 2019

Fantastic @arcanis, I tried to replace the https?: fetching to my own fetch-h2 (which handles HTTP/1(.1) and HTTP/2 transparently) to see if it'd be faster to use HTTP/2 (most public repositories handles it). This was surely painful, due to the implicit dependency of request, and its arguments being passed around the yarn codebase.

A plugin structure sounds great. I'd be happy to make a yarn-plugin-http2 (and if it proves to be fast, hand it over to you guys if you want it in the core).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment