Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yarn2nix #20637

Open
moretea opened this Issue Nov 22, 2016 · 28 comments

Comments

@moretea
Copy link
Contributor

moretea commented Nov 22, 2016

Facebook recently published yarn, a package manager that behaves like Bundler and Cargo by having a lock file that describes the exact versions of all transitive dependencies.

I investigated the yarn.lock format, and was pleasantly surprised that it actually contains both direct download links to (and SHA1 hashes of) the .tgz files that make up packages.

I'm very much open to merge this with @svanderburg's node2nix, but I'm not sure that that's appropriate.

Progress

@offlinehacker

This comment has been minimized.

Copy link
Contributor

offlinehacker commented Dec 11, 2016

Nice :)

@offlinehacker

This comment has been minimized.

Copy link
Contributor

offlinehacker commented Dec 11, 2016

What happens if package.json refrences git/github package?

@phanimahesh

This comment has been minimized.

Copy link
Contributor

phanimahesh commented Dec 15, 2016

Would you mind if yarn2nix was a go/rust binary instead of a ruby script? Or a nodejs script, since we are anyway working with the node ecosystem. Having to pull in ruby to work on node just doesn't feel good. I can volunteer for part of the effort. Nice excuse to try out something new too, IMO.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Dec 15, 2016

I think it would be more maintainable to use yarn's parser directly. Here I made a small program that turns the yarn.lock into JSON: https://github.com/numtide/yarnlock2json . The URL still has to be split to extract the sha from the fragment.

@wmertens

This comment has been minimized.

Copy link
Contributor

wmertens commented Dec 16, 2016

@moretea do you need help with this? This finally seems like a long-term viable approach… Use Yarn to get a fixed set of deps, create the deps, and let yarn do the node_modules build.

I agree with @zimbatm that we should use yarn's own parser. I had a look at the yarn2nix and it seems very simple, so perhaps it can be built directly on top off zimbatm's script?

Also, perhaps each yarn package should be a separate nix expression, so that they get reused between projects? I think it would mean walking the dep tree an building each dep in turn, creating packages as you go, and linking them into the .cache on next invocations.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Dec 16, 2016

Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there. Another solution would be to work with yarn to see if we can pass them the list of source derivations.

Regarding the node_modules folder, it's not practical to use symlinks in there unfortunately. If you search for it you'll see that others are trying to tackle the issue but in short it's due to two things: (1) some modules actually rely on the node_modules structure by doing things like require("../other_module"). (2) node resolves module paths to their absolute path at import. So now you have a module at /nix/store/...-my-module trying to load ../other_module and it breaks. What might work though is have the whole node_modules stored in a derivation so it only gets re-built when the package.json or yarn.lock file changes.

@wmertens

This comment has been minimized.

Copy link
Contributor

wmertens commented Dec 16, 2016

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Dec 16, 2016

@moretea

This comment has been minimized.

Copy link
Contributor Author

moretea commented Dec 16, 2016

@phanimahesh Would you mind if yarn2nix was a go/rust binary instead of a ruby script?

Not at all. It's just a prototype at the moment and I just like to use Ruby to prototype these kinds of things.

@zimbatm I think it would be more maintainable to use yarn's parser directly.

Awesome. I actually know little about javascript; I just want to package some stuff.

@zimbatm Ideally we want the source to be fetched by nix but it's not clear to me how to plug that into yarn yet. Maybe we can reproduce yarn's file layout and ignore yarn install but I suspect there are some hidden complexities there.

@wmertens I was thinking that we can reproduce the cache dir layout. After all, we
have the full source available.

I tried to do exactly that, but yarn still insists on writing random files in that cache directory. When I tried to fill it with the correct packages, it still failed because of missing packages.

@wmertens So the node_modules folder is a bunch of copies, but when that built
application is stored in the nix store, it can be deduped if the optimize
option is turned on. So that's not a problem.

It's a bit more subtile than that. The directory structure of node_modules is a bit weird; npm/yarn support having multiple versions of different packages in there. This in itself is not a weird thing to do; Cargo supports the same features.
The problem is that the directory names do not have a version number. Having multiple dependency versions is handled by having another node_modules directory in a $ROOT_DIR/node_modules/some_package/ directory. Unfortunately, I could not find any documentation on the exact logic of how these decisions are made.

@zimbatm For now my best idea is that $lang provides a lockfile that contains the
dependencies, where to fetch then including the SHA. $lang developer
shouldn't have to commit the generated nix code. Nix knows how to read that
lockfile and prefetch the dependencies from that. After that it's a bit
fuzzy and probably varies. Ideally we can build each dependency in
isolation but it's not always possible.

Fully agreed. Luckily a yarn.lock file contains all that!


I would very much appreciate any help with this project ;)

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Dec 17, 2016

@moretea

This comment has been minimized.

Copy link
Contributor Author

moretea commented Dec 17, 2016

I started to patch node2nix to support importing a yarn.lock file in svanderburg/node2nix#28.

It was actually a lot less work than I anticipated.

I'm closing this issue and proposing that we move the discussion to that PR.

@moretea moretea closed this Dec 17, 2016

@wmertens

This comment has been minimized.

Copy link
Contributor

wmertens commented Dec 17, 2016

@moretea

This comment has been minimized.

Copy link
Contributor Author

moretea commented Dec 17, 2016

Maybe so, but my painpoints were in getting yarn to use the files downloaded with fetchurl, which are all solved by node2nix already. It is something we can explore in parallel to only depending on yarn.

@phanimahesh

This comment has been minimized.

Copy link
Contributor

phanimahesh commented Dec 18, 2016

Agreed. Yarn is a bunch of good ideas, implemented extremely poorly. We shouldn't trust yarn to do things right. If it can be convinced to use pre-downloaded files, that's what we should be doing. If it isn't supported, we should request/contribute that.

@moretea

This comment has been minimized.

Copy link
Contributor Author

moretea commented Jan 16, 2017

I've rewritten yarn2nix. It is able to build packages with dependencies that are in the default registry properly.

I'm abusing the offline cache feature of yarn by generating a directory with $package-$version.tgz files downloaded via Nix.

@zimbatm @offlinehacker @wmertens @phanimahesh could you take a look? I gathered that you are packaging node apps.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Jan 16, 2017

Nice! Looking good

@moretea moretea reopened this Jan 16, 2017

@offlinehacker

This comment has been minimized.

Copy link
Contributor

offlinehacker commented Jan 16, 2017

Nice, I really like the solution as it's simple, now all we need is buildYarnPackage function in nixpkgs :)

@roblabla roblabla referenced this issue Jul 5, 2017

Merged

gitlab: 8.17.6 -> 9.4.5 #27159

0 of 8 tasks complete
@ixmatus

This comment has been minimized.

Copy link
Contributor

ixmatus commented Aug 25, 2017

Integrating yarn2nix into nixpkgs doesn't look all that difficult; I've already integrated it into our own package overrides. Does someone already have work in a branch to do this? If not, I'd love it if we could get a PR open to contribute yarn2nix and a separate buildYarnPackage function...

[EDIT] I'm happy to help :)

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Aug 25, 2017

@ixmatus it's not that hard, the main reason not to merge was to be able to experiment without polluting the mainline. the project is in a pretty good state but still needs a bit of polish.

@flokli

This comment has been minimized.

Copy link
Contributor

flokli commented Feb 6, 2018

@zimbatm Whats the current state of this? moretea/yarn2nix seems to have received some commits so far, but there also is Profpatsch/yarn2nix already coming in from hackage-packages

@moretea

This comment has been minimized.

Copy link
Contributor Author

moretea commented Feb 6, 2018

@flokli @Profpatsch's approach looks way better than the hacks that we do in this repository. If I would need to package a yarn thing again, I would probably take a look at his solution first.

If you can use @Profpatsch's version, I'd go for that :)

@Profpatsch

This comment has been minimized.

Copy link
Member

Profpatsch commented Feb 6, 2018

Please write issues if it takes you too long to get https://github.com/Profpatsch/yarn2nix running.
I’m using it for a few of my projects but haven’t yet written down how to use the nix library that comes with the repository.

I’m more than happy to help with packaging, to get feedback on whether the interface is any good and whether it actually works for all/most use cases. Sadly, the whole npm ecosystem is a bit of a mess and there’s lots and lots of edge cases everywhere you look.

@marsam

This comment has been minimized.

Copy link
Contributor

marsam commented Mar 10, 2018

With #35340 merged, I think we can close this issue

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Mar 11, 2018

Unfortunately yarn2nix depends on Import From Derivation (IDF) to work, and it's not enabled on ofborg and hydra. yarn2nix can be used from nixpkgs but other packages in nixpkgs can't use it.

See #35334

@zimbatm zimbatm self-assigned this Mar 11, 2018

@domenkozar

This comment has been minimized.

Copy link
Member

domenkozar commented Jul 2, 2018

I wonder if we could add yarn2nix stuff to nixpkgs but disallow hydra builds? As a stepping stone.

@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Jul 4, 2018

Can't yarn2nix be modified to generate its .nix expressions like node2nix, emacs2nix & cabal2nix? That is much preferred than disabling hydra. I think that is the right way to do things in Nixpkgs & it has worked very well for cabal2nix at least.

@domenkozar

This comment has been minimized.

Copy link
Member

domenkozar commented Feb 25, 2019

I think we should remove yarn2nix from nixpkgs, as it's quite outdated (or merge all the upstream changes to nixpkgs).

What do you think @zimbatm @moretea

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 25, 2019

agreed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.