Concerns over .lockb format

[thislooksfun]

Hello!

While this project is looking incredible, I have some concerns about making the lock file binary, specifically when it comes to version control (git). While a binary format is fine for small single-user projects, and undeniably is better for speed, it has the potential to make dealing with PRs even harder then it already is. The nice thing about a text-based format is that two PRs can both modify the packages (add/update/remove/etc) while usually not conflicting with each other. However, since git treats binary files as either "exactly the same" or "oh no one bit is different what do I do", there isn't a way for it to resolve any of these situations.

The question is, what do we do about this? On the one hand, the performance is great, but on the other it makes workflows unnecessarily more restricted. At the very least this should be strongly considered so as to avoid unnecessary friction. I really want this project to succeed, but having to rebase every PR merge that changes packages would be a significant headache (and thus likely a blocker) for many potential adopters ranging from switching to bun (when it's eventually stable).

The only 3 options I can see are:

1. Keep the file as a binary format for maximum performance, but with the significant caveat of not being able to easily merge multiple package tree changes
2. Use a more traditional text-based approach (.json, .yml, etc), which would have a performance hit, but would be easily resolvable by git
3. Some novel hybrid approach. Have package data separated by newlines (\n), but each line is minified/encoded in some way. Just because you are restricted to ASCII/UTF-8 characters doesn't mean they HAVE to be human readable.

I personally am leaning towards option 3. I don't care about human readability of the on-disk lock file as long as there is a way to get a human readable format (which there is, with bun install -y), just that git is mostly able to merge multiple changes without human intervention. Extra kudos if you can organize the structure in a way that minimizes/eliminates the change of conflicts that don't modify the exact same package.

[moderation]

@Jarred-Sumner tweeted about the bun pm tool for working with the binary lockfile at https://twitter.com/jarredsumner/status/1545222563877466112

[thislooksfun]

Again, I'm not worried about introspection, I want to be able to easily merge multiple PRs that both update the dependencies without needing manual intervention.

[moderation]

Makes sense. There was this comment from Jarred on Hacker News - https://news.ycombinator.com/item?id=31994498

One piece of feedback, having the lockfile be binary is a HUGE turn off for me. Impossible to diff. Is there another format?

If you do bun install -y, it will output as a yarn v1 lockfile.

If you add this to your .gitattributes:

*.lockb binary diff=lockb

It will print the diff as a yarn lockfile.

[Jarred-Sumner]

Bun needs to provide a solution to this, but I'm not sure yet what

Maybe experimenting with a text-based format would be worth it.

[infodusha]

Maybe we can have a util that just prints out lockb in readable way if needed?
I imagine pipeline can make a diff of output for the command, for example

[ghiscoding]

you're probably looking for this: https://github.com/JacksonKearl/bun-lockb

[infodusha]

I mean same but as an official cli tool but the thing above is close to the idea

[landsman]

It should be in a readable format by default. Not require the whole world to change itself.

[landsman]

Last week I was trying to open this "lock file" with editors on my laptop. I was not successful. That's scary guys!

Please consider to use JSON or other standardized format 🙏
Proprietary ways for problems such as this are not great for healthy growth.

I really love Bun's idea, solution but this is real blocker for me.

[bradchristensen]

I may be in a minority here, but I used to manually edit my package-lock.json or yarn.lock files quite frequently to resolve particular versions of transitive dependencies or to deduplicate package versions. In fairness, in a perfect world, I wouldn't have to do this if the package managers could read my mind and always do the right thing 😄

Unfortunately I've discovered that Bun has some issues with resolving transitive dependencies at the moment (like #5022). I can see the diff (thanks for the good instructions and tools to be able to do that!), and I would just be able to resolve the duplicates myself if I could manually edit the lockfile. As it stands, the only solution I think I have is to just delete it and regenerate it from scratch - in this case it will pull down all the latest dependency versions, which successfully deduplicates them, but I don't necessarily want the latest versions of everything.

tl;dr: An editable version would be very useful, even if it was a power-user/hidden away feature. You may have better ideas around this but one suggestion that comes to mind is: could I generate a yarn.lock file with bun install --yarn, delete the bun lockfile, modify the yarn lockfile and run bun install to generate the bun lockfile based on the yarn lockfile?

[rvramesh]

I am curious about the ability to audit changes when reviewing a PR merge from github UI. Can a malicious person inject a different lock file compared to corresponding package.json?

[landsman]

I think so. You are not able to review changes via GitHub UI, so you are blind...