Generate text lockfile along with binary lockfile (bun.lockb) and add "bun.lockb" to .gitignore

[sudo-barun]

What is the problem this feature would solve?

Bun generates binary lockfile (bun.lockb) by default and is not added to .gitignore file. This means bun.lockb mostly gets added to git repository (for example, https://github.com/elysiajs/elysia has bun.lockb).

The problem arises when merging branches that have changes in bun.lockb file. Since, bun.lockb file cannot be inspected, developer will most likely remove the bun.lockb file and generate a new one. This can be disastrous as the new locked version of a package is different from the previous one.

What is the feature you are proposing to solve the problem?

• Bun should generate text lockfile along with binary lockfile by default.
• Bun should generate binary lockfile based on text lockfile (not the other way around).
• Bun should add bun.lockb to .gitignore file by default to prevent merge conflict. Developer can force add bun.lockb to git repo if needed.

What alternatives have you considered?

No response

[praashie]

For now, see the documentation on how to view lockfile diffs in Git: https://bun.sh/docs/install/lockfile

The main (and only?) benefit of the binary lockfile is performance, and having textual lockfiles as the source of truth requires Bun to quickly determine whether the two are in sync. This could be done by comparing file timestamps, Make-style.

Binary lockfiles will definitely be a headache when viewing pull requests in browsers. Even when using text lockfiles, I think merge conflicts should always be resolved with the package manager, not Git's auto-merge.

[sudo-barun]

@praashie

For now, see the documentation on how to view lockfile diffs in Git: https://bun.sh/docs/install/lockfile

Viewing binary lockfile diffs as text is helpful. But, it is not going to solve the problem that I mentioned i.e. how to deal with merge conflict in bun.lockb without losing version information.

having textual lockfiles as the source of truth requires Bun to quickly determine whether the two are in sync

NPM does not check whether package-lock.json and package.json are in sync when running npm install. It only checks that when running npm ci. Bun can follow the same strategy and not check whether the text lockfile and binary lockfile are in sync when running bun install and have different command for that.

Binary lockfiles will definitely be a headache when viewing pull requests in browsers. Even when using text lockfiles, I think merge conflicts should always be resolved with the package manager, not Git's auto-merge.

Merge conflicts is always a headache whether it is in binary lockfile (like bun.lockb) or text lockfile (like package-lock.json). Browser sometimes even does not show diff in text lockfile when there is huge changes. Package manager can provide a command to determine synchronization issue like npm ci --dry-run.

By having both text lockfile and binary lockfile, developer can fix merge conflict in text lockfile and generate a new binary lockfile from the text lockfile.

[satya164]

Merge conflicts is always a headache whether it is in binary lockfile (like bun.lockb) or text lockfile (like package-lock.json).

It's actually pretty good with Yarn, add long as package.json conflicts are fixed, just running yarn again automatically fixes merge conflicts in the yarn.lock

[sudo-barun]

@satya164
This is about Bun, not Yarn.

[satya164]

@sudo-barun I brought up Yarn because you are comparing with npm's lockfile format. The point is that the package manager could handle the lockfile conflicts automatically as Yarn does and it's not always a headache.

[sudo-barun]

@satya164
I don't see auto merge fixing feature being documented in Yarn website. So, may be what you are describing is destructive approach where the content of yarn.lock is regenerated from package.json and version of packages may get updated. This may fix merge conflict but can sometime lead to bug due to new version of packages.

And, if this auto merge fixing feature really exists in Yarn, it would great to have such feature in Bun or NPM as well.

[satya164]

may be what you are describing is destructive approach where the content of yarn.lock is regenerated from package.json and version of packages may get updated.

If you have other syntax errors in yarn.lock then yarn will fail, so it's explicitly handling merge conflict scenario. You can find many references to automerge feature in this page: https://yarnpkg.com/advanced/error-codes#yn0046---automerge_failed_to_parse

[sudo-barun]

@satya164
Thanks for the link.

[intrnl]

realized there's an existing issue already, and I can agree, they're not very great to work with.

being able to inspect and diff lockfiles easily is a very important functionality of lockfiles, it gives you the insight to know that your package manager is doing things right, and Bun's binary lockfiles just stands against all that. I'd like to go more but it's really just going to be a dupe of what I've said in #6276 (comment)

ideally I'd prefer having an option to make use of text lockfiles entirely and NOT generate binary lockfiles at all, why keep two lockfiles with one of them ignored by Git? makes no sense, it'd be better to only have one source of truth

[Electroid]

We are unlikely to change Bun to generate two lockfiles, but we definitely know that not seeing bun.lockb in git diff is very annoying and not a great experience.

If you use our VSCode extension, it will automatically show you the yarn-compatible output of the bun.lockb when you view it. We have also considered switching to a plain text lockfile, but that is a little more challenging, since we do not want to compromise performance.

We haven't decided on anything right now, just sharing some thoughts.

[LandonSchropp]

For what it's worth, I just ran into this exact issue with a merge conflict in my bun.lockb file. Like @satya164 mentioned, I'm used to Yarn, where I can easily run yarn to automatically fix merge conflicts in my yarn.lock file. My expected behavior for Bun was the same, and I was a bit surprised when it didn't work.

[marcellodesales]

Just had the same though here... although We've all used text for comparison, I'm kinda open to experiment with the fact that human-related changes are specified at the package.json level, while the .lock files were always maintained by machines... I will keep using this until I'm not convinced it could be in binary...

[Thinkscape]

So the diff approach has failed spectacularly for me. After upgrading to bun 1.0.22 I am now unable to diff my lockfiles because it fails with:

Bun exited with code: 1
error parsing lockfile: InvalidLockfile

I suspect bun changed the format of the lockfile and is not BC. Now I can't easily diff and investigate what lib version upgrades broke my app. If we can't rely on BC, then it'll be a major pain to work with git whenever the binary format changes.

[Jarred-Sumner]

@Thinkscape bun's lockfile is always forwards compatible, but not always backwards compatible

are you unable to view the lockfile when using bun v1.0.22? or is an earlier version of bun reading the lockfile?

[Lukasz-pluszczewski]

but we definitely know that not seeing bun.lockb in git diff is very annoying and not a great experience.

see the documentation on how to view lockfile diffs in Git

they're not very great to work with.

There are ways to view the contents of the file. That's not the point. The point is, there is nothing you can do if you have lockfile conflicts. You have to either generate a new lockfile (which is, as mentioned before, disastrous and makes lockfile useless in the first place), or accept one of the conflicted branches (and lock versions from other branch with next install, which can be disastrous as well).
Bun is great but this issue is not "annoying". It makes bun install unusable at all if you're using version control, and practically all projects use version control.

The suggested solutions (either changing the format to a textfile or generating two lockfiles - one for version control and the binary one for speed) would solve the issue it seems. I'm fine with any other solution to be honest, as long as I can use version control with lockfile.

[LandonSchropp]

Just my personal opinion, but any performance improvements from a binary lockfile aren't worth the trouble of dealing with one. I'd rather have text diffs and easy conflict resolution.

[Thinkscape]

@Thinkscape bun's lockfile is always forwards compatible, but not always backwards compatible

are you unable to view the lockfile when using bun v1.0.22? or is an earlier version of bun reading the lockfile?

It was an upgrade from 1.0.1* (with lockfile generated by that one), to 1.0.22 (which generated a new one that I staged).

I got around it by diffing manually, so it seems that the bug is somewhere in the bun vscode extension, or its interaction with newer bun, or something in between 🤔

[intrnl]

After the incident around xz and checking in binary files in source code, I hope we can reconsider the whole binary lockfiles thing 🙂

Binary files should be considered untrustworthy and should not be checked into source code, especially binary files that we don't author ourselves (we don't generate lockfiles, package managers does that for us.)

Bun is even more peculiar as it's the one lockfile format that allows being executed given the presence of a shebang on the start of the lockfile. Sure, it's just a shebang that tells the shell to open the file with bun, but in any case, the fact that you can do it at all is shady.

Additionally, we should not have to be reliant on a GitHub bot to show us a diff during pull requests.

GitHub is not the only place where source code development happens, and having to set up a CI for something that should be trivial seems like a waste of compute and electricity.

[John-Colvin]

I am now running in to conflicts all the time & the only practical resolution is to trash the lock file and reinstall, which somewhat defeats the point.

Unless I'm misunderstanding something, the current situation is very fast but breaks the overwhelmingly most common version control workflows on the planet.

[kumikumi]

The more I spend time with this, the more convinced I am that there is no justification for having a binary lockfile at all. Many orders of magnitude more time gets spent trying to work around the decision of having a binary lockfile (resolving conflicts, coming up with clever ways to show diffs in some tools, etc.) than is saved during install by using a binary lockfile instead of a text based one.

I love the general obsession with performance in this project, but this ain't it Chief.

Edit: To turn this rant into a workable suggestion, allow setting a text based lockfile format in bunfig.toml in a way that only that format gets used, as the single source of truth.

[yus-ham]

I haven't research further, git can be configured to use different diff tool for certain extension, and but this also adds extra deps

https://stackoverflow.com/questions/34885397/using-custom-diff-tool-with-git-show

bun.lockb questions in bun discord: https://discord.com/channels/876711213126520882/1244811479203450911/1244813213250621484

[Jarred-Sumner]

We are planning to replace bun.lockb in a couple months. Tracking in #11863