npm publish from Windows handles CRLF incorrectly

[DukeyToo]

When published from a Windows machine, the package files are uploaded with the windows-style CRLF endings. This is fine for most files, but if there are any shell scripts or a specified bin file, then those files will be corrupt and fail on a Linux machine.

I encountered this behavior publishing some of my packages from Windows from a git repository, using e.g.

$ npm publish git://github.com/myuser/mypackage.git

I understand why this occurs, but since this makes npm publish 50% unusable on Windows, I think it is fair to call it a bug in npm.

[mihaifm]

+1 for this issue, something similar to git's core.autocrlf option needs to be implemented.

[moos]

+1
Just ran into this publishing from Windows. Script installs/runs file on Windows cmd and git-bash. But on Ubuntu VM, kept getting a cryptic ": No such file or directory". Running fromdos on script fixed it.

Hopefully this can be addressed soon.

[domenic]

Oh eww, I didn't know I was doing this to people :(

[isaacs]

Hm, yeah, that's a pita.

It'd be nice for npm to do this automatically when generating the tarball, but I'm not sure how without potentially breaking binary files. There's got to be some kind of way to detect if it's a "text" file, right? Git has this, after all.

[heycalmdown]

+1

Oh it can just be reached easy by implement a function to figure out the first line of the binary which defined in the "bin" field of package.json that starts with #!

[isaacs]

@heycalmdown Well, ideally npm should convert \r\n to \n in ALL non-binary files when creating the tarball on Windows.

[medikoo]

@isaacs maybe ideally it should warn and exit, but optionally with eg --fixeol flag it should do automatic conversion to \n ? Some may be upset that npm just converts new lines without a note.

[ForbesLindesay]

From an implementation point of view:

Ideally the change should not impact the files on disk. It probably needs to happen as part of the packing that goes into ./lib/utils/tar.js but this doesn't seem to have any easy way to perform such a mutation.

As I understand it, https://github.com/isaacs/fstream-npm just emits file name events that tar then packages. Since tar is a very generic utility, we can't have tar do the packaging.

What if tar was to just receive objects containing filename and content (as a stream) as its input?

[jednano]

I have a work-around for this issue that has been verified to work.

@isaacs, I want to point out that no, it's not a good idea to replace all non-binary file line endings with \n. In my case, in CodePainter, I very intentionally left some line endings as \r\n and \r for my tests. These line endings need to be preserved.

[tohagan]

+1 I've hit this one several times. Tonight it got nastier as I'd publish a lib containing a package.json file that had CRLF's in it (unknown to me). I then attempted to publish another dependent package but npm refused to publish it because the lib it depended on contained an unparsable package.json. Really ugly stuff! Cleaning up CRLFs would prevent this.

I now tend to run dos2unix on most of my files prior to publishing just to be sure. This is really getting painful.

[tohagan]

I think you could make it either an npm command line option or perhaps in package.json define a list of file patterns to convert (or not convert).

[domenic]

My solution has been to do all of my development with LF and ban CRLF files from my source code folder.

[tohagan]

Yes I've done the same - but still hitting issues. Enforced with .gitattribute files also. However I use several IDEs/Editors and I think one of them is to blame.

[tohagan]

But getting npm to help would prevent most of these problems.

[tohagan]

Thanks!

[pflannery]

+1

I just ran in to this...turns out Github for Windows is overriding my git line ending preferences and applying CRLF in all my clones. Worse still Github for Windows doesn't let you choose your line ending preferences so forced back to cloning from the console :'(

[jednano]

@pflannery actually it does let you choose. Just go into the repository in question and run the following:

git config core.autocrlf false
git rm --cached -r .
git reset --hard

The first command tells Git to never change line endings (in the future). Next, we refresh each repository by removing every file from Git's index and, finally, rewriting the Git index to pick up all the new line endings. This fixes the CRLFs that were introduced to your local file system when you cloned each repository.

[shenanigans]

I tried this and didn't get anywhere. I also tried it from a linux system and was unable to remove the CRLFs from the remote. I don't disagree that git can do this, just not without giving me a hernia.

[pflannery]

@jedmao but when I git clone from the terminal it's fine, no CRLF. but when I clone from Github for Windows I get CRLF.

[pflannery]

@jedmao looking deeper it seems github uses portable-git and not the git I installed from git-scm.com....I'll try to set the config for portable-git..

[jednano]

Yes, it's important to know that you are using the same Git everywhere. In my experience, the only Git I install is GitHub for Windows. Then, I track down the msysgit installation path by navigating to User\AppData\Local\GitHub\PortableGit_[some-hash]\bin. I copy that path and add it to my user environment PATH variable. This way, any shell I'm in, I know that the git command is always hitting the GitHub for Windows installation. You'll save yourself a lot of grief by uninstalling any and all other versions of Git on your computer. An added bonus is that you get the git credential store for free, so you don't have to punch in your username/pw all the time. Just remember that sometimes GitHub updates create a new PortableGit hash, so if the git command all the sudden stops working you just have to track down that PortableGit bin path and replace the one in your user's PATH environment variable. Clear as mud?

[jednano]

And @pflannery, remember to use the Git Shell. The GitHub for Windows installation should have created a shortcut for you. You can also open a shell from the UI's gear icon.

[othiym23]

Closing this in favor of #12371, which I believe preserves the simplest way to resolve this, as well as a path for those who might be up to sending us a solution before we have the time to do so ourselves. Thanks to all for their time and (especially) patience.

[tivac]

@othiym23 This... is issue #2097 though? 😕

[othiym23]

@tivac ahem updated to point to #12371 correctly. My apologies for the error.