When published from a Windows machine, the package files are uploaded with the windows-style CRLF endings. This is fine for most files, but if there are any shell scripts or a specified bin file, then those files will be corrupt and fail on a Linux machine.
I encountered this behavior publishing some of my packages from Windows from a git repository, using e.g.
$ npm publish git://github.com/myuser/mypackage.git
I understand why this occurs, but since this makes npm publish 50% unusable on Windows, I think it is fair to call it a bug in npm.
+1 for this issue, something similar to git's core.autocrlf option needs to be implemented.
Just ran into this publishing from Windows. Script installs/runs file on Windows cmd and git-bash. But on Ubuntu VM, kept getting a cryptic ": No such file or directory". Running fromdos on script fixed it.
Hopefully this can be addressed soon.
Just ran into this myself.
Oh eww, I didn't know I was doing this to people :(
Hm, yeah, that's a pita.
It'd be nice for npm to do this automatically when generating the tarball, but I'm not sure how without potentially breaking binary files. There's got to be some kind of way to detect if it's a "text" file, right? Git has this, after all.
Oh it can just be reached easy by implement a function to figure out the first line of the binary which defined in the "bin" field of package.json that starts with #!
@heycalmdown Well, ideally npm should convert \r\n to \n in ALL non-binary files when creating the tarball on Windows.
@isaacs maybe ideally it should warn and exit, but optionally with eg --fixeol flag it should do automatic conversion to \n ? Some may be upset that npm just converts new lines without a note.
From an implementation point of view:
Ideally the change should not impact the files on disk. It probably needs to happen as part of the packing that goes into ./lib/utils/tar.js but this doesn't seem to have any easy way to perform such a mutation.
As I understand it, https://github.com/isaacs/fstream-npm just emits file name events that tar then packages. Since tar is a very generic utility, we can't have tar do the packaging.
What if tar was to just receive objects containing filename and content (as a stream) as its input?
I have a work-around for this issue that has been verified to work.
@isaacs, I want to point out that no, it's not a good idea to replace all non-binary file line endings with \n. In my case, in CodePainter, I very intentionally left some line endings as \r\n and \r for my tests. These line endings need to be preserved.
+1 I've hit this one several times. Tonight it got nastier as I'd publish a lib containing a package.json file that had CRLF's in it (unknown to me). I then attempted to publish another dependent package but npm refused to publish it because the lib it depended on contained an unparsable package.json. Really ugly stuff! Cleaning up CRLFs would prevent this.
I now tend to run dos2unix on most of my files prior to publishing just to be sure. This is really getting painful.
I think you could make it either an npm command line option or perhaps in package.json define a list of file patterns to convert (or not convert).
My solution has been to do all of my development with LF and ban CRLF files from my source code folder.
Yes I've done the same - but still hitting issues. Enforced with .gitattribute files also. However I use several IDEs/Editors and I think one of them is to blame.
But getting npm to help would prevent most of these problems.
I'm not sure when or how I got infested with \r but I can think of a great time to get rid of them.
I just ran in to this...turns out Github for Windows is overriding my git line ending preferences and applying CRLF in all my clones. Worse still Github for Windows doesn't let you choose your line ending preferences so forced back to cloning from the console :'(
@pflannery actually it does let you choose. Just go into the repository in question and run the following:
git config core.autocrlf false
git rm --cached -r .
git reset --hard
The first command tells Git to never change line endings (in the future). Next, we refresh each repository by removing every file from Git's index and, finally, rewriting the Git index to pick up all the new line endings. This fixes the CRLFs that were introduced to your local file system when you cloned each repository.
I tried this and didn't get anywhere. I also tried it from a linux system and was unable to remove the CRLFs from the remote. I don't disagree that git can do this, just not without giving me a hernia.
@jedmao but when I git clone from the terminal it's fine, no CRLF. but when I clone from Github for Windows I get CRLF.
@jedmao looking deeper it seems github uses portable-git and not the git I installed from git-scm.com....I'll try to set the config for portable-git..
Yes, it's important to know that you are using the same Git everywhere. In my experience, the only Git I install is GitHub for Windows. Then, I track down the msysgit installation path by navigating to User\AppData\Local\GitHub\PortableGit_[some-hash]\bin. I copy that path and add it to my user environment PATH variable. This way, any shell I'm in, I know that the git command is always hitting the GitHub for Windows installation. You'll save yourself a lot of grief by uninstalling any and all other versions of Git on your computer. An added bonus is that you get the git credential store for free, so you don't have to punch in your username/pw all the time. Just remember that sometimes GitHub updates create a new PortableGit hash, so if the git command all the sudden stops working you just have to track down that PortableGit bin path and replace the one in your user's PATH environment variable. Clear as mud?
And @pflannery, remember to use the Git Shell. The GitHub for Windows installation should have created a shortcut for you. You can also open a shell from the UI's gear icon.
Apparently this sort of logic is good for checking for binary files, I googled it a few times last time a similar bug bit me (someone was using regex to check for an svg file, and we somehow had a binary file that matched it) and it's pretty common: https://github.com/alessioalex/is-binary/blob/master/index.js
I'm not sure if that's how the file utility does it in the shell (and I don't know how to find the source), but the way suggested above seems common enough that I'm guessing it is.
@callumacrae you mean https://github.com/file/file ?
IMO, npm can incorporate .npmattributes in its publishing workflow, analogous to .gitattributes, by enforcing those endings rules in-memory on packaging.
Applying those rules in-memory (on-the-fly; as opposed to file-system itself) will prevent altering any file/content status in source-control on npm publish.
I created a module to address this:
@ben-page that's pretty awesome! Do you have a GitHub repo we can reference? I'd like to read the code and discuss any issues there.
@jedmao Thanks, I didn't realize that I neglected to add the repo to the package.json file. It's in there now. For good measure:
For what it's worth, I built https://github.com/ForbesLindesay/linify ages ago to solve this problem (it's poorly documented but does have a CLI). I really think it's something that should be resolved as a core part of npm though, given that it's ultimately just a cross platform issue.
@ForbesLindesay I didn't find your project when I searched. Does it handle gitignore or gitattributes?
But I agree. I wrote this module because I needed a solution immediately. But after I'm sure it's binary detection and gitigore parsing is solid, I'll probably work on npm PR.
I didn't look at gitignore/gitattributes files. I do support specifying just a specific file or directory to convert though which is sufficient for handling the "bin" case which is the only place this is actually a problem.
CRLF solves this too: https://github.com/kolodny/crlf
AAAaaaargggg I was having this problem a minute ago ¬¬
Convert eol to LF. Fix Linux / OSX bug