npm publish from Windows handles CRLF incorrectly #2097

Closed
DukeyToo opened this Issue Jan 27, 2012 · 36 comments

Comments

Projects
None yet
@DukeyToo

When published from a Windows machine, the package files are uploaded with the windows-style CRLF endings. This is fine for most files, but if there are any shell scripts or a specified bin file, then those files will be corrupt and fail on a Linux machine.

I encountered this behavior publishing some of my packages from Windows from a git repository, using e.g.

$ npm publish git://github.com/myuser/mypackage.git

I understand why this occurs, but since this makes npm publish 50% unusable on Windows, I think it is fair to call it a bug in npm.

@mihaifm

This comment has been minimized.

Show comment
Hide comment
@mihaifm

mihaifm Apr 2, 2012

+1 for this issue, something similar to git's core.autocrlf option needs to be implemented.

mihaifm commented Apr 2, 2012

+1 for this issue, something similar to git's core.autocrlf option needs to be implemented.

@moos

This comment has been minimized.

Show comment
Hide comment
@moos

moos May 31, 2012

+1
Just ran into this publishing from Windows. Script installs/runs file on Windows cmd and git-bash. But on Ubuntu VM, kept getting a cryptic ": No such file or directory". Running fromdos on script fixed it.

Hopefully this can be addressed soon.

moos commented May 31, 2012

+1
Just ran into this publishing from Windows. Script installs/runs file on Windows cmd and git-bash. But on Ubuntu VM, kept getting a cryptic ": No such file or directory". Running fromdos on script fixed it.

Hopefully this can be addressed soon.

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Aug 2, 2012

Member

Oh eww, I didn't know I was doing this to people :(

Member

domenic commented Aug 2, 2012

Oh eww, I didn't know I was doing this to people :(

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Aug 2, 2012

Member

Hm, yeah, that's a pita.

It'd be nice for npm to do this automatically when generating the tarball, but I'm not sure how without potentially breaking binary files. There's got to be some kind of way to detect if it's a "text" file, right? Git has this, after all.

Member

isaacs commented Aug 2, 2012

Hm, yeah, that's a pita.

It'd be nice for npm to do this automatically when generating the tarball, but I'm not sure how without potentially breaking binary files. There's got to be some kind of way to detect if it's a "text" file, right? Git has this, after all.

@heycalmdown

This comment has been minimized.

Show comment
Hide comment
@heycalmdown

heycalmdown Aug 4, 2012

Contributor

+1

Oh it can just be reached easy by implement a function to figure out the first line of the binary which defined in the "bin" field of package.json that starts with #!

Contributor

heycalmdown commented Aug 4, 2012

+1

Oh it can just be reached easy by implement a function to figure out the first line of the binary which defined in the "bin" field of package.json that starts with #!

@isaacs

This comment has been minimized.

Show comment
Hide comment
@isaacs

isaacs Aug 5, 2012

Member

@heycalmdown Well, ideally npm should convert \r\n to \n in ALL non-binary files when creating the tarball on Windows.

Member

isaacs commented Aug 5, 2012

@heycalmdown Well, ideally npm should convert \r\n to \n in ALL non-binary files when creating the tarball on Windows.

@medikoo

This comment has been minimized.

Show comment
Hide comment
@medikoo

medikoo Aug 6, 2012

@isaacs maybe ideally it should warn and exit, but optionally with eg --fixeol flag it should do automatic conversion to \n ? Some may be upset that npm just converts new lines without a note.

medikoo commented Aug 6, 2012

@isaacs maybe ideally it should warn and exit, but optionally with eg --fixeol flag it should do automatic conversion to \n ? Some may be upset that npm just converts new lines without a note.

@ForbesLindesay

This comment has been minimized.

Show comment
Hide comment
@ForbesLindesay

ForbesLindesay May 31, 2013

Member

From an implementation point of view:

Ideally the change should not impact the files on disk. It probably needs to happen as part of the packing that goes into ./lib/utils/tar.js but this doesn't seem to have any easy way to perform such a mutation.

As I understand it, https://github.com/isaacs/fstream-npm just emits file name events that tar then packages. Since tar is a very generic utility, we can't have tar do the packaging.

What if tar was to just receive objects containing filename and content (as a stream) as its input?

Member

ForbesLindesay commented May 31, 2013

From an implementation point of view:

Ideally the change should not impact the files on disk. It probably needs to happen as part of the packing that goes into ./lib/utils/tar.js but this doesn't seem to have any easy way to perform such a mutation.

As I understand it, https://github.com/isaacs/fstream-npm just emits file name events that tar then packages. Since tar is a very generic utility, we can't have tar do the packaging.

What if tar was to just receive objects containing filename and content (as a stream) as its input?

@tleish tleish referenced this issue in jedmao/codepainter Dec 19, 2013

Closed

Issue with line endings #14

@jedmao

This comment has been minimized.

Show comment
Hide comment
@jedmao

jedmao Dec 20, 2013

I have a work-around for this issue that has been verified to work.

@isaacs, I want to point out that no, it's not a good idea to replace all non-binary file line endings with \n. In my case, in CodePainter, I very intentionally left some line endings as \r\n and \r for my tests. These line endings need to be preserved.

jedmao commented Dec 20, 2013

I have a work-around for this issue that has been verified to work.

@isaacs, I want to point out that no, it's not a good idea to replace all non-binary file line endings with \n. In my case, in CodePainter, I very intentionally left some line endings as \r\n and \r for my tests. These line endings need to be preserved.

@tohagan

This comment has been minimized.

Show comment
Hide comment
@tohagan

tohagan Sep 1, 2014

+1 I've hit this one several times. Tonight it got nastier as I'd publish a lib containing a package.json file that had CRLF's in it (unknown to me). I then attempted to publish another dependent package but npm refused to publish it because the lib it depended on contained an unparsable package.json. Really ugly stuff! Cleaning up CRLFs would prevent this.

I now tend to run dos2unix on most of my files prior to publishing just to be sure. This is really getting painful.

tohagan commented Sep 1, 2014

+1 I've hit this one several times. Tonight it got nastier as I'd publish a lib containing a package.json file that had CRLF's in it (unknown to me). I then attempted to publish another dependent package but npm refused to publish it because the lib it depended on contained an unparsable package.json. Really ugly stuff! Cleaning up CRLFs would prevent this.

I now tend to run dos2unix on most of my files prior to publishing just to be sure. This is really getting painful.

@tohagan

This comment has been minimized.

Show comment
Hide comment
@tohagan

tohagan Sep 1, 2014

I think you could make it either an npm command line option or perhaps in package.json define a list of file patterns to convert (or not convert).

tohagan commented Sep 1, 2014

I think you could make it either an npm command line option or perhaps in package.json define a list of file patterns to convert (or not convert).

@domenic

This comment has been minimized.

Show comment
Hide comment
@domenic

domenic Sep 1, 2014

Member

My solution has been to do all of my development with LF and ban CRLF files from my source code folder.

Member

domenic commented Sep 1, 2014

My solution has been to do all of my development with LF and ban CRLF files from my source code folder.

@tohagan

This comment has been minimized.

Show comment
Hide comment
@tohagan

tohagan Sep 1, 2014

Yes I've done the same - but still hitting issues. Enforced with .gitattribute files also. However I use several IDEs/Editors and I think one of them is to blame.

tohagan commented Sep 1, 2014

Yes I've done the same - but still hitting issues. Enforced with .gitattribute files also. However I use several IDEs/Editors and I think one of them is to blame.

@tohagan

This comment has been minimized.

Show comment
Hide comment
@tohagan

tohagan Sep 1, 2014

But getting npm to help would prevent most of these problems.

tohagan commented Sep 1, 2014

But getting npm to help would prevent most of these problems.

@tohagan

This comment has been minimized.

Show comment
Hide comment
@tohagan

tohagan Oct 23, 2014

Thanks!

tohagan commented Oct 23, 2014

Thanks!

@steffenmllr steffenmllr referenced this issue in nwjs-community/nw-builder Nov 18, 2014

Closed

env: node\r: No such file or directory? #109

@pflannery pflannery referenced this issue in bevry/projectz Feb 10, 2015

Closed

: No such file or directory #59

@pflannery

This comment has been minimized.

Show comment
Hide comment
@pflannery

pflannery Feb 10, 2015

+1

I just ran in to this...turns out Github for Windows is overriding my git line ending preferences and applying CRLF in all my clones. Worse still Github for Windows doesn't let you choose your line ending preferences so forced back to cloning from the console :'(

+1

I just ran in to this...turns out Github for Windows is overriding my git line ending preferences and applying CRLF in all my clones. Worse still Github for Windows doesn't let you choose your line ending preferences so forced back to cloning from the console :'(

@jedmao

This comment has been minimized.

Show comment
Hide comment
@jedmao

jedmao Feb 10, 2015

@pflannery actually it does let you choose. Just go into the repository in question and run the following:

git config core.autocrlf false
git rm --cached -r .
git reset --hard

The first command tells Git to never change line endings (in the future). Next, we refresh each repository by removing every file from Git's index and, finally, rewriting the Git index to pick up all the new line endings. This fixes the CRLFs that were introduced to your local file system when you cloned each repository.

jedmao commented Feb 10, 2015

@pflannery actually it does let you choose. Just go into the repository in question and run the following:

git config core.autocrlf false
git rm --cached -r .
git reset --hard

The first command tells Git to never change line endings (in the future). Next, we refresh each repository by removing every file from Git's index and, finally, rewriting the Git index to pick up all the new line endings. This fixes the CRLFs that were introduced to your local file system when you cloned each repository.

@shenanigans

This comment has been minimized.

Show comment
Hide comment
@shenanigans

shenanigans Feb 10, 2015

I tried this and didn't get anywhere. I also tried it from a linux system and was unable to remove the CRLFs from the remote. I don't disagree that git can do this, just not without giving me a hernia.

I tried this and didn't get anywhere. I also tried it from a linux system and was unable to remove the CRLFs from the remote. I don't disagree that git can do this, just not without giving me a hernia.

@pflannery

This comment has been minimized.

Show comment
Hide comment
@pflannery

pflannery Feb 11, 2015

@jedmao but when I git clone from the terminal it's fine, no CRLF. but when I clone from Github for Windows I get CRLF.

@jedmao but when I git clone from the terminal it's fine, no CRLF. but when I clone from Github for Windows I get CRLF.

@pflannery

This comment has been minimized.

Show comment
Hide comment
@pflannery

pflannery Feb 11, 2015

@jedmao looking deeper it seems github uses portable-git and not the git I installed from git-scm.com....I'll try to set the config for portable-git..

@jedmao looking deeper it seems github uses portable-git and not the git I installed from git-scm.com....I'll try to set the config for portable-git..

@jedmao

This comment has been minimized.

Show comment
Hide comment
@jedmao

jedmao Feb 11, 2015

Yes, it's important to know that you are using the same Git everywhere. In my experience, the only Git I install is GitHub for Windows. Then, I track down the msysgit installation path by navigating to User\AppData\Local\GitHub\PortableGit_[some-hash]\bin. I copy that path and add it to my user environment PATH variable. This way, any shell I'm in, I know that the git command is always hitting the GitHub for Windows installation. You'll save yourself a lot of grief by uninstalling any and all other versions of Git on your computer. An added bonus is that you get the git credential store for free, so you don't have to punch in your username/pw all the time. Just remember that sometimes GitHub updates create a new PortableGit hash, so if the git command all the sudden stops working you just have to track down that PortableGit bin path and replace the one in your user's PATH environment variable. Clear as mud?

jedmao commented Feb 11, 2015

Yes, it's important to know that you are using the same Git everywhere. In my experience, the only Git I install is GitHub for Windows. Then, I track down the msysgit installation path by navigating to User\AppData\Local\GitHub\PortableGit_[some-hash]\bin. I copy that path and add it to my user environment PATH variable. This way, any shell I'm in, I know that the git command is always hitting the GitHub for Windows installation. You'll save yourself a lot of grief by uninstalling any and all other versions of Git on your computer. An added bonus is that you get the git credential store for free, so you don't have to punch in your username/pw all the time. Just remember that sometimes GitHub updates create a new PortableGit hash, so if the git command all the sudden stops working you just have to track down that PortableGit bin path and replace the one in your user's PATH environment variable. Clear as mud?

@jedmao

This comment has been minimized.

Show comment
Hide comment
@jedmao

jedmao Feb 11, 2015

And @pflannery, remember to use the Git Shell. The GitHub for Windows installation should have created a shortcut for you. You can also open a shell from the UI's gear icon.

jedmao commented Feb 11, 2015

And @pflannery, remember to use the Git Shell. The GitHub for Windows installation should have created a shortcut for you. You can also open a shell from the UI's gear icon.

@callumacrae

This comment has been minimized.

Show comment
Hide comment
@callumacrae

callumacrae Mar 13, 2015

Apparently this sort of logic is good for checking for binary files, I googled it a few times last time a similar bug bit me (someone was using regex to check for an svg file, and we somehow had a binary file that matched it) and it's pretty common: https://github.com/alessioalex/is-binary/blob/master/index.js

I'm not sure if that's how the file utility does it in the shell (and I don't know how to find the source), but the way suggested above seems common enough that I'm guessing it is.

Apparently this sort of logic is good for checking for binary files, I googled it a few times last time a similar bug bit me (someone was using regex to check for an svg file, and we somehow had a binary file that matched it) and it's pretty common: https://github.com/alessioalex/is-binary/blob/master/index.js

I'm not sure if that's how the file utility does it in the shell (and I don't know how to find the source), but the way suggested above seems common enough that I'm guessing it is.

@bmeck

This comment has been minimized.

Show comment
Hide comment
@bmeck

bmeck Mar 13, 2015

Contributor
Contributor

bmeck commented Mar 13, 2015

@am11 am11 referenced this issue in sass/node-sass Mar 14, 2015

Merged

Repo: Adds .gitattributes #760

@am11

This comment has been minimized.

Show comment
Hide comment
@am11

am11 Mar 14, 2015

IMO, npm can incorporate .npmattributes in its publishing workflow, analogous to .gitattributes, by enforcing those endings rules in-memory on packaging.
Applying those rules in-memory (on-the-fly; as opposed to file-system itself) will prevent altering any file/content status in source-control on npm publish.

am11 commented Mar 14, 2015

IMO, npm can incorporate .npmattributes in its publishing workflow, analogous to .gitattributes, by enforcing those endings rules in-memory on packaging.
Applying those rules in-memory (on-the-fly; as opposed to file-system itself) will prevent altering any file/content status in source-control on npm publish.

@jrit jrit referenced this issue in Automattic/juice May 7, 2015

Closed

[Question/Suggestion] - Command line processing #121

@ben-page

This comment has been minimized.

Show comment
Hide comment
@ben-page

ben-page Sep 29, 2015

Contributor

I created a module to address this:
https://www.npmjs.com/package/npm-text-auto

Contributor

ben-page commented Sep 29, 2015

I created a module to address this:
https://www.npmjs.com/package/npm-text-auto

@jedmao

This comment has been minimized.

Show comment
Hide comment
@jedmao

jedmao Sep 29, 2015

@ben-page that's pretty awesome! Do you have a GitHub repo we can reference? I'd like to read the code and discuss any issues there.

jedmao commented Sep 29, 2015

@ben-page that's pretty awesome! Do you have a GitHub repo we can reference? I'd like to read the code and discuss any issues there.

@ben-page

This comment has been minimized.

Show comment
Hide comment
@ben-page

ben-page Sep 30, 2015

Contributor

@jedmao Thanks, I didn't realize that I neglected to add the repo to the package.json file. It's in there now. For good measure:
https://github.com/ben-page/npm-text-auto

Contributor

ben-page commented Sep 30, 2015

@jedmao Thanks, I didn't realize that I neglected to add the repo to the package.json file. It's in there now. For good measure:
https://github.com/ben-page/npm-text-auto

@ForbesLindesay

This comment has been minimized.

Show comment
Hide comment
@ForbesLindesay

ForbesLindesay Sep 30, 2015

Member

For what it's worth, I built https://github.com/ForbesLindesay/linify ages ago to solve this problem (it's poorly documented but does have a CLI). I really think it's something that should be resolved as a core part of npm though, given that it's ultimately just a cross platform issue.

Member

ForbesLindesay commented Sep 30, 2015

For what it's worth, I built https://github.com/ForbesLindesay/linify ages ago to solve this problem (it's poorly documented but does have a CLI). I really think it's something that should be resolved as a core part of npm though, given that it's ultimately just a cross platform issue.

@ben-page

This comment has been minimized.

Show comment
Hide comment
@ben-page

ben-page Sep 30, 2015

Contributor

@ForbesLindesay I didn't find your project when I searched. Does it handle gitignore or gitattributes?

But I agree. I wrote this module because I needed a solution immediately. But after I'm sure it's binary detection and gitigore parsing is solid, I'll probably work on npm PR.

Contributor

ben-page commented Sep 30, 2015

@ForbesLindesay I didn't find your project when I searched. Does it handle gitignore or gitattributes?

But I agree. I wrote this module because I needed a solution immediately. But after I'm sure it's binary detection and gitigore parsing is solid, I'll probably work on npm PR.

@ForbesLindesay

This comment has been minimized.

Show comment
Hide comment
@ForbesLindesay

ForbesLindesay Sep 30, 2015

Member

I didn't look at gitignore/gitattributes files. I do support specifying just a specific file or directory to convert though which is sufficient for handling the "bin" case which is the only place this is actually a problem.

Member

ForbesLindesay commented Sep 30, 2015

I didn't look at gitignore/gitattributes files. I do support specifying just a specific file or directory to convert though which is sufficient for handling the "bin" case which is the only place this is actually a problem.

@kolodny

This comment has been minimized.

Show comment
Hide comment
@kolodny

kolodny Oct 2, 2015

CRLF solves this too: https://github.com/kolodny/crlf

kolodny commented Oct 2, 2015

CRLF solves this too: https://github.com/kolodny/crlf

@heycalmdown heycalmdown referenced this issue in alm-tools/alm Nov 1, 2015

Closed

.bin/tsb broken #33

@jedmao jedmao referenced this issue in editorconfig/editorconfig-core-js Nov 29, 2015

Closed

Install fails on OS X #29

sarcadass added a commit to sarcadass/gulpit that referenced this issue Dec 5, 2015

@dfalling dfalling referenced this issue in jonatanpedersen/git-json-merge Dec 15, 2015

Closed

npm package published with DOS line endings #1

@M-Zuber M-Zuber referenced this issue in M-Zuber/npm-watch Jan 5, 2016

Closed

Error when I try to run npm-watch #11

@snowdream snowdream referenced this issue in snowdream/node-gitm Jan 11, 2016

Closed

env: node\r: No such file or directory #4

@Mithgol Mithgol referenced this issue in nwjs/npm-installer Feb 12, 2016

Closed

nw bin script has CRLF endings #37

@Mithgol

This comment has been minimized.

Show comment
Hide comment
@Mithgol

Mithgol Feb 12, 2016

+1 to the thought that a binary detection is not really necessary. If the file is listed in the bin section of the package.json file and if its starts with #!/usr/bin/env node (or something similar), then it's a script that should be converted from CRLF to LF and otherwise it won't work.

Why npm should expect Git's core.autocrlf to be set to true on Windows (and should provide some npm's workaround for CRLF line endings)? — that's because Git for Windows recommends it:

(screenshot)

(And it has a good reason to recommend such setting. Many tools on Windows default to CRLF unless explicitly told otherwise, and remembering all of them can be quite tedious.)

Mithgol commented Feb 12, 2016

+1 to the thought that a binary detection is not really necessary. If the file is listed in the bin section of the package.json file and if its starts with #!/usr/bin/env node (or something similar), then it's a script that should be converted from CRLF to LF and otherwise it won't work.

Why npm should expect Git's core.autocrlf to be set to true on Windows (and should provide some npm's workaround for CRLF line endings)? — that's because Git for Windows recommends it:

(screenshot)

(And it has a good reason to recommend such setting. Many tools on Windows default to CRLF unless explicitly told otherwise, and remembering all of them can be quite tedious.)

@bluecmd bluecmd referenced this issue in remy/inliner Feb 27, 2016

Closed

Published NPM has CRLF bug #90

@remixz remixz referenced this issue in remixz/run-js Feb 29, 2016

Closed

DOS line endings in current release (2.1.0) #12

@sculley4 sculley4 referenced this issue in showdownjs/showdown Mar 8, 2016

Closed

Showdown CLI Not Working in OS X or Ubuntu #244

@tivac tivac referenced this issue in MithrilJS/mopt Mar 8, 2016

Closed

Script fails in Linux #19

@jcpst jcpst referenced this issue in M-Zuber/npm-watch Mar 31, 2016

Closed

Line endings break cli.js on Unix Machines #15

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Apr 15, 2016

Contributor

Closing this in favor of #12371, which I believe preserves the simplest way to resolve this, as well as a path for those who might be up to sending us a solution before we have the time to do so ourselves. Thanks to all for their time and (especially) patience.

Contributor

othiym23 commented Apr 15, 2016

Closing this in favor of #12371, which I believe preserves the simplest way to resolve this, as well as a path for those who might be up to sending us a solution before we have the time to do so ourselves. Thanks to all for their time and (especially) patience.

@othiym23 othiym23 closed this Apr 15, 2016

@tivac

This comment has been minimized.

Show comment
Hide comment
@tivac

tivac Apr 15, 2016

@othiym23 This... is issue #2097 though? 😕

tivac commented Apr 15, 2016

@othiym23 This... is issue #2097 though? 😕

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Apr 15, 2016

Contributor

@tivac ahem updated to point to #12371 correctly. My apologies for the error.

Contributor

othiym23 commented Apr 15, 2016

@tivac ahem updated to point to #12371 correctly. My apologies for the error.

@hparra hparra referenced this issue in JimmyBoh/pully May 1, 2016

Closed

osx cli issue #9

@cmgiven cmgiven referenced this issue in alallier/reload Jun 15, 2016

Merged

0.8.1 CLI broken on Linux/Mac #40

@morlay morlay referenced this issue in skolmer/i18n-tag-schema Aug 26, 2016

Closed

CLI broken on unix machine #7

@dagoof dagoof referenced this issue in daviesian/single-page-server Oct 10, 2016

Open

Published NPM version breaks on unix #3

@vanthome vanthome referenced this issue in dcodeIO/protobuf.js Dec 1, 2016

Closed

CLI not usable under Unix #511

@uuf6429 uuf6429 referenced this issue in paulcbetts/xvfb-maybe Apr 27, 2017

Merged

Added support for xvfb-run arguments #5

@wvillegasm wvillegasm referenced this issue in darkguy2008/parallelshell Jun 15, 2017

Closed

/usr/bin/env: ‘node\r’: No such file or directory ... #58

@abrahm abrahm referenced this issue in aluanhaddad/aurelia-types-installer Jun 23, 2017

Closed

npm package #2

@jtrussell jtrussell referenced this issue in jtrussell/semver-tags Aug 31, 2017

Closed

incorrect line endings in bin/semver-tags #9

@iyuuya iyuuya referenced this issue in srph/npm-scripts-info Apr 24, 2018

Open

env: node\r: No such file or directory #13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment