Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

npm publish from Windows handles CRLF incorrectly #2097

Open
DukeyToo opened this Issue · 33 comments
@DukeyToo

When published from a Windows machine, the package files are uploaded with the windows-style CRLF endings. This is fine for most files, but if there are any shell scripts or a specified bin file, then those files will be corrupt and fail on a Linux machine.

I encountered this behavior publishing some of my packages from Windows from a git repository, using e.g.

$ npm publish git://github.com/myuser/mypackage.git

I understand why this occurs, but since this makes npm publish 50% unusable on Windows, I think it is fair to call it a bug in npm.

@mihaifm

+1 for this issue, something similar to git's core.autocrlf option needs to be implemented.

@moos

+1
Just ran into this publishing from Windows. Script installs/runs file on Windows cmd and git-bash. But on Ubuntu VM, kept getting a cryptic ": No such file or directory". Running fromdos on script fixed it.

Hopefully this can be addressed soon.

@tivac

+1

Just ran into this myself.

@domenic
Collaborator

Oh eww, I didn't know I was doing this to people :(

@isaacs
Owner

Hm, yeah, that's a pita.

It'd be nice for npm to do this automatically when generating the tarball, but I'm not sure how without potentially breaking binary files. There's got to be some kind of way to detect if it's a "text" file, right? Git has this, after all.

@heycalmdown

+1

Oh it can just be reached easy by implement a function to figure out the first line of the binary which defined in the "bin" field of package.json that starts with #!

@isaacs
Owner

@heycalmdown Well, ideally npm should convert \r\n to \n in ALL non-binary files when creating the tarball on Windows.

@medikoo

@isaacs maybe ideally it should warn and exit, but optionally with eg --fixeol flag it should do automatic conversion to \n ? Some may be upset that npm just converts new lines without a note.

@ForbesLindesay

From an implementation point of view:

Ideally the change should not impact the files on disk. It probably needs to happen as part of the packing that goes into ./lib/utils/tar.js but this doesn't seem to have any easy way to perform such a mutation.

As I understand it, https://github.com/isaacs/fstream-npm just emits file name events that tar then packages. Since tar is a very generic utility, we can't have tar do the packaging.

What if tar was to just receive objects containing filename and content (as a stream) as its input?

@tleish tleish referenced this issue in jedmao/codepainter
Closed

Issue with line endings #14

@jedmao

I have a work-around for this issue that has been verified to work.

@isaacs, I want to point out that no, it's not a good idea to replace all non-binary file line endings with \n. In my case, in CodePainter, I very intentionally left some line endings as \r\n and \r for my tests. These line endings need to be preserved.

@tohagan

+1 I've hit this one several times. Tonight it got nastier as I'd publish a lib containing a package.json file that had CRLF's in it (unknown to me). I then attempted to publish another dependent package but npm refused to publish it because the lib it depended on contained an unparsable package.json. Really ugly stuff! Cleaning up CRLFs would prevent this.

I now tend to run dos2unix on most of my files prior to publishing just to be sure. This is really getting painful.

@tohagan

I think you could make it either an npm command line option or perhaps in package.json define a list of file patterns to convert (or not convert).

@domenic
Collaborator

My solution has been to do all of my development with LF and ban CRLF files from my source code folder.

@tohagan

Yes I've done the same - but still hitting issues. Enforced with .gitattribute files also. However I use several IDEs/Editors and I think one of them is to blame.

@tohagan

But getting npm to help would prevent most of these problems.

@othiym23 othiym23 removed the nice to have label
@tohagan

Thanks!

@shenanigans

+1

I'm not sure when or how I got infested with \r but I can think of a great time to get rid of them.

@pflannery pflannery referenced this issue in bevry/projectz
Closed

: No such file or directory #59

@pflannery

+1

I just ran in to this...turns out Github for Windows is overriding my git line ending preferences and applying CRLF in all my clones. Worse still Github for Windows doesn't let you choose your line ending preferences so forced back to cloning from the console :'(

@jedmao

@pflannery actually it does let you choose. Just go into the repository in question and run the following:

git config core.autocrlf false
git rm --cached -r .
git reset --hard

The first command tells Git to never change line endings (in the future). Next, we refresh each repository by removing every file from Git's index and, finally, rewriting the Git index to pick up all the new line endings. This fixes the CRLFs that were introduced to your local file system when you cloned each repository.

@shenanigans

I tried this and didn't get anywhere. I also tried it from a linux system and was unable to remove the CRLFs from the remote. I don't disagree that git can do this, just not without giving me a hernia.

@pflannery

@jedmao but when I git clone from the terminal it's fine, no CRLF. but when I clone from Github for Windows I get CRLF.

@pflannery

@jedmao looking deeper it seems github uses portable-git and not the git I installed from git-scm.com....I'll try to set the config for portable-git..

@jedmao

Yes, it's important to know that you are using the same Git everywhere. In my experience, the only Git I install is GitHub for Windows. Then, I track down the msysgit installation path by navigating to User\AppData\Local\GitHub\PortableGit_[some-hash]\bin. I copy that path and add it to my user environment PATH variable. This way, any shell I'm in, I know that the git command is always hitting the GitHub for Windows installation. You'll save yourself a lot of grief by uninstalling any and all other versions of Git on your computer. An added bonus is that you get the git credential store for free, so you don't have to punch in your username/pw all the time. Just remember that sometimes GitHub updates create a new PortableGit hash, so if the git command all the sudden stops working you just have to track down that PortableGit bin path and replace the one in your user's PATH environment variable. Clear as mud?

@jedmao

And @pflannery, remember to use the Git Shell. The GitHub for Windows installation should have created a shortcut for you. You can also open a shell from the UI's gear icon.

@callumacrae

Apparently this sort of logic is good for checking for binary files, I googled it a few times last time a similar bug bit me (someone was using regex to check for an svg file, and we somehow had a binary file that matched it) and it's pretty common: https://github.com/alessioalex/is-binary/blob/master/index.js

I'm not sure if that's how the file utility does it in the shell (and I don't know how to find the source), but the way suggested above seems common enough that I'm guessing it is.

@am11 am11 referenced this issue in sass/node-sass
Merged

Repo: Adds .gitattributes #760

@am11

IMO, npm can incorporate .npmattributes in its publishing workflow, analogous to .gitattributes, by enforcing those endings rules in-memory on packaging.
Applying those rules in-memory (on-the-fly; as opposed to file-system itself) will prevent altering any file/content status in source-control on npm publish.

@janez87

+1

@Volox

+10

@jrit jrit referenced this issue in Automattic/juice
Closed

[Question/Suggestion] - Command line processing #121

@kolodny

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.