cspell fails to install using nodeenv

[ssbarnea]

While trying to enable cspell to be used a pre-commit hook, I faces a new issue, failure to install when doing:

pip install --user nodeenv
git clone https://github.com/Jason3S/cspell
cd cspell
nodeenv nenv
. nenv/bin/activate
npm install -g .

The last command fails with:

$ npm install -g .                                                                                                                                                                                          [21:18:12]
npm ERR! path /home/ssbarnea/cspell/nenv/lib/node_modules/cspell/dist/app.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/home/ssbarnea/cspell/nenv/lib/node_modules/cspell/dist/app.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ssbarnea/.npm/_logs/2018-07-04T21_18_18_526Z-debug.log
(nenv)FAIL: 254

[ssbarnea]

@ekalinin do you have any idea what could have cause this?

[asottile]

I can reproduce this without nodeenv, sorry should have boiled it down further before my comment on that other issue 😆

Dockerfile

FROM ubuntu:bionic
RUN : \
    && apt-get update \
    && DEBIAN_FRONTEND=noninteractive apt-get install \
        -y --no-install-recommends \
        ca-certificates \
        dumb-init \
        git-core \
        nodejs \
        npm \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
RUN git clone https://github.com/Jason3S/cspell /cspell
WORKDIR /cspell
CMD ["dumb-init", "npm", "install", "-g"]

script

docker build -t test .
docker run test

output

$ docker run test
npm WARN lifecycle cspell@3.1.2~prepublish: cannot run in wd %s %s (wd=%s) cspell@3.1.2 npm run lint && npm run clean-build && npm test /cspell
npm ERR! Linux 4.15.0-23-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "-g"
npm ERR! node v8.10.0
npm ERR! npm  v3.5.2
npm ERR! path /usr/local/lib/node_modules/cspell/dist/app.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod

npm ERR! enoent ENOENT: no such file or directory, chmod '/usr/local/lib/node_modules/cspell/dist/app.js'
npm ERR! enoent ENOENT: no such file or directory, chmod '/usr/local/lib/node_modules/cspell/dist/app.js'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

npm ERR! Please include the following file with any support request:
npm ERR!     /cspell/npm-debug.log

[Jason3S]

I see, you are cloning the repo.

To use cspell from a clone, it needs to be built.

npm run build
npm test
npm install -g

Please let me know if that works.

[ssbarnea]

@asottile I am a bit confused regarding where is the bug in this case. Shouldn't pre-commit be able to run build automatically or is cspell which should do this automatically when needed?

My impression was that npm install should work without extra magic commands but I nodejs is not my area expertise.

[Jason3S]

Is there a reason you are cloning the repository? npm install -g cspell Should install everything you need. npm will have the latest release. Cloning master has the risk of breaking.

…

On Thu, 5 Jul 2018 at 13:23, Sorin Sbarnea ***@***.***> wrote: @asottile <https://github.com/asottile> I am a bit confused regarding where is the bug in this case. Shouldn't pre-commit be able to run build automatically or is cspell which should do this automatically when needed? My impression was that npm install should work without extra magic commands but I nodejs is not my area expertise. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADkR6TCrTciY6aWLGLAKLRHoV48WoxzIks5uDfclgaJpZM4VDBz_> .

[asottile]

Is there a reason you are cloning the repository?

yeah the tool uses git revisions to enable easy forking / updating / etc. (It's also designed to work with a bunch of different languages so git is a nice lowest-common-denominator).

Every other npm package I can find works fine when running just npm install -g . -- what's special about cspell that prevents this from working?

Reading more into this, npm install should be calling npm build 🤔 (or at least that's what the docs indicate).

[asottile]

This patch seems to make npm install -g . work:

diff --git a/package.json b/package.json
index e3f34fa..a9516a8 100644
--- a/package.json
+++ b/package.json
@@ -19,6 +19,7 @@
     "lint": "tslint --force --format verbose \"src/**/*.ts\"",
     "lint-travis": "tslint \"src/**/*.ts\"",
     "build": "npm run compile",
+    "prepare": "npm run compile",
     "clean-build": "npm run clean && npm run build && npm run build-dictionaries",
     "build-dictionaries": "npm run build_dictionaries-word-lists",
     "build_dictionaries-word-lists": "node node_modules/cspell-tools/dist/app.js compile \"./dictionaries/!(words)*.txt\" -o ./dist/dictionaries/",

[Jason3S]

The package was using prepublish but that was deprecated. It now uses prepare.

[ssbarnea]

@Jason3S I am afraid that even after this change cspell does not install successfully inside nodeenv if it was not build a-priori.

The original reproduction steps from the bug description are still valid and produce the same failure.

It is critical to assure that cspell can be install in an isolated environment in order to be able to use it later.

[Jason3S]

@ssbarnea why cannot you run npm install?

pip install --user nodeenv
git clone https://github.com/Jason3S/cspell
cd cspell
# Setup cspell
npm install
nodeenv nenv
. nenv/bin/activate
npm install -g .

[ssbarnea]

@Jason3S I am almost sure that adding "npm install" in-between would be a workaround but I am afraid this is not the way to do it. Is like like installing it twice and also performing actions outside the virtual environment, which can have undesired side-effects.

A package should be installable directly, without extra operations. I asked on https://stackoverflow.com/questions/51283422/how-to-correctly-compile-a-nodejs-package-before-installing-it?noredirect=1#comment89544439_51283422 but it may take some time to get a solution. I checked 3-4 other nodejs packages and apparently none of them needed this bit strange compilation phase.

I am not sure if @asottile can and wants to add some extra steps into pre-commit node support in order to accommodate for this non standard installation method.

[asottile]

A package should be installable directly, without extra operations.

have to agree with this -- there has to be a way to make this work, going to look at some other packages and figure out how this is typically done

[Jason3S]

You are using an anti-pattern by directly using the repository. You should only be using the published package. The published package has everything you need.

The repository is not the package, the package is what is on NPM: https://www.npmjs.com/package/cspell

There seems to be something fundamentally wrong with nodeenv if it cannot repackage a published npm package into a virtual environment.

[Jason3S]

Why doesn't this work for you?:

pip install --user nodeenv
nodeenv nenv
. nenv/bin/activate
npm install -g cspell

[asottile]

pre-commit is a git hooks framework which supports many different programming languages. As a decent lowest common denominator that all the current languages support, it uses git repositories as the distribution mechanism (this has a few other benefits in that upgrading is an easy path forward, repeatability is inherent because it is a VCS, and the tooling is already installed).

Installing from source is a pretty common operation even outside of this for site-specific customizations. Every other npm project I can currently find supports installation from source (even npm itself allows direct installation from github!)

I'm fairly certain there is an easy fix here, just needs some more investigation.

An aside, I encourage you to be a little more polite, we're just trying to make the ecosystem a better place and throwing out "using an anti-pattern" and "something fundamentally wrong" isn't productive, polite discourse :) (I understand there's no COC for this repository -- but we're just trying to help!). It's possible too that I've misconstrued your words above as well (and if that's the case I apologize!), text is a really difficult medium!

[ssbarnea]

As long we all work together we should be fine. I love both projects so I hope we will find a way to make it work. I urge both of you not to get personal on this and focus on finding ways to address it. There is no pressure to make it work today or tomorrow

When I encountered pre-commit first time I found the git installation bit peculiar as the "norm" was to install a tool was to use package managers (like pip or npm). Now that I know more about it I do understand why git was picked in the first place: first is universal and not platform dependent. Also as opposed to package managers it gives some extra security by relying on hashes.

We all know that there were previous hacks on centralised package repositories which ended injecting malware in packages. With git+hashes this is almost impossible to happen because once a consumer points to a hash, this will remain there and only the same code would be allowed to install. Shortly what looked weird to me couple weeks ago started to appear as beautiful design once I started to understand it better.

Apparently both pip and npm allow installation of packages directly from git. It is true that cspell is the first package I encounter where this doesn't work, but I am also sure is not the only one. Now the challenge is to find others that had the same problem and resolved it.

Maybe @ekalinin knows something about this.

[ssbarnea]

Based on recent feedback I got from few sourced I am inclined to believe that the bin script should be a wrapper instead of the result of a build process. This should allow it to install and also allow to run a post-install step that builds the other files needed, after installation, in order to avoid the chicken-egg problem.

[Jason3S]

Why won't this work?

pip install --user nodeenv
git clone https://github.com/Jason3S/cspell
cd cspell
# Setup cspell
nodeenv nenv
. nenv/bin/activate
npm install
npm install -g .

The issue is that this git repository is a repository for the source code. The source code is in TypeScript. When this repository is published, it is compiled, tested and packaged.

This is a standard process. Nothing new. What is different between this repo and many other node repos is that the code was not written in JavaScript. If it had been written in JavaScript, then it would most likely (but not guaranteed) to have worked in the way you describe. Repos written in CoffeeScript or any of the other variants will have this issue if they do not check-in compiled code.

It is not a standard practice to check in compiled code. It creates large diffs / pull requests that are not easy to review.

[bisubus]

Of course, commiting compiled code is a shot in the foot.

@Jason3S The package contains problems with installation. I would expect this to work:

git clone https://github.com/Jason3S/cspell node_modules/cspell && cd node_modules/cspell && npm i && npm run build

This won't work because package.json contains "bin": { "cspell": "./dist/app.js" } that prevents the package from being normally installed. npm i exits with error code:

npm ERR! path <...>\node_modules\cspell\dist\app.js
npm ERR! code ENOENT
npm ERR! errno -4058
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '<...>\node_modules\cspell\dist\app.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

The log is:

10226 silly linkStuff cspell@3.1.3 has <...>\node_modules as its parent node_modules
10227 verbose linkBins [ { cspell: './dist/app.js' },
10227 verbose linkBins   '<...>\\node_modules\\.bin',
10227 verbose linkBins   false ]
10228 verbose stack Error: ENOENT: no such file or directory, chmod '<...>\node_modules\cspell\dist\app.js'

npm i cannot succeed because dist\app.js doesn't exist, and npm run build cannot be run to compile it to dist\app.js until you nave the package installed, this is chicken-egg situation.

I suppose there should be static entry point that exists regardless of dist, as it's usually done:

  "bin": {
    "cspell": "bin.js"
  },

[Jason3S]

@bisubus that makes sense because NPM has special behavior when installing to node_modules.
The general git clone and install process works just fine. See: Appveyor cSpell

I would welcome a pull request that adds a ./bin.js which loads ./dist/app.js to bootstrap the app.

I'm at work, so it is not easy for me to make the change myself.

[Jason3S]

Once the ./bin.js file is there, I expect that this should work just fine:

git clone https://github.com/Jason3S/cspell node_modules/cspell && cd node_modules/cspell && npm i

[bisubus]

@Jason3S Sure. I don't use cspell myself by I'd expect this to be enough.

[Jason3S]

@ssbarnea,

This might have broken when the repo moved to a monorepo.

Please use this instead:
streetsidesoftware/cspell-cli: cspell command line that can be used with pre-commit

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.