Permalink
Browse files

deps: Change npm to yarn for reliablity, security, and speed.

  • Loading branch information...
pweaver authored and timabbott committed Jul 27, 2017
1 parent f444c68 commit 1afaa67c7c83ad5a8631f476cf50b3b8373e1586
View
@@ -32,8 +32,9 @@ script:
- unset GEM_PATH
- mispipe "./tools/travis/$TEST_SUITE" ts
cache:
- apt: false
- directories:
yarn: true
apt: false
directories:
- $HOME/zulip-venv-cache
- $HOME/zulip-npm-cache
- $HOME/zulip-emoji-cache
@@ -886,50 +886,23 @@ gigabytes of RAM, which is the minimum Zulip
not, go to your VM settings and increase the RAM, then restart
the VM.
##### npm install errors
##### yarn install warnings
The `tools/provision` script may encounter an error related to `npm install`
that looks something like:
```
==> default: + npm install
==> default: Traceback (most recent call last):
==> default: File "/srv/zulip/tools/provision", line 195, in <module>
==> default:
==> default: sys.exit(main())
==> default: File "/srv/zulip/tools/provision", line 191, in main
==> default:
==> default: run(["npm", "install"])
==> default: File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run
==> default:
==> default: raise subprocess.CalledProcessError(rc, args)
==> default: subprocess
==> default: .
==> default: CalledProcessError
==> default: :
==> default: Command '['npm', 'install']' returned non-zero exit status 34
The SSH command responded with a non-zero exit status. Vagrant
assumes that this means the command failed. The output for this command
should be in the log above. Please read the output to determine what
went wrong.
```
Usually this error is not fatal. Try connecting to the development
environment and re-trying the command from withing the virtual
machine:
```
christie@win10 ~/zulip
$ vagrant ssh
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~
$ cd zulip
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip
$ npm install
npm WARN optional Skipping failed optional dependency /chokidar/fsevents:
npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.0.12
$ yarn install
yarn install v0.24.5
[1/4] Resolving packages...
[2/4] Fetching packages...
warning fsevents@1.1.1: The platform "linux" is incompatible with this module.
info "fsevents@1.1.1" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
$ browserify node_modules/sockjs-client/lib/entry.js --standalone SockJS > node_modules/sockjs-client/sockjs.js
Done in 23.50s.
```
These are just warnings so it is okay to proceed and start the Zulip server.
These are warnings produced by spammy third party JavaScript packages.
It is okay to proceed and start the Zulip server.
#### vagrant-lxc errors
@@ -46,7 +46,7 @@ should work.
Install the following non-Python dependencies:
* libffi-dev — needed for some Python extensions
* postgresql 9.1 or later — our database (client, server, headers)
* nodejs 0.10 (and npm)
* nodejs 0.10 (and yarn)
* memcached (and headers)
* rabbitmq-server
* libldap2-dev
@@ -69,7 +69,7 @@ sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
memcached rabbitmq-server libldap2-dev redis-server \
postgresql-server-dev-all libmemcached-dev python-dev \
python3-dev python-virtualenv hunspell-en-us nodejs \
nodejs-legacy npm git yui-compressor puppet gettext postgresql
nodejs-legacy git yui-compressor puppet gettext postgresql
# If using Ubuntu, install PGroonga from its PPA
sudo add-apt-repository -ys ppa:groonga/ppa
@@ -128,7 +128,7 @@ sudo apt-get update
sudo apt-get install closure-compiler libfreetype6-dev libffi-dev \
memcached rabbitmq-server libldap2-dev redis-server \
postgresql-server-dev-all libmemcached-dev python-dev \
hunspell-en-us nodejs nodejs-legacy npm git yui-compressor \
hunspell-en-us nodejs nodejs-legacy git yui-compressor \
puppet gettext tsearch-extras
```
@@ -146,7 +146,7 @@ https://github.com/zulip/zulip.git`
sudo dnf install libffi-devel memcached rabbitmq-server \
openldap-devel python-devel redis postgresql-server \
postgresql-devel postgresql libmemcached-devel freetype-devel \
nodejs npm yuicompressor closure-compiler gettext
nodejs yuicompressor closure-compiler gettext
```
Now continue with the [Common to Fedora/CentOS](#common-to-fedora-centos-instructions) instructions below.
@@ -324,7 +324,7 @@ Now run these commands:
```
sudo ./scripts/lib/install-node
npm install
yarn install
./tools/install-mypy
sudo mkdir /srv/zulip-emoji-cache
sudo chown -R `whoami`:`whoami` /srv/zulip-emoji-cache
@@ -364,10 +364,10 @@ proxy in the environment as follows:
export http_proxy=http://proxy_host:port
```
- And set the npm proxy and https-proxy using:
- And set the yarn proxy and https-proxy using:
```
npm config set proxy http://proxy_host:port
npm config set https-proxy http://proxy_host:port
yarn config set proxy http://proxy_host:port
yarn config set https-proxy http://proxy_host:port
```
## Using Docker (experimental)
@@ -61,7 +61,7 @@ templating systems.
* `static/third/` Third-party JavaScript and CSS that has been vendored.
* `node_modules/` Third-party JavaScript installed via `npm`.
* `node_modules/` Third-party JavaScript installed via `yarn`.
* `static/assets/` For assets not to be served to the web (e.g. the system to
generate our favicons).
@@ -23,11 +23,11 @@ To add a static file to the app (JavaScript, CSS, images, etc), first
add it to the appropriate place under `static/`.
- Third-party files that we haven't patched should be installed via
`npm`, so that it's easy to upgrade them and third-party code
`yarn`, so that it's easy to upgrade them and third-party code
doesn't bloat the Zulip repository. You can then access them in
`webpack.assets.json` via their paths under `node_modules`.
You'll want to add these to the `package.json` in the root of the
repository, and then provision (to have `npm` download them) before
repository, and then provision (to have `yarn` download them) before
continuing. Your commit should also update `PROVISION_VERSION` in
`version.py`. When adding modules to `package.json`, please pin
specific versions of them (don't using carets `^`, tildes `~`, etc).
View
@@ -1460,6 +1460,19 @@ $ tools/clean-branches --reviews
Deleting local branch review-original-5156 (was 5a1e982)
```
### Merge conflict on yarn.lock file
If there is a merge conflict on yarn.lock, yarn.lock, yarn should be run to
regenerate the file. *Important* don't delete the yarn.lock file. Checkout the
latest one from origin/master so that yarn knows the previous asset versions.
Run the following commands
```
git checkout origin/master -- yarn.lock
yarn install
git add yarn.lock
git rebase --continue
```
[gitbook-rebase]: https://git-scm.com/book/en/v2/Git-Branching-Rebasing
[gitbook-git-rebase]: https://git-scm.com/docs/git-rebase
@@ -9,7 +9,6 @@
$closure_compiler_package,
"nodejs",
"nodejs-legacy",
"npm",
"yui-compressor",
# Used by makemessages i18n
"gettext",
View
@@ -1,25 +1,26 @@
#!/usr/bin/env bash
set -e
ZULIP_PATH=$(dirname "$0")
ZULIP_PATH="$(dirname "$0")/../.."
ZULIP_SRV="/srv"
if [ "$TRAVIS" ] ; then
ZULIP_SRV="/home/travis"
fi
YARN_BIN="$ZULIP_SRV/zulip-yarn/bin/yarn"
node_version=6.6.0
npm_version=3.10.3
yarn_version=0.27.5
current_npm_version="none"
if hash npm 2>/dev/null; then
current_npm_version="$(npm --version)"
fi
current_node_version="none"
if hash node 2>/dev/null; then
current_node_version="$(node --version)"
fi
if [ "$current_npm_version" = "$npm_version" ] && [ "$current_node_version" = "v$node_version" ]; then
echo "Node version $node_version and npm version $npm_version are already installed."
if [ "$($YARN_BIN --version 2>/dev/null)" = "$yarn_version" ] && [ "$current_node_version" = "v$node_version" ]; then
echo "Node version $node_version and yarn version $yarn_version are already installed."
exit 0
fi
if true; then
if [ "$current_node_version" != "v$node_version" ]; then
export NVM_DIR=/usr/local/nvm
if ! [ -e "$NVM_DIR/nvm.sh" ]; then
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash
@@ -28,17 +29,17 @@ if true; then
source "$NVM_DIR/nvm.sh"
nvm install "$node_version" && nvm alias default "$node_version"
export NODE_BIN="$(nvm which default)"
export NPM_BIN=$(echo "$NODE_BIN" | sed 's/node$/npm/')
# Fix messed-up uid=500 and group write bits produced by nvm
n=$(which node)
n=${n%/bin/node}
chown -R root:root "$n"
chmod -R go-w "$n"
# Install node and npm wrappers to /usr/local/bin
cp "$ZULIP_PATH/../../scripts/setup/node-wrapper" /usr/local/bin/node
# Install node wrapper to /usr/local/bin
cp "$ZULIP_PATH/scripts/setup/node-wrapper" /usr/local/bin/node
sed -i "s|NODE_PATH|$NODE_BIN|" /usr/local/bin/node
cp "$ZULIP_PATH/../../scripts/setup/npm-wrapper" /usr/local/bin/npm
sed -i "s|NPM_PATH|$NPM_BIN|" /usr/local/bin/npm
fi
# Install yarn if not installed
bash "$ZULIP_PATH/scripts/lib/third/install-yarn.sh" "$ZULIP_SRV" --version "$yarn_version"
View
@@ -10,41 +10,49 @@
from scripts.lib.zulip_tools import subprocess_text_output, run
ZULIP_PATH = dirname(dirname(dirname(abspath(__file__))))
NODE_MODULES_CACHE_PATH = "/srv/zulip-npm-cache"
ZULIP_SRV_PATH = "/srv"
if 'TRAVIS' in os.environ:
# In Travis CI, we don't have root access
NODE_MODULES_CACHE_PATH = "/home/travis/zulip-npm-cache"
ZULIP_SRV_PATH = "/home/travis"
def generate_sha1sum_node_modules(npm_args=None):
NODE_MODULES_CACHE_PATH = os.path.join(ZULIP_SRV_PATH, 'zulip-npm-cache')
YARN_BIN = os.path.join(ZULIP_SRV_PATH, 'zulip-yarn/bin/yarn')
def generate_sha1sum_node_modules(yarn_args=None):
# type: (Optional[List[str]]) -> str
sha1sum = hashlib.sha1()
sha1sum.update(subprocess_text_output(['cat', 'package.json']).encode('utf8'))
sha1sum.update(subprocess_text_output(['npm', '--version']).encode('utf8'))
sha1sum.update(subprocess_text_output(['cat', 'yarn.lock']).encode('utf8'))
sha1sum.update(subprocess_text_output([YARN_BIN, '--version']).encode('utf8'))
sha1sum.update(subprocess_text_output(['node', '--version']).encode('utf8'))
if npm_args is not None:
sha1sum.update(''.join(sorted(npm_args)).encode('utf8'))
if yarn_args is not None:
sha1sum.update(''.join(sorted(yarn_args)).encode('utf8'))
return sha1sum.hexdigest()
def setup_node_modules(production=False, stdout=None, stderr=None, copy_modules=False):
# type: (bool, Optional[IO], Optional[IO], bool) -> None
def setup_node_modules(production=False, stdout=None, stderr=None, copy_modules=False,
prefer_offline=False):
# type: (bool, Optional[IO], Optional[IO], bool, bool) -> None
if production:
npm_args = ["--production"]
yarn_args = ["--prod"]
else:
npm_args = []
sha1sum = generate_sha1sum_node_modules(npm_args)
yarn_args = []
if prefer_offline:
yarn_args.append("--prefer-offline")
sha1sum = generate_sha1sum_node_modules(yarn_args)
target_path = os.path.join(NODE_MODULES_CACHE_PATH, sha1sum)
cached_node_modules = os.path.join(target_path, 'node_modules')
success_stamp = os.path.join(target_path, '.success-stamp')
# Check if a cached version already exists
if not os.path.exists(success_stamp):
do_npm_install(target_path,
npm_args,
success_stamp,
stdout=stdout,
stderr=stderr,
copy_modules=copy_modules)
do_yarn_install(target_path,
yarn_args,
success_stamp,
stdout=stdout,
stderr=stderr,
copy_modules=copy_modules)
print("Using cached node modules from %s" % (cached_node_modules,))
cmds = [
@@ -54,21 +62,26 @@ def setup_node_modules(production=False, stdout=None, stderr=None, copy_modules=
for cmd in cmds:
run(cmd, stdout=stdout, stderr=stderr)
def do_npm_install(target_path, npm_args, success_stamp, stdout=None, stderr=None,
copy_modules=False):
def do_yarn_install(target_path, yarn_args, success_stamp, stdout=None, stderr=None,
copy_modules=False):
# type: (str, List[str], str, Optional[IO], Optional[IO], bool) -> None
cmds = [
["rm", "-rf", target_path],
['mkdir', '-p', target_path],
['cp', 'package.json', target_path],
['cp', 'package.json', "yarn.lock", target_path],
]
cached_node_modules = os.path.join(target_path, 'node_modules')
if copy_modules:
print("Cached version not found! Copying node modules.")
cmds.append(["cp", "-rT", "prod-static/serve/node_modules", cached_node_modules])
else:
print("Cached version not found! Installing node modules.")
cmds.append(['npm', 'install'] + npm_args + ['--prefix', target_path])
# Copy the existing node_modules to speed up install
if os.path.exists("node_modules"):
cmds.append(["cp", "-R", "node_modules/", cached_node_modules])
cd_exec = os.path.join(ZULIP_PATH, "scripts/lib/cd_exec")
cmds.append([cd_exec, target_path, YARN_BIN, "install", "--non-interactive"] +
yarn_args)
cmds.append(['touch', success_stamp])
for cmd in cmds:
Oops, something went wrong.

0 comments on commit 1afaa67

Please sign in to comment.