Devasena Inupakutika (di1c13): new volunteer via Mozilla Science Lab Collaborate

[DevasenaInupakutika]

Reproducible science aspect of this project. Automating the installation of refinery platform and making it available for researchers across the world.

I can help with the automation of the refinery platform and its provisioning procedure with the skills (my interest in reproducible research) that I developed in reproducible research and vagrant (we are currently working on Microsoft Azure as a cloud platform) during my current work involving setting up of cloud infrastructure with recomputation.org team in making scientific experiments available across wide research community.

This issue was created by @DevasenaInupakutika via Mozilla Science Lab Collaborate

[ngehlenborg]

@DevasenaInupakutika: Thanks for your interest in the project. We would be glad to have you input on tasks related automated deployment of Refinery.

A good first step would be to try to get Refinery to work using the current deployment that we are suggesting for developers (https://github.com/parklab/refinery-platform/blob/develop/README.rst).

Then you could check out the tasks for the "Automated Deployment" milestone (https://github.com/parklab/refinery-platform/milestones/Automated%20Deployment) and get back to us with suggestions for how you would address them.

Of course we'd be very happy to answer whatever questions you have about the project and the software.

We are looking forward to hearing from you!

[DevasenaInupakutika]

@ngehlenborg Thank you very much for the information.
I'm trying to get Refinery working as per the current deployment steps mentioned in README. And looked at the issues under automated development milestones. I'd update them with ideas once that's done.

Regarding setting up my machine (Mac OSX 10.9.2) for current deployment of Refinery, I would like to know if its required to get 'apt-get' on MAC OSX using 'Fink' installer or it would just work fine with 'brew' or 'yum'. When I'm trying to run 'Vagrant provision' multiple times in order to install dependencies, sometimes it shows errors related to '/usr/bin/apt-get' but most of the times it just doesn't.

[hackdna]

Hi, you don't need apt-get on the OS X host to run Refinery: only git, VirtualBox and Vagrant. Also, make sure you have a fast and reliable Internet connection when running vagrant up or vagrant provision: this process pulls a lot of dependencies from various online repositories.

[DevasenaInupakutika]

@hackdna Thank you. I got the current deployment working successfully.
I would like to add few things which I had to run additionally while getting this work on Mac OSX:

1. After doing vagrant up, I got error message as below after a log of npm ERR with notice: :

err: /Stage[main]/Ui/Exec[npm_local_modules]/returns: change from notrun to 0 failed: /usr/bin/npm install returned 1 instead of one of [0] at /tmp/vagrant-puppet-3/manifests/default.pp:208

==> default: notice: Finished catalog run in 937.98 seconds
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

puppet apply --modulepath /vagrant/deployment/modules --manifestdir /tmp/vagrant-puppet-3/manifests --detailed-exitcodes /tmp/vagrant-puppet-3/manifests/default.pp

So, in order to get rid of this error due to npm install, I ran this command prior to vagrant provision:

cd refinery\ui
sudo npm install -g grunt-cli && npm install

Followed by vagrant provision resolves this issue.

Could you please let me know if it would be good to include this step in README.rst.

[hackdna]

Congratulations, you've discovered your first deployment bug! :) grunt-cli should be installed automatically (https://github.com/parklab/refinery-platform/blob/develop/deployment/manifests/default.pp#L199). It'd be great if you could figure out why is this not always happening.

[DevasenaInupakutika]

@hackdna I could figure out the reason based on the error logs that I got and system directory default settings:

From the path that is mentioned in exec section after package in default.pp i.e.
(https://github.com/parklab/refinery-platform/blob/develop/deployment/manifests/default.pp#L203)

which says that exec scripts that will be run will be placed in /usr/bin/npm directory. Ideally this should be used taking into account privacy of user accounts.

However, it could be possible that in case of OSX (like in this case), path hasn't been set up in ~/.bash_profile and hence, exec scripts are by default getting placed in /usr/local/bin/* i.e. /usr/local/bin/npm instead. This happens when we want the scripts to be available under all user accounts or don't have any other user account or just don't care about the location.

[hackdna]

This has nothing to do with the OS X host. The NPM executable is getting installed into /usr/bin in the Vagrant guest (which is expected by the exec resource in default.pp#L203):

vagrant@refinery:~$ which npm
/usr/bin/npm

It appears that grunt-cli package sometimes may not be getting installed before /usr/bin/npm install command is executed. What are the actual error messages reported? You can obtain more detailed output by adding --debug and --verbose to puppet.options in Vagrantfile.

[DevasenaInupakutika]

Here's the detailed error log:

==> default: notice: /Stage[main]//Exec[pip_requirements_install]/returns: executed successfully
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm WARN package.json refinery-platform-ui@0.0.0 No README data
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! Error: Attempt to unlock grunt@~0.4.2, which hasn't been locked
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at unlock (/usr/lib/node_modules/npm/lib/utils/locker.js:44:11)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/lib/cache/add-named.js:32:5)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/cache/add-named.js:41:20
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/utils/locker.js:22:20
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/node_modules/inflight/inflight.js:22:7
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at Array.forEach (native)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at res (/usr/lib/node_modules/npm/node_modules/inflight/inflight.js:21:9)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at f (/usr/lib/node_modules/npm/node_modules/once/once.js:16:25)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at afterMkdir (/usr/lib/node_modules/npm/lib/cache/get-stat.js:53:14)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/node_modules/mkdirp/index.js:46:53
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! If you need help, you may report this *entire* log,
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! including the npm and node versions, at:
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     <http://github.com/npm/npm/issues>
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: 
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! System Linux 3.2.0-23-generic-pae
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! cwd /vagrant/refinery/ui
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! node -v v0.10.32
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! npm -v 1.4.28
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! Error: Attempt to unlock grunt-contrib-less@~0.9.0, which hasn't been locked
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at unlock (/usr/lib/node_modules/npm/lib/utils/locker.js:44:11)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/lib/cache/add-named.js:32:5)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/cache/add-named.js:41:20
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/utils/locker.js:30:7
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:149:38)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:177:38
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at Object.oncomplete (fs.js:107:15)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! If you need help, you may report this *entire* log,
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! including the npm and node versions, at:
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     <http://github.com/npm/npm/issues>
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: 
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! System Linux 3.2.0-23-generic-pae
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! cwd /vagrant/refinery/ui
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! node -v v0.10.32
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! npm -v 1.4.28
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! Error: Attempt to unlock grunt-contrib-uglify@~0.2.7, which hasn't been locked
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at unlock (/usr/lib/node_modules/npm/lib/utils/locker.js:44:11)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/lib/cache/add-named.js:32:5)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/cache/add-named.js:41:20
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/utils/locker.js:30:7
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:149:38)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:177:38
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at Object.oncomplete (fs.js:107:15)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! If you need help, you may report this *entire* log,
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! including the npm and node versions, at:
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     <http://github.com/npm/npm/issues>
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: 
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! System Linux 3.2.0-23-generic-pae
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! cwd /vagrant/refinery/ui
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! node -v v0.10.32
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! npm -v 1.4.28
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! Error: Attempt to unlock grunt-contrib-watch@~0.5.3, which hasn't been locked
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at unlock (/usr/lib/node_modules/npm/lib/utils/locker.js:44:11)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/lib/cache/add-named.js:32:5)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/cache/add-named.js:41:20
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/lib/utils/locker.js:30:7
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at cb (/usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:149:38)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at /usr/lib/node_modules/npm/node_modules/lockfile/lockfile.js:177:38
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     at Object.oncomplete (fs.js:107:15)
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! If you need help, you may report this *entire* log,
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! including the npm and node versions, at:
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     <http://github.com/npm/npm/issues>
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: 
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! System Linux 3.2.0-23-generic-pae
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! command "/usr/bin/node" "/usr/bin/npm" "install"
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! cwd /vagrant/refinery/ui
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! node -v v0.10.32
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! npm -v 1.4.28
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! 
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! Additional logging details can be found in:
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR!     /vagrant/refinery/ui/npm-debug.log
==> default: notice: /Stage[main]/Ui/Exec[npm_local_modules]/returns: npm ERR! not ok code 0
==> default: err: /Stage[main]/Ui/Exec[npm_local_modules]/returns: change from notrun to 0 failed: /usr/bin/npm install returned 1 instead of one of [0] at /tmp/vagrant-puppet-3/manifests/default.pp:208

Something to do with permissions on .npm folder? The same issue has been reported here #412 .

[hackdna]

Thank you. I've found two threads discussing this error message: npm/npm#5226 and npm/npm#5348. It'd be great if you could investigate further and come up with a fix.

[DevasenaInupakutika]

After some trial and investigation, following these steps solve the issue:

sudo chown -R `whoami` ~/.npm
sudo chown -R `whoami` /usr/local/lib/node_modules

Also, as discussed here: [http://stackoverflow.com/questions/16151018/npm-throws-error-without-sudo], above two steps are as good as changing the permissions on the node and npm directories to be as if they have been installed by current user.

[hackdna]

Thanks for looking into this. I've just provisioned the VM completely from scratch (vagrant destroy then vagrant up) and didn't run into this error. Also, neither ~vagrant/.npm nor /usr/local/lib/node_modules directories exist in the VM. Actually, according to npm/npm#5226 (comment) npm@1.4.28 (the version we are running) solves this issue. Could you try pulling the latest code Refinery code, deleting deployment/Puppetfile.lock, deployment/.librarian, deployment/modules and rerunning provisioning from scratch (vagrant destroy then vagrant up) to check if you still get this error?

[DevasenaInupakutika]

Yes, I could find the npm version while trying to fix this by managing node installation in my system i.e. installed node with NVM as it installs node without root permissions and also allows to install many versions of node to play easily with them.

And I provisioned VM from scratch now which ran without any error. Thanks very much!

However, there is this thing which I observed during this deployment process. Even if I run through the error (npm ERR!) discussed here, I can still open (http://192.168.50.50:8000/) in browser which opens up refinery interface.

[hackdna]

It's great that you were able to provision the VM. Now that you've seen how unreliable this process can be, would you like to try to help us evaluate some alternatives? For example, using Ansible as a provisioner instead of Puppet (issue #416)?

[DevasenaInupakutika]

Sure! I would like do that and update the issue here with my findings.

[hackdna]

This might be of help: http://www.ansible.com/ansible-book

[DevasenaInupakutika]

Thank you.
Where can I find the system dependencies for refinery platform that should be included in ansible's setup playbook? Are they listed in requirements.txt file?

[hackdna]

Yes, Python module dependencies are listed in requirements.txt. All other dependencies are listed in the Puppet manifest (deployment/manifests/default.pp).