Deployment (for testing) failure #285

Closed
alanruttenberg opened this Issue Jan 10, 2017 · 14 comments

Projects

None yet

3 participants

@alanruttenberg
Member
alanruttenberg commented Jan 10, 2017 edited

On OSX Yosemite:

brew cask install Caskroom/cask/vagrant
brew cask install Caskroom/cask/virtualbox
brew install ansible
vagrant up
vagrant ssh

so far so good.

Then, (in box)

cd /var/www/purl.obolibrary.org
make

which results in:

rm -rf temp tests
kwalify -f tools/config.schema.yml config/*.yml \
    | awk '{print} /INVALID/ {status=1} END {exit status}'
bash: kwalify: command not found
mkdir backup
mkdir -p temp/obo/aeo
tools/translate-entries.py config/aeo.yml temp/obo/aeo/.htaccess
Traceback (most recent call last):
  File "tools/translate-entries.py", line 59, in <module>
import argparse, sys, yaml, re
ImportError: No module named 'yaml'
make: *** [temp/obo/aeo/.htaccess] Error 1
@alanruttenberg
Member

In case it matters, the first time I ran vagrant up I got a message saying I needed a new version of virtualbox. Installed that. Second time it complained ansible wasn't installed. Third time "vagrant up" succeeded, with message:

==> default: Checking if box 'ubuntu/trusty64' is up to date...
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
@alanruttenberg
Member

Even though it said box was provisioned, I tried running vagrant provision (I'm at monkey now, though)

Output:

vagrant provision
/opt/vagrant/embedded/gems/gems/vagrant-1.9.0/lib/vagrant/util/platform.rb:24: warning: Insecure world writable dir /opt/local/sbin in PATH, mode 040777
==> default: Running provisioner: ansible...
default: Running ansible-playbook...
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and 
make sure become_method is 'sudo' (default).
This feature will be removed in a 
future release. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
ok: [default]

TASK [Update apt-get cache] ****************************************************
changed: [default]

TASK [Install system-wide packages (not Travis)] *******************************
changed: [default] => (item=[u'ntp', u'git', u'ruby', u'ruby-dev', u'python3'])

TASK [Install system-wide packages (All)] **************************************
changed: [default] => (item=[u'apache2', u'python3-pip'])

TASK [Install Travis command-line] *********************************************
fatal: [default]: FAILED! => {"changed": true, "cmd": ["gem", "install", "travis", "-v", "1.8.0", "--no-rdoc", "--no-ri"], "delta": "0:01:02.071028", "end": "2017-01-10 20:48:15.689382", "failed": true, "rc": 1, "start": "2017-01-10 20:47:13.618354", "stderr": "ERROR:  Error installing travis:\n\tjson requires Ruby version ~> 2.0.", "stdout": "Building native extensions.  This could take a while...", "stdout_lines": ["Building native extensions.  This could take a while..."], "warnings": []}
    to retry, use: --limit @/Users/alanr/repos/purl.obolibrary.org/tools/site.retry

PLAY RECAP *********************************************************************
default                    : ok=4    changed=3    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
@cmungall
Contributor
@jamesaoverton
Member

Thanks for trying this Alan. I don't think anybody else has tried the Vagrant stuff, and I haven't run it in a while either, so there's bound to be some problems.

The travis gem is only used to see if the build is green before updating. You don't need it for a local test, so you can remove those lines from the site.yml and see if the rest works. That gem has given me trouble in the past, and it would be nice to avoid using it entirely.

I kind of like Ansible, and I used it to deploy to the curent PURL server (Debian on EC2). It's nice to use the same deployment tool aimed at the local Vagrant machine, and I made it work in Travis after jumping through some hoops. But I've had some trouble with Vagran file sharing in the past, and using it for Travis tests is slow.

I've had nothing but grief from Docker, and I won't be investing any more time in it until it is much more mature. For something as simple as this, it might work fine, but I leave that task to another brave soul.

@alanruttenberg
Member

Removing the travis install lets vagrant provision complete. Will try the rest now. Thanks! (should I update the instructions after success or do you want to?)

@cmungall
Contributor
@alanruttenberg
Member

make test

In the below, is the first warning/stacktrace something I should worry about?
I think the validate error at the bottom is iao.owl (next alphabetically) but it doesn't say so. "would be nice".

rm -rf temp tests
kwalify -f tools/config.schema.yml config/*.yml \
    | awk '{print} /INVALID/ {status=1} END {exit status}'
/usr/lib/ruby/1.9.1/psych.rb:206:in `parse': (<unknown>): mapping values are not allowed in this context at line 103 column 14 (Psych::SyntaxError)
    from /usr/lib/ruby/1.9.1/psych.rb:206:in `parse_stream'
    from /usr/lib/ruby/1.9.1/psych.rb:289:in `load_stream'
    from /usr/lib/ruby/1.9.1/psych/deprecated.rb:28:in `load_documents'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:299:in `block in _validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:267:in `each'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:267:in `_validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:215:in `validate_files'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:111:in `execute'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/lib/kwalify/main.rb:124:in `main'
    from /var/lib/gems/1.9.1/gems/kwalify-0.7.2/bin/kwalify:13:in `<top (required)>'
    from /usr/local/bin/kwalify:23:in `load'
    from /usr/local/bin/kwalify:23:in `<main>'
config/aeo.yml#0: valid.
config/aero.yml#0: valid.
config/agro.yml#0: valid.
...
config/hp.yml#0: valid.
config/hsapdv.yml#0: valid.
make: *** [validate] Error 1
@jamesaoverton
Member

Yes. kwalify is checking the YAML for each file against a schema, and the check is failing for one of the files. I agree that it's not friendly output. You're usually pretty sure that it's the file you're working is the one that failed validation.

@jamesaoverton
Member

@cmungall I'm not sure what you mean. The Travis tests run kwalify, but they also install Apache (using Ansible), run Apache with the PURL configuration, and check a large portion of the redirects against their targets.

The Travis gem uses their REST API to tell Make whether master is green, so Make can pull and update.

If we trust that master is always green, we don't need the Travis gem. But last week one of our maintainers commited directly to master, so I'm not sure it's safe.

@alanruttenberg
Member

Ok, It's working for me. Thanks!

Doc suggestions (while fresh in my mind)

  • Document ansible dependency
  • Mention vagrant provision
  • Suggest that on OSX brew be used to install vagrant, ansible, virtualbox (commands above)
  • Give command line example to validate one yml file (faster debugging)
  • Mention setting up /etc/hosts to locally test (the raw IP is ok for testing a single file, but fails when testing against ontologies that import other ontologies).

Other suggestions:

  • Set up so that inside the machine there's a git remote which is the outside machine's cloned repo (the one you're running vagrant from). git push/pull local my-branch can then be used to sync.
  • That needs an ssh key from host to be copied to box. Since I don't know vagrant well, I was fumbling around how to transfer a file in :)
  • Document the travis issue, or remove it from the default Site.yml

Thanks again. I found and fixed bugs it my iao.yml, which I was rearranging because it was a mess.

@alanruttenberg
Member

just noticed shared folder. Nice. so nix: - Set up so that inside the machine there's a git remote which is the outside machine's cloned repo (the one you're running vagrant from). git push/pull local my-branch can then be used to sync.

@alanruttenberg
Member

BTW, I committed changes to iao.yml, but they haven't shown up in the last day. Is there something that needs to be poked to refresh the server?

@jamesaoverton jamesaoverton added a commit that referenced this issue Jan 12, 2017
@jamesaoverton jamesaoverton Only require travis gem on production, see #285
Allow latest version of the gem.
715caa8
@jamesaoverton
Member

Thanks for pointing this out!

Updates are automatic and usually take <20 minutes. But the automatic updates were broken by the problem you reported in #287. I fixed the immediale problem in #288, then created an issue for the general problem in #290.

@jamesaoverton
Member

Hi @alanruttenberg. PRs #289, #291, and #292 should address all of your suggestions except the one about an SSH key for Vagrant. This shouldn't be necessary because the files on the host machines are synced into the VM. The new docs make this more clear.

Please take a look at #292.

@cmungall cmungall closed this in 29fc516 Jan 31, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment