Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Ubuntu Xenial in development #3108

Merged
merged 18 commits into from Jun 21, 2019
Merged

Conversation

@hectcastro
Copy link
Contributor

hectcastro commented Jun 12, 2019

Overview

  • Vagrant base box updated for all virtual machines
  • Services responsible for consuming logs and metrics removed from services
  • services virtual machine provisions cleanly
  • Services on services virtual machine are functional
  • tiler virtual machine provisions cleanly
  • Services on tiler virtual machine are functional
  • app virtual machine provisions cleanly
  • Services on app virtual machine are functional
  • worker virtual machine provisions cleanly
  • Services on worker virtual machine are functional
  • ...

Fixes #3075

Demo

Screen Shot 2019-06-12 at 7 10 03 PM

Notes

  • Analysis jobs completed, but almost always returned table cells with 0s
  • Delineate watershed did not work because I do not have the necessary data volume
  • Tile service is not functional
  • CI hangs on one of the Testem tests.

Testing Instructions

  • Destroy all of the local virtual machines and clear out the azavea.* roles
  • Follow the instructions in the README to setup the local development environment
$ vagrant up
$ ./scripts/manage.sh migrate

From within the app virtual machine:

$ cd /vagrant
./scripts/setupdb.sh -b  # These can take a few minutes
./scripts/setupdb.sh -s
  • Interact with the application to determine what functionality is not working properly
hectcastro added 14 commits Jun 12, 2019
- Migrate service to systemd
- Centralize logs in syslog via systemd
- Update dependencies

(Things are broken due to old version of Windshaft.)
- Redis
- Logstash

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

All of the log and metrics aggregation services have been removed. Follow-up tasks exist to replace these with external services.

@@ -12,12 +12,7 @@ A combination of Vagrant 1.6+ and Ansible 1.8+ is used to setup the development
The `app` virtual machine contains an instance of the Django application, `services` contains:

- PostgreSQL
- Pgweb

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

pgweb was not consistently being used, so it was removed from the development environment.

@@ -28,7 +23,6 @@ The `app` virtual machine contains an instance of the Django application, `servi

- Celery
- Docker
- Spark Job Server (container)

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

This service has not been around for a while, and was not removed in this PR.


### Caching

In order to speed up things up, you may want to consider leveraging the `vagrant-cachier` plugin. If installed, it is automatically used by Vagrant.

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

This plugin is no longer actively maintained and is not installed is most environments where this Vagrant setup is instantiated.

@@ -89,7 +68,7 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
worker.vm.hostname = "worker"
worker.vm.network "private_network", ip: ENV.fetch("MMW_WORKER_IP", "33.33.34.20")

worker.vm.synced_folder "src/mmw", "/opt/app/"
worker.vm.synced_folder "src/mmw", "/opt/app"

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

This was done to trim the trailing _ from mount names in df output. Mount names also coincide with systemd targets.

- "libjpeg8-dev=8c*"
- "libgif-dev=4.1.*"
apt:
pkg: ["libcairo2-dev=1.14.*",

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

Ubuntu 16.04 specific package versions.

@@ -2,7 +2,8 @@ server {
listen *:80;
server_name _;

access_log /var/log/nginx/mmw-tiler.access.log logstash_json;
error_log syslog:server=unix:/dev/log,facility=local7,tag=nginx_tiler,severity=error;

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

Write directly to syslog from Nginx.

@@ -17,14 +17,11 @@
from utils.constants import (
ALLOW_ALL_CIDR,
EC2_INSTANCE_TYPES,
GRAPHITE,

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

All of the changes in this file should really not be in this PR because they are staging specific concerns. I ended up removing them because I was interactively going through find/replace.

Actually testing that these changes are working should be deferred to another PR creation/review cycle.

@@ -13,7 +13,6 @@
from django.core.mail.backends.base import BaseEmailBackend
from django.core.exceptions import ImproperlyConfigured

from django_statsd.clients import statsd

This comment has been minimized.

Copy link
@hectcastro

hectcastro Jun 20, 2019

Author Contributor

Statsite/StatsD has been removed, so application level libraries interacting with those services has been removed as well.

src/mmw/apps/core/mail/backends/boto_ses_mailer.py Outdated Show resolved Hide resolved
@hectcastro hectcastro marked this pull request as ready for review Jun 20, 2019
@hectcastro hectcastro requested a review from rajadain Jun 20, 2019
@rajadain

This comment was marked as resolved.

Copy link
Member

rajadain commented Jun 20, 2019

Provisioning in a fresh check-out failed for the worker with the following error:

RUNNING HANDLER [azavea.nginx : Restart Nginx] *********************************
Thursday 20 June 2019  16:55:13 -0400 (0:00:00.104)       0:09:35.197 *********
fatal: [worker]: FAILED! => {"changed": false, "msg": "Unable to start service nginx: Job for nginx.service failed because the control process exited with error code. See \"systemctl status nginx.service\" and \"journalctl -xe\" for details.\n"}

Details:

> vagrant ssh worker -c 'sudo systemctl status nginx.service'
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-06-20 20:55:14 UTC; 1min 47s ago
     Docs: man:nginx(8)
  Process: 3169 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Jun 20 20:55:14 worker systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 20 20:55:14 worker nginx[3169]: nginx: [emerg] unknown log format "logstash_json" in /etc/nginx/sites-enabled/geoprocessing:5
Jun 20 20:55:14 worker nginx[3169]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 20 20:55:14 worker systemd[1]: nginx.service: Control process exited, code=exited status=1
Jun 20 20:55:14 worker systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Jun 20 20:55:14 worker systemd[1]: nginx.service: Unit entered failed state.
Jun 20 20:55:14 worker systemd[1]: nginx.service: Failed with result 'exit-code'.
Connection to 127.0.0.1 closed.

Looking in to journalctl -xe, I see:

-- Unit nginx.service has begun starting up.
Jun 20 20:55:14 worker nginx[3169]: nginx: [emerg] unknown log format "logstash_json" in /etc/nginx/sites-enabled/geoprocessing:5
Jun 20 20:55:14 worker nginx[3169]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 20 20:55:14 worker systemd[1]: nginx.service: Control process exited, code=exited status=1
@rajadain

This comment was marked as resolved.

Copy link
Member

rajadain commented Jun 20, 2019

Succeeded on vagrant reload worker --provision

hectcastro added 2 commits Jun 21, 2019
@hectcastro

This comment has been minimized.

Copy link
Contributor Author

hectcastro commented Jun 21, 2019

Provisioning in a fresh check-out failed for the worker with the following error:

RUNNING HANDLER [azavea.nginx : Restart Nginx] *********************************
Thursday 20 June 2019  16:55:13 -0400 (0:00:00.104)       0:09:35.197 *********
fatal: [worker]: FAILED! => {"changed": false, "msg": "Unable to start service nginx: Job for nginx.service failed because the control process exited with error code. See \"systemctl status nginx.service\" and \"journalctl -xe\" for details.\n"}

I believe this was fixed in 12928c0.

Copy link
Member

rajadain left a comment

+1 tested, everything's working (tested Analysis, Monitoring, Modeling, which exercised Django, Ulmo, Celery, and the Geoprocessing Service) except for the tile service, but that's expected. I think we should detach that from this PR and let #3109 handle it.

image

Thanks for all the explanatory comments in the PR, and also for updating the README to match the new state.

Regarding the CI failure, I don't think it's a specific front-end test, but that the front-end testing doesn't exit when all the tests are run. Feel free to defer that to a later card. With this work in we can begin all the other upgrades.

Excellent work 👏 🙇

@hectcastro

This comment has been minimized.

Copy link
Contributor Author

hectcastro commented Jun 21, 2019

TY. I agree with deferring Windshaft things to #3109. I also opened #3110 to deal with the Testem test suite.

I'm going to take a closer look at the service logging aspects of this PR in another pass, then merge. Thanks for the end-to-end testing.

hectcastro added 2 commits Jun 21, 2019
@hectcastro hectcastro changed the title Add support for Ubuntu Xenial (16.04) Add support for Ubuntu Xenial in development Jun 21, 2019
@hectcastro hectcastro merged commit e70fef2 into develop Jun 21, 2019
@hectcastro hectcastro deleted the feature/hmc/ubuntu-16.04 branch Jun 24, 2019
@rajadain rajadain added the WPF label Jun 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.