New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set up dev server #1560

Closed
ultrasaurus opened this Issue Oct 9, 2017 · 6 comments

Comments

Projects
None yet
5 participants
@ultrasaurus
Contributor

ultrasaurus commented Oct 9, 2017

We'd like to set up a dev instance of OpenOpps with a vanilla config to make it easier for the open source community developers to work with the new US gov team at OPM (Office of Personnel Management). The goal is to have a running instance that always has the latest dev branch. (We used to have this at 18F, but never re-created it when we moved to a separate private-public open source org.)

Background:

  • We support docker for development, see install instructions
  • The server needs a database (postgres) and runs migrations to set up the schema, but we only want to do that when first setting up the instance.
  • I think we want to be able to deploy a new app, but keep the old database. This worked really well before -- developers don't have to keep re-creating accounts and we can persist experimental test data across software changes to reproduce specific issues and then verify that a fix works experimentally, and also have a place where new developers can experiment and understand how the software works. (We would reset the database now and then for our own convenience, usually when we were experimenting with new demo data.)

Current approach:

  • Use AWS, so that we can have simpler deploy using RDS/postgres and hosted redis
  • Only use docker-compose.yml for local development

TO DO:

  • what is the recommended VM size? small instance (512MB) is fine (prior memory issue was mis-diagnosed)
  • run scripts with non-local config (shared config for app and running migrate, init, demo)
  • node_modules should be installed outside of spinning up (mounting a local volume into the VM) the docker image or even better copied into it the Docker image before publishing it to Docker Hub. ok for the image to be large as it should contain all the vendor dependencies for a faster startup time.
  • package-lock.json: open question #1742
  • document steps for setting up a prod instance using docker

Notes:

  • Dockerfile:
    • changed from specific node version to node:carbon so it will pick up security patches and critical fixes
    • added vim, since there isn't a text editor installed on base docker file
    • fixed warnings by setting up user & group, so all is not installed as root
@ultrasaurus

This comment has been minimized.

Contributor

ultrasaurus commented Oct 9, 2017

@rogeruiz @bmcorum do we have documentation anywhere about the min / recommended size for a VM/container running OpenOpps? or would you mind sharing (or pointing to) what we do for the gov instance?

@rogeruiz

This comment has been minimized.

Contributor

rogeruiz commented Oct 10, 2017

We have the Dockerfile and the Docker Compose file people could use as a starting point. Those are development configurations, but we could modify the process to publish production-ready docker images to DockerHub. @ultrasaurus

@lawrencek76

This comment has been minimized.

Member

lawrencek76 commented Oct 12, 2017

@ultrasaurus Currently staging has been running fine on a 512M instance in cloud.gov and prd has 2 1GB instances. So for dev test 512M seems to running just fine. part of producing the docker images should include copying in the node_modules thus not require an attempt to install them on startup that would be unpredictable at best in a production situation. Also I'd look into setting --max-old-space-size node option when running node in the docker container. cloud foundry sets this option appropriately for us when we set
env:
OPTIMIZE_MEMORY: true
In the manifest.yml

If there is an environment variable in the docker container with the current memory allocation we could create a script similar to this one to add it dynamically at start up

@rogeruiz

This comment has been minimized.

Contributor

rogeruiz commented Oct 12, 2017

what is the recommended VM size? with a small instance (512MB), we run out of memory installing node_modules, 2gb was enough to install the app. We want this to be small, but not too small!

For the docker image, we should worry more about runtime, like @lawrencek76 said. The node_modules should be installed outside of spinning up (mounting a local volume into the VM) the docker image or even better copied into it the Docker image before publishing it to Docker Hub. I think it's okay for the image to be large as it should contain all the vendor dependencies for a faster startup time.

@ultrasaurus

This comment has been minimized.

Contributor

ultrasaurus commented Feb 18, 2018

I've updated the top issue description to reflect current approach of deploying on AWS.

In case these are needed in the future, below are my prior notes from experimenting on digital ocean:

docker-machine create --driver digitalocean --digitalocean-size 2gb --digitalocean-access-token $PERSONAL_TOKEN openopps-ocean-test
docker-machine env openopps-ocean
eval $(docker-machine env openopps-ocean)
docker-compose up

then I get an error running the migrations:

db_1          | ERROR:  relation "migrations" does not exist at character 15
db_1          | STATEMENT:  select * from migrations;
app_1         | ERROR:  relation "migrations" does not exist
app_1         | LINE 1: select * from migrations;
@ultrasaurus

This comment has been minimized.

Contributor

ultrasaurus commented Apr 2, 2018

moved open questions and TODOs from WIP PR #1743 to top of this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment