Permalink
..
Failed to load latest commit information.
img Add walkthough for deployment on OS X Nov 29, 2016
.gitignore Add walkthough for deployment on OS X Nov 29, 2016
README.md Update walkthrough for deloyment Nov 29, 2016
asrmaster-controller-https.yaml Docker: v2.2 Oct 18, 2016
asrmaster-controller.yaml Docker: v2.2 Oct 18, 2016
asrmaster-service.yaml tools: Add Kubernetes scripts called deploy Jul 7, 2016
asrworker-controller-https.yaml Docker: v2.2 Oct 18, 2016
asrworker-controller.yaml Docker: v2.2 Oct 18, 2016
ca-controller.yaml Docker: v2.2 Oct 18, 2016
ca-service.yaml deploy: internal MongoDB now Jul 14, 2016
cluster_up_linux.sh Kubernetes: Add OS X scripts Nov 24, 2016
cluster_up_osx.sh Add walkthough for deployment on OS X Nov 29, 2016
dig-controller.yaml Docker: v2.2 Oct 18, 2016
dig-service.yaml Upgrade to v2.1: with 3 Djinn services Sep 4, 2016
face-controller.yaml Docker: v2.2 Oct 18, 2016
face-service.yaml Upgrade to v2.1: with 3 Djinn services Sep 4, 2016
imc-controller.yaml Docker: v2.2 Oct 18, 2016
imc-service.yaml Upgrade to v2.1: with 3 Djinn services Sep 4, 2016
imm-controller.yaml Docker: v2.2 Oct 18, 2016
imm-service.yaml deploy: internal MongoDB now Jul 14, 2016
instructions.ipynb Udpate walkthrough for deployment Nov 29, 2016
memcached-controller.yaml Add Kubernetes scripts for Memcached Aug 21, 2016
memcached-service.yaml Add Kubernetes scripts for Memcached Aug 21, 2016
mongo-controller.yaml deploy: internal MongoDB now Jul 14, 2016
mongo-service.yaml Add Kubernetes scripts for Memcached Aug 21, 2016
qa-controller.yaml Kubernetes: Default DNS policy for QA Oct 19, 2016
qa-service.yaml deploy: internal MongoDB now Jul 14, 2016
start_services.sh Minor .sh changes (#133) Nov 23, 2016
web-controller-https.yaml Docker: v2.2 Oct 18, 2016
web-controller.yaml Docker: v2.2 Oct 18, 2016
web-service-https.yaml Create web-service-https.yaml Jul 9, 2016
web-service.yaml Update Kubernetes scripts to claritylab/lucida:v2.0 Aug 30, 2016

README.md

Deploy Lucida using Kubernetes

instructions.ipynb is a complete walkthrough for deplyoment of Lucida on Mac using a Ubuntu 14.04 virtual machine. If you are familiar with Linux commands, however, the following steps should be enough.

Steps

  1. Prerequisites: Docker is installed, port 8080 is not in use, and you have at least 18 GB of disk space and 7 GB of memory. If you deploy on OS X, Virtualbox or VMWare Fusion must be installed. The Docker image contains all the compiled dependencies, ASR models, DNN models, Stanford CoreNLP packagesetc., and make sure your docker allows you to pull an image of 18 GB. The disk usgae will increase as users add data to Lucida. If you need to set memory and CPU limits for Kubernetes, please refer to this.

  2. Run sudo ./cluster_up_<your_os>.sh to create a Kubernetes cluster on a single machine via Docker. If you want to create a cluster with more than one machines, please refer to the official documentation.

  3. Open mongo-controller.yaml and qa-controller.yaml and modify the hostPath fields to point to the directories where you want to store the data for MongoDB and OpenEphyra. Make sure you have write access to the directories you specify.

    If you choose to use Wikipedia as an addition to the user-input knowledge base, move it to the correct directory according to the inline comment in qa-controller.yaml. Otherwise, remove export wiki_indri_index=... from the args field.

    Modify the number of replicas in *-controllers if the default parameter does not suffice.

  4. If you prefer to build the Docker image from the top level Dockerfile rather than pulling from our Dockerhub, you need to modify the image fields of all *-controllers and set up a local Kubernetes container registry.

  5. If you have SSL certificates generated by letsencrypt and want to set up https, please modify the following files according to their inline comments:

    web-controller-https.yaml
    asrmaster-controller-https.yaml
    

    , and then rename the following files:

    mv asrworker-controller-https.yaml asrworker-controller.yaml
    mv asrmaster-controller-https.yaml asrmaster-controller.yaml
    mv web-controller-https.yaml web-controller.yaml
    mv web-service-https.yaml web-service.yaml
    
  6. Run sudo ./start_services.sh to launch all Kubernetes services and pods. It assumes that a local cluster is set up. Pulling the images might take a while, and you may see an error status ImagePullBackOff if there is no space left on the device. To debug, you can run kubectl get service|pod to check the services or pods, kubectl describe pod <pod_name> to see the details (recommended), docker ps | grep <controller_name> followed by docker exec -it <running_container_id> bash to go inside the running containers. For example, if you see "Internal Server Error", you should check the web container, and see the error logs in /usr/local/lucida/lucida/commandcenter/apache/logs/. Also, if MongoDB container is constantly being created without making progress, run sudo netstat -tulpn | grep 27017 and kill the currently running MongoDB instance which also uses the port 27017. This also applies to other containers, e.g. Memcached, qa, etc. whose ports are already used and thus cannot be started.

  7. Open your browser and visit http://localhost:30000 (or https://<YOUR_DOMAIN_NAME>:30000 if you have set up https). It may take up to several minutes for the Apache server to start working, but if it seems to take forever for the index page to show up, please debug as described above.

  8. To destroy the cluster, run docker ps, then stop and rm all the containers related to Kubernetes. The following function may be helpful if you want to stop and remove all the Docker containers.

    function docker-flush(){
      dockerlist=$(docker ps -a -q)
        if [ "${dockerlist}" != "" ]; then
          for d in ${dockerlist}; do
            echo "***** ${d}"
            docker stop ${d} 2>&1 > /dev/null
            docker rm ${d} 2>&1 > /dev/null
            done
          fi
    }