Walkthrough of How I Complete Project Option 1 for 1660
- Build Main App
I built the main application by altering the example application used in https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app. Built this applications container and stored it to gcr with these commands:
a. git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples
b. cd kubernetes-engine-samples/hello-app
c. Changed main application to display the messages that I want in the hello.go file
d. docker build -t REGION-docker.pkg.dev/${PROJECT_ID}/hello-repo/hello-app:v1 .
- Tested other containers for hadoop, sonarqube/sonarscanner, spark, and jupyter.
- For all images other then main application I pulled, tagged, then pushed to gcr
Example Commands for Jupyter Image (Not Used on Main Application): docker pull jupyter/datascience-notebook
docker tag jupyter/datascience-notebook gcr.io/cs1660proj/jupyter
docker push gcr.io/cs1660proj/jupyter
- Created my cluster named my-cluster, set to autopilot a. Command: gcloud container clusters create-auto my-cluster
- Set the project ID a. Command: export PROJECT_ID=test-app1660
- Deployed containers in GCR to my-cluster. I editted the datanode to have two replicas while the rest were editted to have 1 replicas.
For hadoop namenode and datanodes I had to add environment variables that were within this file: https://github.com/big-data-europe/docker-hadoop/blob/master/hadoop.env (First 9 Lines) I deployed these by going to container registry >> deploy >> added ports >> added env variables >> then continued
Add this env variable for namenode: - CLUSTER_NAME=test Add this env variable for datanode: SERVICE_PRECONDITION: "namenode:9870" Example Commands for Jupyter (Variation of these commands were used for all containers except for hadoop datanodes and namenode):
kubectl create deployment jupyter --image=jupyter/datascience-notebook kubectl scale deployment jupyter --replicas=1 kubectl autoscale deployment jupyter --cpu-percent=10 --min=1 --max=5
- Edit yaml files so they use 250m cpu and 1Gi memory so that I had enough for all
- Expose the pods.
Jupyter was exposed to port 8888
Sonarqube: 9000
Main application: 8080
Spark: 8080
Hadoop: 9870:9000
Example Command for Jupyter: kubectl expose deployment jupyter --name=jupyter-service --type=LoadBalancer --port 80 --target-port 8888
*For hadoop I manually created the service so I could expose both ports.
- Update the Main Terminal Application so that it displays the links for all the pods
Commands: kubectl get svc (to see the external ip addresses)
docker build -t us-east1-docker.pkg.dev/${PROJECT_ID}/hello-repo/hello-app:v2 .
docker push us-east1-docker.pkg.dev/${PROJECT_ID}/hello-repo/hello-app:v2
kubectl set image deployment/hello-app hello-app=us-east1-docker.pkg.dev/${PROJECT_ID}/hello-repo/hello-app:v2 (This line was executed after I changed print statements to have the external IP addresses)
How to Use Application:
Navigate to 34.75.201.101
Follow Instructions on page
A CLOSE VARIATION OF EXAMPLE COMMANDS WERE USED FOR ALL CONTAINERS/DEPLOYMENTS/PODS UNLESS I SPECIFIED IT WAS NOT USED FOR A SPECIFIC ONE