Managing Apps, Machines, and Logs
We strive to make it really easy to run Google App Engine applications over your virtualized clusters, private clouds, or in public clouds. To do this, we give you a simple tool set, called the AppScale Tools, that will handle starting up App Engine apps for you as well as all the services that they need to run at scale. This post details how to use the AppScale Tools and what functions they provide to administer your AppScale cloud.
Of course, if you run into any problems with the AppScale Tools, feel free to join us on IRC (#appscale on freenode.net) and ask away!
The AppScale Tools use a configuration file, that we call an AppScalefile, to let you specify how many machines you want to run over, how to set up API services, and so on. To create an AppScalefile, run "appscale init cloud" if you're deploying over Amazon EC2 or Eucalyptus, or "appscale init cluster" if running over a virtualized cluster. That will create an AppScalefile in the current working directory, which you can edit if you want to change how AppScale is deployed. This produces output similar to the following:
$ appscale init cluster
AppScalefile successfully created! Be sure to customize it for your particular cloud or cluster.
Once you've got your AppScalefile set up, you can start a new AppScale deployment by running "appscale up". This produces output similar to the following:
$ appscale up
Starting AppScale 2.2.0 over a virtualized cluster.
Log in to your head node: ssh -i /Users/you/.appscale/appscale.key root@192.168.10.2
Copying over local copy of AppScale from ~/appscale
Head node successfully initialized at 192.168.10.2. It is now starting up cassandra.
Copying over deployment credentials
Starting AppController at 192.168.10.2
Please wait for the AppController to finish pre-processing tasks.
Please wait for AppScale to prepare your machines for use.
AppController just started
UserAppServer is at 192.168.10.2
Enter your desired admin e-mail address: test@example.com
Enter new password:
Confirm password:
Creating new user account test@example.com
Creating new user account test@192.168.10.2
Your XMPP username is test@192.168.10.2
Granting admin privileges to test@example.com
AppScale successfully started!
View status information about your AppScale deployment at http://192.168.10.2/status
Once you've got AppScale started, you can deploy Google App Engine applications to it. This produces output similar to the following:
$ appscale deploy ~/sample-apps/python/guestbook
Enter your desired e-mail address: test@example.com
Uploading guestbook
We have reserved guestbook for your app
Creating remote directory to copy app into
Tarring application
Copying over application
Please wait for your app to start serving.
Your app can be reached at the following URL: http://192.168.10.2:8080
Of course, the app has to exist at the path you're pointing "deploy" at. If it doesn't exist, you'll get a message like this:
$ appscale deploy ~/baz
Couldn't find an app.yaml or appengine-web.xml file in /Users/you/baz
For Java apps, make sure you point "appscale deploy" at the directory that contains your "war" directory (not the "war" directory itself). If you don't, you'll get the same error message as above.
Just do the same thing as when you deployed your app. It will quickly start serving the newer version of your app.
appscale deploy ~/baz
You can specify which ports to use for you app. Give it the application ID, the HTTP port, and the HTTPS port, like such:
$ appscale relocate guestbook 80 443
Once you decide you no longer want to have AppScale host an application, you can run "appscale remove appname" to remove that app (presuming that your app was called "appname"). This produces output similar to the following:
$ appscale remove guestbook
Are you sure you want to remove this application? (Y/N) yes
Please wait for your app to shut down.
Done shutting down guestbook
If you remove an application that doesn't exist or isn't running, that will fail:
$ appscale remove guestbook2
Are you sure you want to remove this application? (Y/N) yes
The given application is not currently running.
To run your application on port 80/443 do the following:
appscale relocate <app_id> 80 443
You can also use the AppScale Tools to query the status of your AppScale deployment, by running "appscale status". This produces output similar to the following:
$ appscale status
Status of node at 192.168.10.2:
Currently using 6.5 Percent CPU and 66.33 Percent Memory
Hard disk is 7 Percent full
Is currently: load_balancer, shadow, db_master, zookeeper, login, memcache, taskqueue_master, appengine
Database is at 192.168.10.2
Is in cloud: cloud1
Current State: Preparing to run AppEngine apps if needed
Hosting the following apps: none
View status information about your AppScale deployment at http://192.168.10.2/status
AppScale runs over Ubuntu Precise virtual machines. Should you need to log into the machines for any reason, you can run "appscale ssh integer" to log into the machine with id "integer" as root. Since "appscale ssh 0" is the most common machine to log into, you can also use "appscale ssh" as shorthand for "appscale ssh 0". This produces output similar to the following:
$ appscale ssh
Linux appscale-image0 2.6.32-38-server #83-Ubuntu SMP Wed Jan 4 11:26:59 UTC 2012 x86_64 GNU/Linux
Ubuntu 12.04.4 LTS
Welcome to the Ubuntu Server!
* Documentation: http://www.ubuntu.com/server/doc
53 packages can be updated.
45 updates are security updates.
Run 'do-release-upgrade' to upgrade to it.
Welcome to your Vagrant-built virtual machine.
You have new mail.
Last login: Wed Mar 27 15:38:13 2013 from 192.168.10.1
root@appscale-image0:~#
For debugging AppScale (or if you're just curious what AppScale does to host your apps), you can take a look at the logs that AppScale creates. These logs are stored at "/var/log/appscale" on each of your VMs, so you could use "appscale ssh" to log into each of your machines and examine the logs individually there. However, this can be a bit of a hassle, especially if you want to e-mail us the logs to look over, so we provide "appscale logs location" to automatically gather your logs and copy them to "location". This produces output similar to the following:
$ appscale logs ~/Desktop/my-logs
Successfully copied logs to /Users/you/Desktop/my-logs
$ tree ~/Desktop/my-logs/
/Users/you/Desktop/my-logs/
└── 192.168.10.2
└── appscale
├── app___apichecker-19997.log
├── app___apichecker-19998.log
├── app___apichecker-19999.log
├── app___guestbook-20000.log
├── appmanagerserver-49934.log
├── autoscale.log
├── blobstore-6106.log
├── celery-apichecker-9999.log
├── celery-guestbook-9999.log
├── celery_workers
│ ├── apichecker.log
│ └── guestbook.log
├── controller-17443.log
├── datastore_server-4000.log
├── datastore_server-4001.log
├── ejabberd-4369.log
├── haproxy-9999.log
├── iaas_manager-17444.log
├── loadbalancer-8000.log
├── loadbalancer-8001.log
├── loadbalancer-8002.log
├── memcached-11211.log
├── monitr-8003.log
├── neptune_manager-17445.log
├── rabbitmq-5672.log
├── taskqueue-64839.log
├── uaserver-4343.log
└── zoo_keeper-2181.log
3 directories, 27 files
Typically if you're having problems starting AppScale, you'll want to check out the "controller-17443.log" file on each node to make sure that the AppController was able to start up all the needed API services on its own machine. If you're running into problems with your App Engine app, check out "app___appname*.log", where "appname" is the app id you set for your app in your app.yaml or appengine-web.xml file.
If you want to "tail" logs instead of copying them over, you can use "appscale tail id file", where "id" is an integer corresponding to the id of the VM you want to tail from and "file-regex" is the file in "/var/log/appscale" you want to tail. So if you want to tail the AppController logs on your head node, you could run "appscale tail 0 'controller*'" (since "controller-17443.log" matches "controller*"). Since tailing the AppController logs on the head node is the most common use case of this command, just running "appscale tail" is shorthand for "appscale tail 0 'controller*'". Let's demonstrate with an example:
$ appscale tail
/usr/lib/ruby/1.8/openssl/ssl.rb:166: warning: using default DH parameters.
/usr/lib/ruby/1.8/webrick/https.rb:25: warning: SSL session is not started yet.
warning: peer certificate won't be verified in this SSL session
/usr/lib/ruby/1.8/openssl/ssl.rb:166: warning: using default DH parameters.
/usr/lib/ruby/1.8/webrick/https.rb:25: warning: SSL session is not started yet.
scp: /tmp/*.log: No such file or directory
[Wed Mar 27 21:52:36 +0100 2013] UAServer port open on 192.168.10.2 - using it!
[Wed Mar 27 21:52:36 +0100 2013] Starting appengine - pbserver is at [192.168.10.2]
[Wed Mar 27 21:52:36 +0100 2013] CPU used: 6.9, mem used: 92.15
[Wed Mar 27 21:52:36 +0100 2013] Not enough memory is free to spawn up a new python27 AppServer (92.15 memory used > 90.0 maximum)
-- Control-C --
Once you're done running AppScale and you want to tear it down, run "appscale down". If running in a cloud deployment, this terminates the machines that "appscale up" created, but if running in a cluster deployment, this stops all the API services we started when we ran "appscale up". This produces output similar to the following:
$ appscale down
Terminating instances in a virtualized cluster with keyname appscale
Shutting down AppScale API services at 192.168.10.2
Terminated AppScale on 1 machines.
Successfully shut down your AppScale deployment.
This will remove your applications and delete application data.
$ appscale clean
Successfully shut down your AppScale deployment.
If you'd like to use any of the above commands but don't want to have an AppScalefile in your current working directory, you can use the following commands:
appscale init --> N/A (since AppScalefiles aren't used here)
appscale up --> appscale-add-keypair (if on a virtualized cluster), then appscale-run-instances
appscale deploy --> appscale-upload-app
appscale remove --> appscale-remove-app
appscale status --> appscale-describe-instance
appscale logs --> appscale-gather-logs
appscale down --> appscale-terminate-instances
Run the following command with the keyname you can find at the bottom of your AppScalefile:
~/appscale-reset-pwd --keyname <keyname>