Initial version of GCE timeout sample #1

Merged
merged 27 commits into from Dec 31, 2012

Projects

None yet

4 participants

@briandorsey
Contributor

Compute folks, please review.

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+`main.py`: Change the value of `GCE_PROJECT_ID` to your project id which has GCE enabled.
+
+Give your App Engine application's service account `edit` access to your GCE project.
+
+ * Log into the App Engine Admin Console.
+ * Click on the application you want to authorize.
+ * Click on Application Settings under the Administration section on the left-hand side.
+ * Copy the value under Service Account Name. This is the service account name of your application, in the format application-id@appspot.gserviceaccount.com. If you are using an App Engine Premier Account, the service account name for your application is in the format application-id.example.com@appspot.gserviceaccount.com.
+ * Use the Google APIs Console to add the service account name of the app as a team member to the project. Give the account `edit` permission.
+
+
+Verify
+------
+
+> As long as PRETEND_MODE is set to `True` (the default) in `main.py`, the application will only log deletes.
@fredsa
fredsa Dec 28, 2012 Contributor

Suggest DRY_RUN instead of PRETEND_MODE

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+
+import datetime
+import httplib2
+import logging
+from pprint import pformat
+
+import jinja2
+import webapp2
+from apiclient.discovery import build
+from google.appengine.api import memcache
+from oauth2client.appengine import AppAssertionCredentials
+
+SAMPLE_NAME = 'Instance timeout helper'
+# Be careful, this application will delete instances unless they're tagged
+# with one of the SAFE_TAGS below.
+GCE_PROJECT_ID = 'briandpe-api'
@fredsa
fredsa Dec 28, 2012 Contributor

perhaps this should be 'replace-with-your-compute-engine-project-id'

@fredsa fredsa commented on an outdated diff Dec 28, 2012
@@ -0,0 +1,21 @@
+application: REPLACE-WITH-YOUR-APP-ID
@fredsa
fredsa Dec 28, 2012 Contributor

Suggest 'replace-with-your-app-id' to reinforce the fact that app ids are lower case

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+runtime: python27
+api_version: 1
+threadsafe: yes
+
+handlers:
+- url: /favicon\.ico
+ static_files: favicon.ico
+ upload: favicon\.ico
+
+- url: .*
+ script: main.app
+ login: admin
+
+libraries:
+- name: webapp2
+ version: "2.5.2"
@fredsa
fredsa Dec 28, 2012 Contributor

using 'latest' here might be better for a sample app

@fredsa fredsa commented on an outdated diff Dec 28, 2012
@@ -0,0 +1,4 @@
+cron:
+- description: check for expired Compute Engine instances to delete
+ url: /cron/delete
+ schedule: every 1 hours from 00:55 to 23:59
@fredsa
fredsa Dec 28, 2012 Contributor

Can we make start/end times less random?

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+def annotate_instances(instances):
+ """loops through the instances and adds exclusion, age and timeout"""
+ for instance in instances:
+ # set _excluded
+ excluded = False
+ for tag in instance.get('tags', []):
+ if tag.lower() in SAFE_TAGS:
+ excluded = True
+ break
+ instance['_excluded'] = excluded
+
+ # set _age and _timeout_expired (never True for _excluded instances)
+ creation = parse_iso8601tz(instance['creationTimestamp'])
+ now = datetime.datetime.now()
+ delta = now - creation
+ instance['_age'] = delta.seconds / 60
@fredsa
fredsa Dec 28, 2012 Contributor

suggest _age_hours instead

@fredsa fredsa and 1 other commented on an outdated diff Dec 28, 2012
+
+import jinja2
+import webapp2
+from apiclient.discovery import build
+from google.appengine.api import memcache
+from oauth2client.appengine import AppAssertionCredentials
+
+SAMPLE_NAME = 'Instance timeout helper'
+# Be careful, this application will delete instances unless they're tagged
+# with one of the SAFE_TAGS below.
+GCE_PROJECT_ID = 'briandpe-api'
+TIMEOUT = 60 * 8 # in minutes, defaulting to 8 hours
+SAFE_TAGS = "production safetag".lower().split()
+# In pretend mode, deletes are only logged. Set this to False after you've
+# double-checked the status page and you're ready to enable the deletes.
+PRETEND_MODE = True
@fredsa
fredsa Dec 28, 2012 Contributor

Recommend moving all these config items into settings.py and then pulling them in via google.appengine.api.lib_config:
https://developers.google.com/appengine/docs/python/tools/appengineconfig?hl=en#Configuring_Your_Own_Modules

@briandorsey
briandorsey Dec 28, 2012 Contributor

Thanks for the pointer to lib_config. It's very nice.

I tried a version with lib_config (08c0ab9) , but I think the extra layer of abstraction is a bit too much for a sample. Also, it becomes attractive to change the defaults right in main.py, but have them overwritten by the values in settings.py.

@fredsa
fredsa Dec 29, 2012 Contributor

You could have None as the default in main.py, and explicitly put defaults
in settings.py only.

Fred on my Android
On Dec 28, 2012 3:39 PM, "Brian Dorsey" notifications@github.com wrote:

In main.py:

+import jinja2
+import webapp2
+from apiclient.discovery import build
+from google.appengine.api import memcache
+from oauth2client.appengine import AppAssertionCredentials
+
+SAMPLE_NAME = 'Instance timeout helper'
+# Be careful, this application will delete instances unless they're tagged
+# with one of the SAFE_TAGS below.
+GCE_PROJECT_ID = 'briandpe-api'
+TIMEOUT = 60 * 8 # in minutes, defaulting to 8 hours
+SAFE_TAGS = "production safetag".lower().split()
+# In pretend mode, deletes are only logged. Set this to False after you've
+# double-checked the status page and you're ready to enable the deletes.
+PRETEND_MODE = True

Thanks for the pointer to lib_config. It's very nice.

I tried a version with lib_config (08c0ab908c0ab9)
, but I think the extra layer of abstraction is a bit too much for a
sample. Also, it becomes attractive to change the defaults right in
main.py, but have them overwritten by the values in settings.py.


Reply to this email directly or view it on GitHubhttps://github.com/GoogleCloudPlatform/compute-appengine-timeout-python/pull/1/files#r2518886.

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+
+def list_instances():
+ """returns a list of dictionaries containing GCE instance data"""
+ request = compute.instances().list(project=GCE_PROJECT_ID)
+ response = request.execute()
+ instances = response.get('items', [])
+ annotate_instances(instances)
+ return instances
+
+
+class MainHandler(webapp2.RequestHandler):
+ """index handler, displays app configuration and instance data"""
+ def get(self):
+ instances = list_instances()
+
+ config = {}
@fredsa fredsa commented on an outdated diff Dec 28, 2012
+ else:
+ logging.info("DELETE: %s", name)
+ request = compute.instances().delete(project=GCE_PROJECT_ID,
+ instance=name)
+ response = request.execute()
+ logging.info(response)
+
+
+class DeleteHandler(webapp2.RequestHandler):
+ """delete handler - HTTP endpoint for the GAE cron job"""
+ def get(self):
+ delete_expired_instances()
+
+
+app = webapp2.WSGIApplication([
+ ('/', MainHandler),
@fredsa
fredsa Dec 28, 2012 Contributor

recommend putting '/' after the more specific handlers

@fredsa fredsa commented on an outdated diff Dec 28, 2012
+# Be careful, this application will delete instances unless they're tagged
+# with one of the SAFE_TAGS below.
+GCE_PROJECT_ID = 'briandpe-api'
+TIMEOUT = 60 * 8 # in minutes, defaulting to 8 hours
+SAFE_TAGS = "production safetag".lower().split()
+# In pretend mode, deletes are only logged. Set this to False after you've
+# double-checked the status page and you're ready to enable the deletes.
+PRETEND_MODE = True
+
+# Obtain App Engine AppAssertion credentials and authorize HTTP connection.
+# https://developers.google.com/appengine/docs/python/appidentity/overview
+credentials = AppAssertionCredentials(
+ scope='https://www.googleapis.com/auth/compute')
+HTTP = credentials.authorize(httplib2.Http(memcache))
+
+compute = build('compute', 'v1beta13', http=HTTP)
@fredsa
fredsa Dec 28, 2012 Contributor

Can you explain v1beta13 in a comment?

@dhermes dhermes commented on an outdated diff Dec 28, 2012
+-------------
+
+1. A Google API project with the Google Compute Engine API enabled.
+2. Admin rights on an App Engine application.
+3. The App Engine Python SDK installed on your computer.
+
+
+Setup
+-----
+
+Clone the git repository for this project to your computer:
+
+ $ git clone REPOSITORY-URL
+
+Install [Goolge API Python Client with dependencies for App Engine](http://code.google.com/p/google-api-python-client/downloads/list).
@dhermes
dhermes Dec 28, 2012

Typo "Goolge API"

@dhermes dhermes commented on an outdated diff Dec 28, 2012
@@ -0,0 +1,12 @@
+indexes:
@dhermes
dhermes Dec 28, 2012

Does this really need to be included if no indexes are created?

@dhermes dhermes commented on an outdated diff Dec 28, 2012
+import logging
+from pprint import pformat
+
+import jinja2
+import webapp2
+from apiclient.discovery import build
+from google.appengine.api import memcache
+from oauth2client.appengine import AppAssertionCredentials
+
+SAMPLE_NAME = 'Instance timeout helper'
+# Be careful, this application will delete instances unless they're tagged
+# with one of the SAFE_TAGS below.
+GCE_PROJECT_ID = 'replace-with-your-compute-engine-project-id'
+TIMEOUT = 60 * 8 # in minutes, defaulting to 8 hours
+SAFE_TAGS = "production safetag".lower().split()
+# In pretend mode, deletes are only logged. Set this to False after you've
@dhermes
dhermes Dec 28, 2012

Should change this comment "In pretend mode" to reflect the change in the constant name to DRY_RUN

@dhermes dhermes commented on an outdated diff Dec 28, 2012
+ data['title'] = SAMPLE_NAME
+ data['instances'] = instances
+ data['raw_instances'] = pformat(instances)
+
+ template = jinja_environment.get_template('index.html')
+ self.response.out.write(template.render(data))
+
+
+def delete_expired_instances():
+ """logs all expired instances, calls delete API when not DRY_RUN"""
+ instances = list_instances()
+
+ # filter instances, keep only expired instances
+ instances = [i for i in instances if i['_timeout_expired']]
+
+ logging.info("delete cron: %s instance%s to delete",
@dhermes
dhermes Dec 28, 2012

Inconsistent use of single quote and double quote.

@dhermes dhermes commented on an outdated diff Dec 28, 2012
+ """return a datetime object for a string in ISO 8601 format.
+
+ This function parses strings in exactly this format:
+ '2012-12-26T13:31:47.823-08:00'
+
+ Sadly, datetime.strptime's %z format is unavailable on many platforms,
+ so we can't use a single strptime() call.
+ """
+
+ dt = datetime.datetime.strptime(date_string[:-6],
+ '%Y-%m-%dT%H:%M:%S.%f')
+
+ # parse the timezone offset separately
+ delta = datetime.timedelta(minutes=int(date_string[-2:]),
+ hours=int(date_string[-5:-3]))
+ if date_string[-6:-5] == u'-':
@dhermes
dhermes Dec 28, 2012

You don't need a slice here, nor a unicode u'-' for comparison.

@dhermes

Using lowercase as global is a bit scary, especially because you end up using it in MainHandler.get

briandorsey added some commits Dec 28, 2012
@briandorsey briandorsey fix quoting inconsistency; simplify parse_iso8601tz() implementation a641a66
@briandorsey briandorsey convert from lib_config to a config dictionary
lib_config is nice, but I think the extra layer of abstraction is too
much for a sample. Also, it's too attractive to change the defaults right
in main.py, but have them overwritten by the values in settings.py.
8026624
@briandorsey briandorsey fix bug in date calculation and corrected _age_hours --> _age_minutes f4fab25
@briandorsey briandorsey removed index.yaml - no indexes 8be4366
@marcacohen marcacohen was assigned Dec 28, 2012
@marcacohen marcacohen commented on an outdated diff Dec 30, 2012
+from pprint import pformat
+
+import jinja2
+import webapp2
+from apiclient.discovery import build
+from google.appengine.api import memcache
+from oauth2client.appengine import AppAssertionCredentials
+
+SAMPLE_NAME = 'Instance timeout helper'
+
+CONFIG= {
+ # In DRY_RUN mode, deletes are only logged. Set this to False after you've
+ # double-checked the status page and you're ready to enable the deletes.
+ 'DRY_RUN': True,
+
+ # Be careful, this application will delete all instances in the project
@marcacohen
marcacohen Dec 30, 2012 Contributor

This comment seems to apply to the next dict key (SAFE_TAGS). Perhaps this should be moved to be adjacent to that def and a comment could be added here re: where to find the project id?

@marcacohen marcacohen commented on the diff Dec 30, 2012
+ 'SAFE_TAGS': ['production', 'safetag'],
+
+ # Instances are deleted after they have been running for TIMEOUT minutes.
+ 'TIMEOUT': 60 * 8, # in minutes, defaulting to 8 hours
+}
+CONFIG['SAFE_TAGS'] = [t.lower() for t in CONFIG['SAFE_TAGS']]
+
+# Obtain App Engine AppAssertion credentials and authorize HTTP connection.
+# https://developers.google.com/appengine/docs/python/appidentity/overview
+credentials = AppAssertionCredentials(
+ scope='https://www.googleapis.com/auth/compute')
+HTTP = credentials.authorize(httplib2.Http(memcache))
+
+# Build object for the 'v1beta13' version of the GCE API.
+# https://developers.google.com/compute/docs/reference/v1beta13/
+compute = build('compute', 'v1beta13', http=HTTP)
@marcacohen
marcacohen Dec 30, 2012 Contributor

The API version is likely to change often. Should that be pulled out into a separate constant for easy maintenance?

@briandorsey
briandorsey Dec 31, 2012 Contributor

This is a good idea... I'd like to take it bit further and say that we should have a variable name we agree to use in all samples, so it's easy to update them all at once. Something like GCE_API_VERSION? I'll make a separate proposal about this.

@marcacohen marcacohen and 1 other commented on an outdated diff Dec 30, 2012
+# https://developers.google.com/compute/docs/reference/v1beta13/
+compute = build('compute', 'v1beta13', http=HTTP)
+jinja_environment = jinja2.Environment(
+ loader=jinja2.FileSystemLoader('templates'))
+
+
+def annotate_instances(instances):
+ """loops through the instances and adds exclusion, age and timeout"""
+ for instance in instances:
+ # set _excluded
+ excluded = False
+ for tag in instance.get('tags', []):
+ if tag.lower() in CONFIG['SAFE_TAGS']:
+ excluded = True
+ break
+ instance['_excluded'] = excluded
@marcacohen
marcacohen Dec 30, 2012 Contributor

I've seen the underscore convention for instance variables but not for dictionary keys. Is that a standard, or just a personal preference? Not a problem either way, just curious.

@briandorsey
briandorsey Dec 31, 2012 Contributor

It's not a standard. Or even personal preference in general... but in this case I wanted it to be obvious which keys were added by the sample when looking at the overview web page. Could also use a sample-specific prefix, I suppose.

@marcacohen marcacohen commented on an outdated diff Dec 30, 2012
+ for instance in instances:
+ # set _excluded
+ excluded = False
+ for tag in instance.get('tags', []):
+ if tag.lower() in CONFIG['SAFE_TAGS']:
+ excluded = True
+ break
+ instance['_excluded'] = excluded
+
+ # set _age_minutes and _timeout_expired
+ # _timeout_expired is never True for _excluded inst
+ creation = parse_iso8601tz(instance['creationTimestamp'])
+ now = datetime.datetime.now()
+ delta = now - creation
+ instance['_age_minutes'] = delta.seconds / 60
+ if delta.seconds > CONFIG['TIMEOUT'] * 60 and not instance['_excluded']:
@marcacohen
marcacohen Dec 30, 2012 Contributor

This might just be me but I prefer parenthesizing (CONFIG['TIMEOUT'] * 60), not because it's needed (it's not) but because I find it more readabie.

@marcacohen
marcacohen Dec 30, 2012 Contributor

Consider reversing and operands to avoid doing calculation on excluded instances?

@briandorsey briandorsey merged commit b59a5ad into master Dec 31, 2012
@briandorsey briandorsey deleted the initial branch Dec 31, 2012
@marcacohen marcacohen was unassigned by briandorsey Sep 24, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment