Skip to content

Terminate AppServer when process is over soft memory cap #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shatterednirvana merged 3 commits into from
Nov 13, 2012
Merged

Terminate AppServer when process is over soft memory cap #82

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89f8d8c
Merge pull request #81 from AppScale/testing
nlake44 Oct 30, 2012
fe60b49
Having dev_appserver shutdown if over a soft memory cap
nlake44 Nov 1, 2012
703ae80
Forgot a period
nlake44 Nov 1, 2012
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 25 additions & 2 deletions AppServer/google/appengine/tools/dev_appserver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,8 @@
import tempfile
import yaml


#AppScale
import resource



Expand Down Expand Up @@ -185,7 +186,6 @@
MAX_URL_LENGTH = 2047



DEFAULT_ENV = {
'GATEWAY_INTERFACE': 'CGI/1.1',
'AUTH_DOMAIN': 'gmail.com',
Expand Down Expand Up @@ -232,6 +232,16 @@
DEVEL_FAKE_IS_ADMIN_HEADER = 'HTTP_X_APPENGINE_FAKE_IS_ADMIN'
DEVEL_FAKE_IS_ADMIN_RAW_HEADER = 'X-AppEngine-Fake-Is-Admin'

#AppScale
Copy link
Copy Markdown
Contributor

@shatterednirvana shatterednirvana Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the "AppScale" tag?

Copy link
Copy Markdown
Contributor Author

@nlake44 nlake44 Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you know where we made the changes

# Soft cap on memory. If over dev_appserver will stop serving traffic and
# shut down. It will randomly choose to exit based on MAX_RANDOM_TARGET,
# hence the reason it is soft and not hard . Units are in KBs.
SOFT_CAP_MEM = 150000

# Max number for randomly killing the dev_appserver when over the soft
Copy link
Copy Markdown
Contributor

@shatterednirvana shatterednirvana Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this comment. Should this instead be something like "The probability that we should terminate an AppServer that has exceeded its memory limit"?

Copy link
Copy Markdown
Contributor Author

@nlake44 nlake44 Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed.

# memory cap.
MAX_RANDOM_TARGET = 25

class Error(Exception):
"""Base-class for exceptions in this module."""

Expand Down Expand Up @@ -3634,6 +3644,19 @@ def serve_forever(self):
"""Handle one request at a time until told to stop."""
while not self._stopped:
self.handle_request()

# AppScale
# If this process is using too much memory kill it randomly
Copy link
Copy Markdown
Contributor

@shatterednirvana shatterednirvana Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making each component monitor their own memory usage seems unnecessary when we have a process monitoring system (god) already. Furthermore, you rely on god to revive this AppServer after its failed, so it seems as though the proper fix is to just write a god config file with memory thresholds set in it and leave it to god.

Additionally, since god is built for this (and not our AppServer), it has termination conditions that are a lot smarter than this. The condition here is "kill this AppServer if it exceeds X memory one time, depending on a coin flip", which makes a lot less sense than what god would do, "kill this AppServer if it exceeds X memory 3/5 times in a row".

Copy link
Copy Markdown
Contributor Author

@nlake44 nlake44 Nov 1, 2012

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted. This will be addressed with the other task of fixing god.

if resource.getrusage(resource.RUSAGE_SELF).ru_maxrss > SOFT_CAP_MEM:
# Stagger the killing so all processes do not go down at the same
# time. The lower the MAX_RANDOM_TARGET the higher probability it
# will shut down when over the soft memory cap.
rand = random.randint(0,MAX_RANDOM_TARGET)
if rand == 0:
logging.error("Usage of memory exceeded soft cap of " + \
str(SOFT_CAP_MEM) + ". Exiting.")
break

self.server_close()

def stop_serving_forever(self):
Expand Down