Browse files

added notifications (twitter, email) for when the mirrors are out of …

…date. Added mailgun logo, added twitter follow link, and fixed the forkme at github image, it was broken, moved it internally. also added another daily cronjob
  • Loading branch information...
1 parent 1363111 commit eaee2803974a4a0ae7fd33ae44b943f52f2b275e @kencochrane committed Apr 23, 2012
View
2 .gitignore
@@ -1,2 +1,4 @@
*.DS_Store
*.pyc
+environment.json
+environment.txt
View
51 README.rst
@@ -10,29 +10,62 @@ Config
It requires redis in order to cache some of the data. For local development it is assuming it to be running
at localhost:6379 db:1 and no password. see ``config.py`` for more info.
-In order to get the IP address geolocation lookup to work correctly you need to sign up for an account
-from http://ipinfodb.com/register.php and then set an environment variables called ``PYPI_MIRRORS_API_KEY`` with the key they
-give you so you can access the API. If you don't have the env variable set, you will not have access to the geo location information.
+In order to get the IP address geolocation lookup, you need to sign up for an account from http://ipinfodb.com/register.php . If you don't have the env variable set, you will not have access to the geo location information. set IPLOC_API_KEY with the API key they give you.
-For installing the API Key on dotCloud you need to run the following command.
+To get the twitter and email notifications to work correctly you need to create an environment.json file in ``/tmp`` with the variables and values shown below. replace <value> with the real values.
+
+``/tmp/environment.json``::
+
+ {
+ "IPLOC_API_KEY": "<value>",
+ "TWITTER_CONSUMER_KEY" : "<value>",
+ "TWITTER_CONSUMER_SECRET" : "<value>",
+ "TWITTER_ACCESS_KEY" : "<value>",
+ "TWITTER_ACCESS_SECRET" : "<value>",
+ "EMAIL_HOST" : "<value>",
+ "EMAIL_USER" : "<value>",
+ "EMAIL_PASSWORD" : "<value>",
+ "EMAIL_FROM" : "<value>",
+ "EMAIL_TO" : "<value>",
+ "EMAIL_BCC" : "<value>",
+ "EMAIL_TO_ADMIN": "<value>"
+ }
+
+
+For installing the API Key on dotCloud you need to run the following command. replace <value> with the real values.
+
+env variables::
+
+ dotcloud var set pypimirrors \
+ 'IPLOC_API_KEY="<value>"' \
+ 'TWITTER_CONSUMER_KEY="<value>"' \
+ 'TWITTER_CONSUMER_SECRET="<value>"' \
+ 'TWITTER_ACCESS_KEY="<value>"' \
+ 'TWITTER_ACCESS_SECRET="<value>"' \
+ 'EMAIL_HOST="<value>"' \
+ 'EMAIL_USER="<value>"' \
+ 'EMAIL_PASSWORD="<value>"' \
+ 'EMAIL_FROM="<value>"' \
+ 'EMAIL_TO="<value>"' \
+ 'EMAIL_BCC="<value>"' \
+ 'EMAIL_TO_ADMIN="<value>"'
- $ dotcloud var set myapp PYPI_MIRRORS_API_KEY=<api key>
How it works
------------
-The ``pypi_mirrors.py`` script runs via a cron job and outputs a simple web page. That is all.
+The ``pypi_mirrors.py`` script runs via a cron job and puts data into redis. There is one webpage that pull the data from redis and
+displays it. There is a daily cron job that runs and sends out notifications if the mirrors are out of date.
Demo
----
http://www.pypi-mirrors.org
How to help
-----------
-Pick one of the things on the TODO list and implement it and send a pull request.
+Pick one of the things on the TODO list and implement it and send a pull request.
TODO:
-----
- Create a setup.py and add to PyPI
-- Add notifications to mirror maintainers if their mirror is out of sync.
-- send out twitter notifications when a mirror is out of date.
+- Add better documentation
View
50 config.py
@@ -1,6 +1,13 @@
import os
import json
+# Add non-official mirrors here
+UNOFFICIAL_MIRRORS = [
+ 'pypi.crate.io',
+]
+
+EMAIL_OVERRIDE = None # None or "blah@example.com"
+
def load_config():
# if at dotcloud load the dotcloud settings
dotcloud_config = '/home/dotcloud/environment.json'
@@ -10,14 +17,41 @@ def load_config():
'port': env['DOTCLOUD_CACHE_REDIS_PORT'],
'password': env['DOTCLOUD_CACHE_REDIS_PASSWORD'],
'db': 1,
- 'ip_api_key': env.get('PYPI_MIRRORS_API_KEY', None),
+ 'ip_api_key': env.get('IPLOC_API_KEY', None),
+ 'twitter_consumer_key' : env.get('TWITTER_CONSUMER_KEY', None),
+ 'twitter_consumer_secret' : env.get('TWITTER_CONSUMER_SECRET', None),
+ 'twitter_access_key' : env.get('TWITTER_ACCESS_KEY', None),
+ 'twitter_access_secret' : env.get('TWITTER_ACCESS_SECRET', None),
+ 'email_host' : env.get('EMAIL_HOST', None),
+ 'email_user' : env.get('EMAIL_USER', None),
+ 'email_password' : env.get('EMAIL_PASSWORD', None),
+ 'email_from' : env.get('EMAIL_FROM', None),
+ 'email_to' : env.get('EMAIL_TO', None),
+ 'email_bcc' : env.get('EMAIL_BCC', None),
+ 'email_to_admin': env.get('EMAIL_TO_ADMIN', None),
}
else:
# local config
- ip_api_key = os.getenv('PYPI_MIRRORS_API_KEY')
- return { 'host': 'localhost',
- 'port': 6379,
- 'password': None,
- 'db': 0,
- 'ip_api_key':ip_api_key}
-
+ dotcloud_config = '/tmp/environment.json'
+ if os.path.exists(dotcloud_config):
+ env = json.load(open(dotcloud_config))
+ return { 'host': 'localhost',
+ 'port': 6379,
+ 'password': None,
+ 'db': 0,
+ 'ip_api_key': env.get('IPLOC_API_KEY', None),
+ 'twitter_consumer_key' : env.get('TWITTER_CONSUMER_KEY', None),
+ 'twitter_consumer_secret' : env.get('TWITTER_CONSUMER_SECRET', None),
+ 'twitter_access_key' : env.get('TWITTER_ACCESS_KEY', None),
+ 'twitter_access_secret' : env.get('TWITTER_ACCESS_SECRET', None),
+ 'email_host' : env.get('EMAIL_HOST', None),
+ 'email_user' : env.get('EMAIL_USER', None),
+ 'email_password' : env.get('EMAIL_PASSWORD', None),
+ 'email_from' : env.get('EMAIL_FROM', None),
+ 'email_to' : env.get('EMAIL_TO', None),
+ 'email_bcc' : env.get('EMAIL_BCC', None),
+ 'email_to_admin': env.get('EMAIL_TO_ADMIN', None),
+ }
+ else:
+ print("can't find a local envirornment file here '/tmp/environment.json' ")
+ return None #TODO throw exception?
View
3 crontab
@@ -1,2 +1,3 @@
MAILTO=""
-*/6 * * * * /home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py > /home/dotcloud/output/index.html
+*/6 * * * * /home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py >> /var/log/supervisor/cron.log
+* 12 * * * /home/dotcloud/env/bin/python /home/dotcloud/current/daily.py >> /var/log/supervisor/cron-daily.log
View
44 daily.py
@@ -0,0 +1,44 @@
+from pypimirrors import find_out_of_date_mirrors
+from config import UNOFFICIAL_MIRRORS
+from notification import (update_twitter_status, send_warning_email,
+ send_status_email)
+
+
+def __tweet_outofdate(mirror, last_update):
+ """ Send a tweet saying we have a mirror out of date """
+ status = "{0} is out of date, it was last updated {1}".format(mirror,
+ last_update)
+ update_twitter_status(status)
+
+
+def daily_out_of_date_mirror_check():
+ """ run everything """
+ results = find_out_of_date_mirrors(unofficial_mirrors=UNOFFICIAL_MIRRORS)
+
+ if results:
+ email_message = ""
+ for res in results:
+ email_message += "{0} was last updated {1}\n".format(
+ res.mirror,
+ res.time_diff_human)
+
+ print("{0} is out of date. {1}".format(
+ res.mirror, res.time_diff_human))
+
+ # one tweet for each out of date mirror
+ __tweet_outofdate(res.mirror, res.time_diff_human)
+
+ # one email for all out of date mirrors
+ send_warning_email(email_message)
+ else:
+ print("All is good, sending Good message!")
+ send_status_email("[All Mirrors are up to date]")
+
+
+def run():
+ """ run all of the daily cron jobs."""
+ daily_out_of_date_mirror_check()
+
+
+if __name__ == '__main__':
+ run()
View
99 notification.py
@@ -0,0 +1,99 @@
+import tweepy
+import smtplib
+
+from config import load_config, EMAIL_OVERRIDE
+
+CONFIG = load_config()
+
+def prepare_twitter_message(status):
+ """ shrink to the right size and add link to site. """
+ link = "http://www.pypi-mirrors.org"
+ link_len = len(link) + 4
+ message_len = 140 - link_len
+ status_new = status[:message_len]
+ if len(status) > message_len:
+ status_new += "..."
+ status_new += " {0}".format(link)
+ return status_new
+
+
+def update_twitter_status(status):
+ """ update the twitter account's status """
+
+ consumer_key=CONFIG.get('twitter_consumer_key')
+ consumer_secret=CONFIG.get('twitter_consumer_secret')
+
+ access_token=CONFIG.get('twitter_access_key')
+ access_token_secret=CONFIG.get('twitter_access_secret')
+
+ message = prepare_twitter_message(status)
+
+ auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
+ auth.set_access_token(access_token, access_token_secret)
+ api = tweepy.API(auth)
+ api.update_status(message)
+
+
+def send_warning_email(message):
+ """ send a message saying a mirror(s) is out of date. """
+ email_to = CONFIG.get('email_to')
+ email_from = CONFIG.get('email_from')
+ email_template = '''Subject: [pypi-mirrors] Mirror is out of Date Notice
+
+ This is an automated email from http://www.pypi-mirrors.org to let you
+ know that the following mirrors are out of date.
+
+ {message}
+
+ --
+ This automated message is sent to you by http://www.pypi-mirrors.org If you no
+ longer want to receive these emails, please contact Ken Cochrane (@KenCochrane) on twitter
+ or reply to this email.
+ '''
+ email_body = email_template.format(message=message)
+
+ send_email(email_body, email_to, email_from)
+
+
+def send_status_email(message):
+ """ send a daily status message """
+ email_to = CONFIG.get('email_to_admin')
+ email_from = CONFIG.get('email_from')
+ email_template = '''Subject: [pypi-mirrors] Mirrors are all up to date
+
+ This is an automated email from http://www.pypi-mirrors.org to let you
+ know that the following mirrors are all up to date.
+
+ {message}
+ --
+ This automated message is sent to you by http://www.pypi-mirrors.org If you no
+ longer want to receive these emails, please contact Ken Cochrane (@KenCochrane) on twitter
+ or reply to this email.
+ '''
+
+ email_body = email_template.format(message=message)
+
+ send_email(email_body, email_to, email_from)
+
+
+def send_email(email_body, email_to, email_from):
+ """ Send an email using the configuration provided """
+ email_host = CONFIG.get('email_host')
+ email_user = CONFIG.get('email_user')
+ email_password = CONFIG.get('email_password')
+ email_bcc = CONFIG.get('email_bcc')
+
+ if EMAIL_OVERRIDE:
+ print 'Over-riding email with {0}.'.format(EMAIL_OVERRIDE)
+ email = EMAIL_OVERRIDE
+ else:
+ email = email_to
+
+ print("email to {0} , bcc: {1}; from {2}".format(email, email_bcc, email_from))
+ smtp = smtplib.SMTP(email_host)
+ smtp.starttls()
+ smtp.login(email_user, email_password)
+ smtp.sendmail(email_from, [email, email_bcc], email_body)
+ smtp.quit()
+
+
View
15 postinstall
@@ -1,9 +1,16 @@
#!/bin/sh
+echo "HOSTNAME=$(cat /etc/hostname)"
+case "$(cat /etc/hostname)" in
+*-mirror-0)
+echo "We are on mirror-0; executing specific management commands..."
+
echo "Install crontab"
crontab ~/current/crontab
-echo "Make output directory"
-mkdir -p ~/output
-
echo "Run script to get updated information"
-/home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py > /home/dotcloud/output/index.html
+/home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py >> /var/log/supervisor/cron
+;;
+*)
+echo "We are not on mirror-0; skipping specific management commands..."
+;;
+esac
View
6 pypi_mirrors.py
@@ -6,11 +6,7 @@
get_connection, store_page_data, find_number_of_packages,
get_location_for_mirror)
-# Add non-official mirrors here
-UNOFFICIAL_MIRRORS = [
- 'pypi.crate.io',
-]
-
+from config import UNOFFICIAL_MIRRORS
def process_results(results):
""" process the results and gather data """
View
6 requirements.txt
@@ -1,7 +1,7 @@
-pypi-mirrors==0.0.3
+pypi-mirrors==0.0.5
redis
hiredis
-Jinja2
requests
lxml
-flask
+flask
+tweepy
View
BIN static/img/forkme.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN static/img/mailgun.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN static/img/mailgun_small.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
31 templates/index.html
@@ -22,22 +22,31 @@
<ul class="nav">
<li><a href="http://pypi.python.org" target="_new">PyPI Home</a></li>
<li><a href="http://www.python.org" target="_new">Python.org</a></li>
- </ul>
+ </ul>
</div>
</div>
</div>
-
+
<!-- fork me on github -->
- <a target='_new' href="https://github.com/kencochrane/pypi-mirrors"><img style="position: absolute; top: 40px; right: 0; border: 0;" src="https://a248.e.akamai.net/assets.github.com/img/e6bef7a091f5f3138b8cd40bc3e114258dd68ddf/687474703a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork me on GitHub"></a>
+ <a target='_new' href="https://github.com/kencochrane/pypi-mirrors"><img style="position: absolute; top: 40px; right: 0; border: 0;" src="static/img/forkme.png" alt="Fork me on GitHub"></a>
<div class="container-fluid">
<div class="row-fluid">
- <div class="span12">
+ <div class="span8">
<div class="page-header">
<h1>PyPI Mirror Status</h1>
<p>Here is a list of the PyPI mirrors and the last time they were updated</p>
- </div>
-
+ </div> <!-- page header -->
+ </div>
+
+ <div class="span4">
+ <a href="https://twitter.com/PyPIMirrors" class="twitter-follow-button" data-show-count="false" data-size="large">Follow @PyPIMirrors</a>
+ </div>
+ </div>
+
+ <div class="row-fluid">
+ <div class="span12">
+
<table class="table table-bordered table-striped">
<thead><tr><th>Mirror</th><th>Location</th><th># of Packages</th><th>Last update</th><th>Age</th><th>Response Time (ms)*</th><th>Status</th></tr></thead>
<tbody>
@@ -63,7 +72,7 @@
</tbody>
<tfoot><tr><td colspan='7'>* Response time from Virginia, US</td></tr></tfoot>
</table>
- </div>
+ </div> <!-- span12 -->
</div>
<div class="row-fluid">
<div class="span3">
@@ -80,14 +89,15 @@
<div class="span9"></div>
</div>
<div class="row-fluid">
- <div class="span10">
+ <div class="span9">
Page last updated at {{date_now}} <br />
Built by: <a target='_new' href='http://twitter.com/kencochrane'>@KenCochrane</a><br />
Built with:
<a target='_new' href='https://github.com/kencochrane/pypi-mirrors'>pypi-mirrors</a>, <a target='_new' href='http://twitter.github.com/bootstrap/'>Bootstrap</a> and <a target='_new' href='http://flask.pocoo.org'>Flask</a>
</div>
- <div class='span2'><a target='_new' href='https://www.dotCloud.com'><img src='static/img/powered-by-dotcloud.png' border='0'/></a></div>
- </div>
+ <div class='span3'><a target='_new' href='https://www.dotCloud.com'><img src='static/img/powered-by-dotcloud.png' border='0'/></a>
+ <a target='_new' href='http://mailgun.net'><img src='static/img/mailgun_small.png' border='0'/></a></div>
+ </div>
<!-- /container -->
</div>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
@@ -97,6 +107,7 @@
$('.sparklines').sparkline();
});
</script>
+<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
</body>
</html>

0 comments on commit eaee280

Please sign in to comment.