Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

added notifications (twitter, email) for when the mirrors are out of …

…date. Added mailgun logo, added twitter follow link, and fixed the forkme at github image, it was broken, moved it internally. also added another daily cronjob
  • Loading branch information...
commit eaee2803974a4a0ae7fd33ae44b943f52f2b275e 1 parent 1363111
@kencochrane authored
View
2  .gitignore
@@ -1,2 +1,4 @@
*.DS_Store
*.pyc
+environment.json
+environment.txt
View
51 README.rst
@@ -10,18 +10,52 @@ Config
It requires redis in order to cache some of the data. For local development it is assuming it to be running
at localhost:6379 db:1 and no password. see ``config.py`` for more info.
-In order to get the IP address geolocation lookup to work correctly you need to sign up for an account
-from http://ipinfodb.com/register.php and then set an environment variables called ``PYPI_MIRRORS_API_KEY`` with the key they
-give you so you can access the API. If you don't have the env variable set, you will not have access to the geo location information.
+In order to get the IP address geolocation lookup, you need to sign up for an account from http://ipinfodb.com/register.php . If you don't have the env variable set, you will not have access to the geo location information. set IPLOC_API_KEY with the API key they give you.
-For installing the API Key on dotCloud you need to run the following command.
+To get the twitter and email notifications to work correctly you need to create an environment.json file in ``/tmp`` with the variables and values shown below. replace <value> with the real values.
+
+``/tmp/environment.json``::
+
+ {
+ "IPLOC_API_KEY": "<value>",
+ "TWITTER_CONSUMER_KEY" : "<value>",
+ "TWITTER_CONSUMER_SECRET" : "<value>",
+ "TWITTER_ACCESS_KEY" : "<value>",
+ "TWITTER_ACCESS_SECRET" : "<value>",
+ "EMAIL_HOST" : "<value>",
+ "EMAIL_USER" : "<value>",
+ "EMAIL_PASSWORD" : "<value>",
+ "EMAIL_FROM" : "<value>",
+ "EMAIL_TO" : "<value>",
+ "EMAIL_BCC" : "<value>",
+ "EMAIL_TO_ADMIN": "<value>"
+ }
+
+
+For installing the API Key on dotCloud you need to run the following command. replace <value> with the real values.
+
+env variables::
+
+ dotcloud var set pypimirrors \
+ 'IPLOC_API_KEY="<value>"' \
+ 'TWITTER_CONSUMER_KEY="<value>"' \
+ 'TWITTER_CONSUMER_SECRET="<value>"' \
+ 'TWITTER_ACCESS_KEY="<value>"' \
+ 'TWITTER_ACCESS_SECRET="<value>"' \
+ 'EMAIL_HOST="<value>"' \
+ 'EMAIL_USER="<value>"' \
+ 'EMAIL_PASSWORD="<value>"' \
+ 'EMAIL_FROM="<value>"' \
+ 'EMAIL_TO="<value>"' \
+ 'EMAIL_BCC="<value>"' \
+ 'EMAIL_TO_ADMIN="<value>"'
- $ dotcloud var set myapp PYPI_MIRRORS_API_KEY=<api key>
How it works
------------
-The ``pypi_mirrors.py`` script runs via a cron job and outputs a simple web page. That is all.
+The ``pypi_mirrors.py`` script runs via a cron job and puts data into redis. There is one webpage that pull the data from redis and
+displays it. There is a daily cron job that runs and sends out notifications if the mirrors are out of date.
Demo
----
@@ -29,10 +63,9 @@ http://www.pypi-mirrors.org
How to help
-----------
-Pick one of the things on the TODO list and implement it and send a pull request.
+Pick one of the things on the TODO list and implement it and send a pull request.
TODO:
-----
- Create a setup.py and add to PyPI
-- Add notifications to mirror maintainers if their mirror is out of sync.
-- send out twitter notifications when a mirror is out of date.
+- Add better documentation
View
50 config.py
@@ -1,6 +1,13 @@
import os
import json
+# Add non-official mirrors here
+UNOFFICIAL_MIRRORS = [
+ 'pypi.crate.io',
+]
+
+EMAIL_OVERRIDE = None # None or "blah@example.com"
+
def load_config():
# if at dotcloud load the dotcloud settings
dotcloud_config = '/home/dotcloud/environment.json'
@@ -10,14 +17,41 @@ def load_config():
'port': env['DOTCLOUD_CACHE_REDIS_PORT'],
'password': env['DOTCLOUD_CACHE_REDIS_PASSWORD'],
'db': 1,
- 'ip_api_key': env.get('PYPI_MIRRORS_API_KEY', None),
+ 'ip_api_key': env.get('IPLOC_API_KEY', None),
+ 'twitter_consumer_key' : env.get('TWITTER_CONSUMER_KEY', None),
+ 'twitter_consumer_secret' : env.get('TWITTER_CONSUMER_SECRET', None),
+ 'twitter_access_key' : env.get('TWITTER_ACCESS_KEY', None),
+ 'twitter_access_secret' : env.get('TWITTER_ACCESS_SECRET', None),
+ 'email_host' : env.get('EMAIL_HOST', None),
+ 'email_user' : env.get('EMAIL_USER', None),
+ 'email_password' : env.get('EMAIL_PASSWORD', None),
+ 'email_from' : env.get('EMAIL_FROM', None),
+ 'email_to' : env.get('EMAIL_TO', None),
+ 'email_bcc' : env.get('EMAIL_BCC', None),
+ 'email_to_admin': env.get('EMAIL_TO_ADMIN', None),
}
else:
# local config
- ip_api_key = os.getenv('PYPI_MIRRORS_API_KEY')
- return { 'host': 'localhost',
- 'port': 6379,
- 'password': None,
- 'db': 0,
- 'ip_api_key':ip_api_key}
-
+ dotcloud_config = '/tmp/environment.json'
+ if os.path.exists(dotcloud_config):
+ env = json.load(open(dotcloud_config))
+ return { 'host': 'localhost',
+ 'port': 6379,
+ 'password': None,
+ 'db': 0,
+ 'ip_api_key': env.get('IPLOC_API_KEY', None),
+ 'twitter_consumer_key' : env.get('TWITTER_CONSUMER_KEY', None),
+ 'twitter_consumer_secret' : env.get('TWITTER_CONSUMER_SECRET', None),
+ 'twitter_access_key' : env.get('TWITTER_ACCESS_KEY', None),
+ 'twitter_access_secret' : env.get('TWITTER_ACCESS_SECRET', None),
+ 'email_host' : env.get('EMAIL_HOST', None),
+ 'email_user' : env.get('EMAIL_USER', None),
+ 'email_password' : env.get('EMAIL_PASSWORD', None),
+ 'email_from' : env.get('EMAIL_FROM', None),
+ 'email_to' : env.get('EMAIL_TO', None),
+ 'email_bcc' : env.get('EMAIL_BCC', None),
+ 'email_to_admin': env.get('EMAIL_TO_ADMIN', None),
+ }
+ else:
+ print("can't find a local envirornment file here '/tmp/environment.json' ")
+ return None #TODO throw exception?
View
3  crontab
@@ -1,2 +1,3 @@
MAILTO=""
-*/6 * * * * /home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py > /home/dotcloud/output/index.html
+*/6 * * * * /home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py >> /var/log/supervisor/cron.log
+* 12 * * * /home/dotcloud/env/bin/python /home/dotcloud/current/daily.py >> /var/log/supervisor/cron-daily.log
View
44 daily.py
@@ -0,0 +1,44 @@
+from pypimirrors import find_out_of_date_mirrors
+from config import UNOFFICIAL_MIRRORS
+from notification import (update_twitter_status, send_warning_email,
+ send_status_email)
+
+
+def __tweet_outofdate(mirror, last_update):
+ """ Send a tweet saying we have a mirror out of date """
+ status = "{0} is out of date, it was last updated {1}".format(mirror,
+ last_update)
+ update_twitter_status(status)
+
+
+def daily_out_of_date_mirror_check():
+ """ run everything """
+ results = find_out_of_date_mirrors(unofficial_mirrors=UNOFFICIAL_MIRRORS)
+
+ if results:
+ email_message = ""
+ for res in results:
+ email_message += "{0} was last updated {1}\n".format(
+ res.mirror,
+ res.time_diff_human)
+
+ print("{0} is out of date. {1}".format(
+ res.mirror, res.time_diff_human))
+
+ # one tweet for each out of date mirror
+ __tweet_outofdate(res.mirror, res.time_diff_human)
+
+ # one email for all out of date mirrors
+ send_warning_email(email_message)
+ else:
+ print("All is good, sending Good message!")
+ send_status_email("[All Mirrors are up to date]")
+
+
+def run():
+ """ run all of the daily cron jobs."""
+ daily_out_of_date_mirror_check()
+
+
+if __name__ == '__main__':
+ run()
View
99 notification.py
@@ -0,0 +1,99 @@
+import tweepy
+import smtplib
+
+from config import load_config, EMAIL_OVERRIDE
+
+CONFIG = load_config()
+
+def prepare_twitter_message(status):
+ """ shrink to the right size and add link to site. """
+ link = "http://www.pypi-mirrors.org"
+ link_len = len(link) + 4
+ message_len = 140 - link_len
+ status_new = status[:message_len]
+ if len(status) > message_len:
+ status_new += "..."
+ status_new += " {0}".format(link)
+ return status_new
+
+
+def update_twitter_status(status):
+ """ update the twitter account's status """
+
+ consumer_key=CONFIG.get('twitter_consumer_key')
+ consumer_secret=CONFIG.get('twitter_consumer_secret')
+
+ access_token=CONFIG.get('twitter_access_key')
+ access_token_secret=CONFIG.get('twitter_access_secret')
+
+ message = prepare_twitter_message(status)
+
+ auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
+ auth.set_access_token(access_token, access_token_secret)
+ api = tweepy.API(auth)
+ api.update_status(message)
+
+
+def send_warning_email(message):
+ """ send a message saying a mirror(s) is out of date. """
+ email_to = CONFIG.get('email_to')
+ email_from = CONFIG.get('email_from')
+ email_template = '''Subject: [pypi-mirrors] Mirror is out of Date Notice
+
+ This is an automated email from http://www.pypi-mirrors.org to let you
+ know that the following mirrors are out of date.
+
+ {message}
+
+ --
+ This automated message is sent to you by http://www.pypi-mirrors.org If you no
+ longer want to receive these emails, please contact Ken Cochrane (@KenCochrane) on twitter
+ or reply to this email.
+ '''
+ email_body = email_template.format(message=message)
+
+ send_email(email_body, email_to, email_from)
+
+
+def send_status_email(message):
+ """ send a daily status message """
+ email_to = CONFIG.get('email_to_admin')
+ email_from = CONFIG.get('email_from')
+ email_template = '''Subject: [pypi-mirrors] Mirrors are all up to date
+
+ This is an automated email from http://www.pypi-mirrors.org to let you
+ know that the following mirrors are all up to date.
+
+ {message}
+ --
+ This automated message is sent to you by http://www.pypi-mirrors.org If you no
+ longer want to receive these emails, please contact Ken Cochrane (@KenCochrane) on twitter
+ or reply to this email.
+ '''
+
+ email_body = email_template.format(message=message)
+
+ send_email(email_body, email_to, email_from)
+
+
+def send_email(email_body, email_to, email_from):
+ """ Send an email using the configuration provided """
+ email_host = CONFIG.get('email_host')
+ email_user = CONFIG.get('email_user')
+ email_password = CONFIG.get('email_password')
+ email_bcc = CONFIG.get('email_bcc')
+
+ if EMAIL_OVERRIDE:
+ print 'Over-riding email with {0}.'.format(EMAIL_OVERRIDE)
+ email = EMAIL_OVERRIDE
+ else:
+ email = email_to
+
+ print("email to {0} , bcc: {1}; from {2}".format(email, email_bcc, email_from))
+ smtp = smtplib.SMTP(email_host)
+ smtp.starttls()
+ smtp.login(email_user, email_password)
+ smtp.sendmail(email_from, [email, email_bcc], email_body)
+ smtp.quit()
+
+
View
15 postinstall
@@ -1,9 +1,16 @@
#!/bin/sh
+echo "HOSTNAME=$(cat /etc/hostname)"
+case "$(cat /etc/hostname)" in
+*-mirror-0)
+echo "We are on mirror-0; executing specific management commands..."
+
echo "Install crontab"
crontab ~/current/crontab
-echo "Make output directory"
-mkdir -p ~/output
-
echo "Run script to get updated information"
-/home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py > /home/dotcloud/output/index.html
+/home/dotcloud/env/bin/python /home/dotcloud/current/pypi_mirrors.py >> /var/log/supervisor/cron
+;;
+*)
+echo "We are not on mirror-0; skipping specific management commands..."
+;;
+esac
View
6 pypi_mirrors.py
@@ -6,11 +6,7 @@
get_connection, store_page_data, find_number_of_packages,
get_location_for_mirror)
-# Add non-official mirrors here
-UNOFFICIAL_MIRRORS = [
- 'pypi.crate.io',
-]
-
+from config import UNOFFICIAL_MIRRORS
def process_results(results):
""" process the results and gather data """
View
6 requirements.txt
@@ -1,7 +1,7 @@
-pypi-mirrors==0.0.3
+pypi-mirrors==0.0.5
redis
hiredis
-Jinja2
requests
lxml
-flask
+flask
+tweepy
View
BIN  static/img/forkme.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN  static/img/mailgun.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
BIN  static/img/mailgun_small.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View
31 templates/index.html
@@ -22,22 +22,31 @@
<ul class="nav">
<li><a href="http://pypi.python.org" target="_new">PyPI Home</a></li>
<li><a href="http://www.python.org" target="_new">Python.org</a></li>
- </ul>
+ </ul>
</div>
</div>
</div>
-
+
<!-- fork me on github -->
- <a target='_new' href="https://github.com/kencochrane/pypi-mirrors"><img style="position: absolute; top: 40px; right: 0; border: 0;" src="https://a248.e.akamai.net/assets.github.com/img/e6bef7a091f5f3138b8cd40bc3e114258dd68ddf/687474703a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f7265645f6161303030302e706e67" alt="Fork me on GitHub"></a>
+ <a target='_new' href="https://github.com/kencochrane/pypi-mirrors"><img style="position: absolute; top: 40px; right: 0; border: 0;" src="static/img/forkme.png" alt="Fork me on GitHub"></a>
<div class="container-fluid">
<div class="row-fluid">
- <div class="span12">
+ <div class="span8">
<div class="page-header">
<h1>PyPI Mirror Status</h1>
<p>Here is a list of the PyPI mirrors and the last time they were updated</p>
- </div>
-
+ </div> <!-- page header -->
+ </div>
+
+ <div class="span4">
+ <a href="https://twitter.com/PyPIMirrors" class="twitter-follow-button" data-show-count="false" data-size="large">Follow @PyPIMirrors</a>
+ </div>
+ </div>
+
+ <div class="row-fluid">
+ <div class="span12">
+
<table class="table table-bordered table-striped">
<thead><tr><th>Mirror</th><th>Location</th><th># of Packages</th><th>Last update</th><th>Age</th><th>Response Time (ms)*</th><th>Status</th></tr></thead>
<tbody>
@@ -63,7 +72,7 @@
</tbody>
<tfoot><tr><td colspan='7'>* Response time from Virginia, US</td></tr></tfoot>
</table>
- </div>
+ </div> <!-- span12 -->
</div>
<div class="row-fluid">
<div class="span3">
@@ -80,14 +89,15 @@
<div class="span9"></div>
</div>
<div class="row-fluid">
- <div class="span10">
+ <div class="span9">
Page last updated at {{date_now}} <br />
Built by: <a target='_new' href='http://twitter.com/kencochrane'>@KenCochrane</a><br />
Built with:
<a target='_new' href='https://github.com/kencochrane/pypi-mirrors'>pypi-mirrors</a>, <a target='_new' href='http://twitter.github.com/bootstrap/'>Bootstrap</a> and <a target='_new' href='http://flask.pocoo.org'>Flask</a>
</div>
- <div class='span2'><a target='_new' href='https://www.dotCloud.com'><img src='static/img/powered-by-dotcloud.png' border='0'/></a></div>
- </div>
+ <div class='span3'><a target='_new' href='https://www.dotCloud.com'><img src='static/img/powered-by-dotcloud.png' border='0'/></a>
+ <a target='_new' href='http://mailgun.net'><img src='static/img/mailgun_small.png' border='0'/></a></div>
+ </div>
<!-- /container -->
</div>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
@@ -97,6 +107,7 @@
$('.sparklines').sparkline();
});
</script>
+<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
</body>
</html>
Please sign in to comment.
Something went wrong with that request. Please try again.