Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Complete rewrite of chr for version 2.0.0

  • Loading branch information...
commit cb8092b87f806c16b2accb58042755dd68e9db23 1 parent f7bf69f
@sysr-q authored
Showing with 2,100 additions and 1,344 deletions.
  1. +1 −0  .gitignore
  2. +1 −0  MANIFEST.in
  3. +0 −55 README.md
  4. +54 −0 README.rst
  5. 0  chrd/__init__.py
  6. +0 −294 chrd/chrf.py
  7. +0 −180 chrd/chru.py
  8. +0 −156 chrd/daemon.py
  9. +0 −53 chrd/logger.py
  10. +0 −330 chrd/pysql_wrapper.py
  11. +0 −39 chrd/root_check.py
  12. +0 −33 chrd/schema/schema.sql
  13. +0 −72 chrd/settings.py
  14. +0 −30 chrd/utility.py
  15. +0 −87 chrm.py
  16. +9 −0 chru/__init__.py
  17. +118 −0 chru/chra.py
  18. +2 −0  chru/utility/__init__.py
  19. +14 −11 {chrd → chru/utility}/base62.py
  20. +128 −0 chru/utility/funcs.py
  21. +13 −0 chru/utility/struct.py
  22. +2 −0  chru/version.py
  23. +1 −0  chru/web/__init__.py
  24. +339 −0 chru/web/routes.py
  25. +82 −0 chru/web/settings.py
  26. +257 −0 chru/web/slug.py
  27. 0  {chrd → chru/web}/static/960.css
  28. 0  {chrd → chru/web}/static/flashes.js
  29. +4 −0 {chrd → chru/web}/static/index.js
  30. 0  {chrd → chru/web}/static/jqplot/jqplot.barRenderer.min.js
  31. 0  {chrd → chru/web}/static/jqplot/jqplot.categoryAxisRenderer.min.js
  32. 0  {chrd → chru/web}/static/jqplot/jqplot.cursor.min.js
  33. 0  {chrd → chru/web}/static/jqplot/jqplot.highlighter.min.js
  34. 0  {chrd → chru/web}/static/jqplot/jqplot.pieRenderer.min.js
  35. 0  {chrd → chru/web}/static/jqplot/jqplot.pointLabels.min.js
  36. 0  {chrd → chru/web}/static/jqplot/jquery.jqplot.min.css
  37. 0  {chrd → chru/web}/static/jqplot/jquery.jqplot.min.js
  38. 0  {chrd → chru/web}/static/main.css
  39. +4 −4 {chrd → chru/web}/templates/index.html
  40. 0  {chrd → chru/web}/templates/layout.html
  41. 0  {chrd → chru/web}/templates/stats.html
  42. +153 −0 docs/Makefile
  43. +99 −0 docs/autodoc.rst
  44. +246 −0 docs/conf.py
  45. +14 −0 docs/index.rst
  46. +190 −0 docs/make.bat
  47. +129 −0 docs/settings.rst
  48. +186 −0 docs/usage.rst
  49. +6 −0 requirements.txt
  50. +48 −0 setup.py
View
1  .gitignore
@@ -17,6 +17,7 @@ develop-eggs
.installed.cfg
lib
lib64
+_build
# Installer logs
pip-log.txt
View
1  MANIFEST.in
@@ -0,0 +1 @@
+include README.rst
View
55 README.md
@@ -1,55 +0,0 @@
-chr
-===
-__chr__ is a python based url shortening service which uses sqlite and Flask.
-We wanted to make a clean, simplistic url shortener, that wasn't written in a horrific language like PHP. What else but Python could come to mind?
-
-Features
----
-+ Can shorten several billion (yes!) unique urls to a less than 6 character slug.
-+ Verifies the shrunk URLs are legitimate, to stop abuse.
-+ Uses reCAPTCHA to stop spammers from using the service for evil, not good.
-+ Slugs are the base62 representation of their ID, so they'll work in all browsers.
-+ A live chr is located at [chr.so](http://chr.so)
-
-Dependencies
----
-+ [Python 2.7](http://python.org)
-+ [python-requests](http://docs.python-requests.org/en/latest/) (`pip install requests`)
-+ [Flask](http://flask.pocoo.org/) (`pip install flask`)
-+ [Flask-KVSession](https://github.com/mbr/flask-kvsession) (`pip install flask-kvsession`)
-+ [recaptcha-client](http://pypi.python.org/pypi/recaptcha-client) (`pip install recaptcha-client`)
-
-These ones are required, but __you don't have to worry about them__.
-+ sqlite3 (part of a regular python install)
-+ [jqPlot](http://www.jqplot.com) (bundled with the source)
-
-Notes
----
-+ It's __highly__ recommended by the chr developers that if you're putting this in a production environment (read: _any computer with a public IP_) that you look at the various [Flask deployment](http://flask.pocoo.org/docs/deploying/) options, such as putting it behind nginx, lighttpd, or something.
-+ It's also recommended that you get your server (nginx, or hell, even Apache) serve out the static folder, rather than letting Flask do it.
-+ This will take a while to get fully featured, but we have a lot planned.
-+ [jqPlot](http://www.jqplot.com) comes bundled with chr, which is alright as it's MIT licensed.
-
-Running
----
-1. Clone the repo.
-2. Edit the `settings.py` file, adding in required info etc.
-3. `python chrm.py start`
-`chrm.py` has other launch params, so just `python chrm.py --help` to check them out.
-
-Todo
----
-+ Clean up some logic with shortened slugs.
-+ Move from utility.hash_pass to `werkzeug.security.generate_password_hash()` and `werkzeug.security.check_password_hash()`
-+ Administration panel allowing the owner to search for urls, remove them, edit them, etc.
- + Ability to reserve certain custom urls.
- + Enable or disable various settings from the website.
-
-Author
----
-+ [plausibility](https://github.com/plausibility)
-
-Contributors
----
-+ huey
-+ [Chris Leonello](http://www.jqplot.com) (made jqPlot)
View
54 README.rst
@@ -0,0 +1,54 @@
+chr url shortener
+=================
+
+.. _docs: http://chr.rtfd.org
+
+**chr** (coded under the name ``chru``) is a Python based URL shortening service which uses Flask as a front end, and pysqlw as the SQL backend, to interface with sqlite3.
+
+It can shrink billions of unique URLs with less than 6 characters, run in the background with no human interaction, and it can fly like a bird -- or is that Super Man?
+
+Features
+--------
+
+- Can shorten several billion (yes!) unique urls to a less than 6 character slug.
+- Verifies the shrunk URLs are legitimate, to stop abuse.
+- Uses reCAPTCHA to stop spammers from using the service for evil, not good.
+- Slugs are the base62 representation of their ID, so they'll work in all browsers.
+- A live chr instance is located at `chr.so <http://chr.so>`_.
+
+Dependencies
+------------
+
+- `Python 27 <http://python.org>`_ (``>=2.7`` required because of use of ``argparse`` module)
+- `requests <http://docs.python-requests.org>`_ (``python-requests``)
+- `Flask <http://flask.pocoo.org>`_ (``flask``)
+- `Flask KVSession <https://github.com/mbr/flask-kvsession>`_ (``flask-kvsession``)
+- `recaptcha client <http://pypi.python.org/pypi/recaptcha-client>`_ (``recaptcha-client``)
+- `mattdaemon <http://pypi.python.org/pypi/mattdaemon>`_ (``mattdaemon>=1.1.0``)
+- `pysqlw <http://pypi.python.org/pypi/pysqlw>`_ (``pysqlw>=1.3.0``)
+
+To install all of these: ``pip -r requirements.txt install`` (if installing from source)
+
+Notes
+-----
+
+- It's **highly** recommended by the chr developers that if you're putting this in a production environment (read: *any computer with a public IP*) that you look at the various `Flask deployment <http://flask.pocoo.org/docs/deploying>`_ options, such as putting it behind nginx, lighttpd, or something.
+- It's also recommended that you get your server (nginx, lighttpd, or hell, even Apache) serve out the static folder, rather than letting Flask do it.
+- This will take a while to get fully featured, but we have a lot planned.
+- `jqPlot <http://www.jqplot.com>`_ comes bundled with chr, which is alright as it's MIT licensed.
+
+Running
+-------
+
+Visit the `docs`_ page, and click **Usage** for information on how to run chr.
+
+Author
+------
+
+- `plausibility <https://github.com/plausibility>`_
+
+Contributors
+------------
+
+- huey
+- `Chris Leonello <http://www.jqplot.com>`_ (made jqPlot)
View
0  chrd/__init__.py
No changes.
View
294 chrd/chrf.py
@@ -1,294 +0,0 @@
-# -*- coding: utf-8 -*-
-# chr's flask handler.
-
-# flask stuff
-from flask import Flask, render_template, session, abort, request, g
-from flask import redirect, url_for, flash, make_response, jsonify
-# recaptcha related
-from recaptcha.client import captcha
-# requests for url validation
-import requests
-# python modules
-import re
-import time
-from datetime import datetime
-# chr modules
-import chru
-import settings
-import logger
-import utility
-import daemon
-
-# revision() is in utility.py
-
-app = Flask(__name__)
-app.debug = settings.flask_debug
-app.secret_key = settings.flask_secret_key
-app.jinja_env.globals.update(sorted=sorted)
-app.jinja_env.globals.update(len=len)
-app.jinja_env.globals.update(date_strip_day=utility.date_strip_day)
-
-# List of used routes, so we can't use it as a slug.
-reserved = ('index', 'submit')
-
-@app.route("/index")
-@app.route("/")
-def index():
- return render_template('index.html', recaptcha_key=settings.captcha_public_key)
-
-@app.route("/<slug>")
-def slug_redirect(slug):
- logger.debug("slug:", slug,
- ", valid:", chru.is_valid_slug(slug),
- ", id:", chru.slug_to_id(slug),
- ", exists:", chru.slug_exists(slug))
- if not chru.is_valid_slug(slug) or not chru.slug_exists(slug):
- return redirect(url_for('index'))
- url = chru.url_from_slug(slug)
- chru.add_hit(slug, request.remote_addr, request.user_agent)
- return redirect(url)
-
-@app.route("/<slug>/delete/<delete>")
-def slug_delete(slug, delete):
- logger.debug("delete slug:", slug,
- ", valid:", chru.is_valid_slug(slug),
- ", id:", chru.slug_to_id(slug),
- ", exists:", chru.slug_exists(slug))
- if not chru.is_valid_slug(slug) or not chru.slug_exists(slug):
- return redirect(url_for('index'))
- row = chru.slug_to_row(slug)
- if delete != row['delete']:
- return redirect(url_for('index'))
- chru.delete_slug(slug)
- logger.info("Successfully deleted", settings.flask_url.format(slug=row['short']))
- flash("Successfully deleted {0}".format(settings.flask_url.format(slug=row['short'])), "success")
- return redirect(url_for('index'))
-
-@app.route("/<slug>/stats")
-def slug_stats(slug):
- if not chru.is_valid_slug(slug) or not chru.slug_exists(slug):
- return redirect(url_for('index'))
- row = chru.slug_to_row(slug)
- clicks = utility.sql().where('url', row['id']).get('clicks')
-
- click_info = {
- "platforms": {"unknown": 0},
- "browsers": {"unknown": 0},
- "pd": {}, # clicks per day
- }
- month_ago = int(time.time()) - (60 * 60 * 24 * 30) # Last 30 days
- month_ago_copy = month_ago
- for x in xrange(1, 31):
- # Prepopulate the per day clicks dictionary.
- # This makes life much much easier for everything.
- month_ago_copy += (60 * 60 * 24) # one day
- strf = str(datetime.fromtimestamp(month_ago_copy).strftime('%m/%d'))
- click_info['pd'][strf] = 0
-
- for click in clicks:
- if click['time'] > month_ago:
- # If the click is less than a month old
- strf = str(datetime.fromtimestamp(click['time']).strftime('%m/%d'))
- click_info['pd'][strf] += 1
-
- if len(click['agent']) == 0 \
- or len(click['agent_platform']) == 0 \
- or len(click['agent_browser']) == 0:
- click_info['platforms']['unknown'] += 1
- click_info['browsers']['unknown'] += 1
- continue
-
- platform_ = click['agent_platform']
- browser_ = click['agent_browser']
- if not platform_ in click_info['platforms']:
- click_info['platforms'][platform_] = 1
- else:
- click_info['platforms'][platform_] += 1
- if not browser_ in click_info['browsers']:
- click_info['browsers'][browser_] = 1
- else:
- click_info['browsers'][browser_] += 1
- print click_info
-
- # Yes, formatting in stuff without escaping it, I know, I'm bad!
- # But these are *safe* variables, we know they're good in advance.
- unique = utility.sql().query('SELECT COUNT(DISTINCT `{0}`) AS "{1}" FROM `{2}`'.format('ip', 'unq', settings._SCHEMA_CLICKS))
- unique = unique[0]['unq']
- rpt = len(clicks) - unique
- hits = {
- "all": len(clicks),
- "unique": unique,
- "return": rpt,
- "ratio": str(round(float(unique) / float(len(clicks)), 2))[2:] # (unique / all)% of visitors only come once.
- }
-
- stats = {
- "hits": utility.Struct(**hits),
- "clicks": utility.Struct(**click_info),
- "long": row['long'],
- "long_clip": row['long'],
- "short": row['short'],
- "short_url": settings.flask_url.format(slug=row['short'])
- }
- if len(row['long']) > 30:
- stats['long_clip'] = row['long'][:30] + '...'
- return render_template('stats.html', stats=utility.Struct(**stats))
-
-@app.route("/submit", methods=["GET"])
-def submit_get():
- return redirect(url_for('index'))
-
-@app.route("/submit", methods=["POST"])
-def submit():
- response = captcha.submit(
- request.form['recaptcha_challenge_field'],
- request.form['recaptcha_response_field'],
- settings.captcha_private_key,
- request.remote_addr
- )
- if not response.is_valid:
- data = {
- "error": "true",
- "message": "The captcha typed was incorrect.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
-
- if settings.validate_urls:
- try:
- logger.debug("Sending HEAD request to:", request.form['chr_text_long'])
- head = requests.head(request.form['chr_text_long'], timeout=1)
- logger.debug("Reply:", head.status_code, head.headers)
- if head.status_code == 404:
- raise requests.exceptions.HTTPError
- except requests.exceptions.InvalidSchema as e:
- logger.debug(e)
- data = {
- "error": "true",
- "message": "The URL provided was not valid.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError) as e:
- logger.debug(e)
- data = {
- "error": "true",
- "message": "The URL provided was unable to be found.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- except requests.exception.RequestException as e:
- logger.debug(e)
- data = {
- "error": "true",
- "message": "Something went wrong validating your URL.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- else:
- logger.debug("Matching '{0}' against '{1}'.".format(request.form['chr_text_long'], settings.validate_regex))
- if not re.match(settings.validate_regex, request.form['chr_text_long'], re.I):
- data = {
- "error": "true",
- "message": "The URL provided was not valid.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
-
- if len(request.form['chr_text_long']) > settings.soft_url_cap:
- # Soft cap so that people can't abuse it and waste space.
- data = {
- "error": "true",
- "message": "The given URL was actually <i>too</i> long to shrink.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonfiy(data)
-
- custom_slug = False
- if len(request.form['chr_text_short']) > 0:
- slug_ = request.form['chr_text_short']
- if not re.match('^[\w -]+$', slug_):
- data = {
- "error": "true",
- "message": "The custom URL was invalid. Only alphanumeric characters, dashes, spaces and underscores, please!",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- if len(slug_) > settings._SCHEMA_MAX_SLUG:
- data = {
- "error": "true",
- "message": "The custom URL given is too long. Less than 32 characters, please!",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- if slug_ in reserved:
- data = {
- "error": "true",
- "message": "Sorry, that custom URL is reserved.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- if chru.slug_exists(settings._CUSTOM_CHAR + slug_):
- data = {
- "error": "true",
- "message": "Sorry, that custom URL already exists.",
- "url": "",
- "delete": "",
- "given": request.form['chr_text_long']
- }
- return jsonify(data)
- custom_slug = settings._CUSTOM_CHAR + slug_
-
-
- slug_, id_, delete_, url_, old_ = chru.url_to_slug(request.form['chr_text_long'], ip=request.remote_addr, slug=custom_slug)
- delete_ = settings.flask_url.format(slug=slug_) + "/delete/" + delete_
- delete_ = delete_ if 'chr_check_delete' in request.form \
- and request.form['chr_check_delete'] == "yes" \
- and not old_ \
- else ""
- data = {
- "error": "false",
- "message": "Successfully shrunk your URL!" if not old_ else "URL already shrunk, here it is!",
- "url": settings.flask_url.format(slug=slug_),
- "delete": delete_,
- "given": url_
- }
- return jsonify(data)
-
-def run():
- check_sql()
- app.run(host=settings.flask_host, port=settings.flask_port)
-
-def check_sql():
- # This is here because if it's in utilities, you get
- # an import loop, which is ugly.
- import os
- if os.path.isfile(settings.sql_path):
- return
- with open('chrd/schema/schema.sql', 'rb') as f:
- utility.sql()._cursor.executescript(f.read())
- sl = chru.url_to_slug("https://github.com/PigBacon/chr", slug=settings._CUSTOM_CHAR + "source")
- if sl:
- logger.info("Successfully setup SQL and added first redirect.")
- logger.debug(sl)
-
-if __name__ == "__main__":
- run()
View
180 chrd/chru.py
@@ -1,180 +0,0 @@
-# -*- coding: utf-8 -*-
-# chr's class to convert slugs to urls, urls to slugs, delete slugs, etc.
-
-import time
-import re
-
-import logger
-import base62
-import settings
-import utility
-
-#--------------------
-# THE REAL DEAL (SRS)
-#--------------------
-
-def url_to_slug(url, ip=False, slug=False):
- """Turns a URL into a shortened slug.
- returns (slug, id, delete, url, old,) or False if failure.
- Doesn't impose any character length limits,
- those should be done before calling.
- """
- old_slug = utility.sql().where('long', url).get(settings._SCHEMA_REDIRECTS)
- if len(old_slug) > 0:
- # url already shortened, give that.
- old_slug = old_slug[0]
- return (old_slug['short'], old_slug['id'], old_slug['delete'], old_slug['long'], True,)
- data = {
- "long": url,
- "short": "",
- "delete": make_delete_slug(),
- "ip": ip if ip else ""
- }
- sq = utility.sql()
- row = sq.insert(settings._SCHEMA_REDIRECTS, data)
- if not row:
- logger.debug("Unable to insert:", data)
- return False
- id = sq.row()
- if not slug:
- logger.debug("Got", id, "from slug add:", url)
- slug = id_to_slug(id)
- update = {
- "short": slug
- }
- if utility.sql().where('id', id).update(settings._SCHEMA_REDIRECTS, update):
- slug = utility.sql().where('id', id).get(settings._SCHEMA_REDIRECTS)[0]
- return (slug['short'], slug['id'], slug['delete'], slug['long'], False,)
- else:
- return False
-
-#------------------
-# CONVERSIONS
-#------------------
-
-def slug_to_id(slug):
- """Turns valid slugs into their integer equivalent.
- Custom slugs are found by searching the DB.
- """
- if slug[0] == settings._CUSTOM_CHAR:
- rows = utility.sql().where('short', slug).get(settings._SCHEMA_REDIRECTS)
- if len(rows) != 1:
- return False
- return rows[0]['id']
- if not is_valid_slug(slug):
- return False
- if not isinstance(slug, basestring):
- raise ValueError('Wanted str, got ' + str(type(slug)))
- return base62.saturate(slug)
-
-def id_to_slug(id):
- """Turns an integer into its slug equivalent.
- """
- if not isinstance(id, int):
- raise ValueError('Wanted int, got ' + str(type(id)))
- return base62.dehydrate(id)
-
-def slug_to_row(slug):
- if not is_valid_slug(slug):
- return False
- if slug[0] == settings._CUSTOM_CHAR:
- rows = utility.sql().where('short', slug).get(settings._SCHEMA_REDIRECTS)
- else:
- rows = utility.sql().where('id', slug_to_id(slug)).get(settings._SCHEMA_REDIRECTS)
- return rows[0] if len(rows) == 1 else False
-
-#------------------
-# INFO
-#------------------
-
-def slug_hits(slug):
- """Return the number of clicks a slug has had.
- """
- if not is_valid_slug(slug):
- return 0
- id = slug_to_id(slug)
- rows = utility.sql().where('url', id).get(settings._SCHEMA_CLICKS)
- return len(rows)
-
-def slug_exists(slug):
- """Check if a slug exists.
- Converts slug -> id to check.
- """
- if not is_valid_slug(slug):
- return False
- if slug[0] == settings._CUSTOM_CHAR:
- rows = utility.sql().where('short', slug).get(settings._SCHEMA_REDIRECTS)
- else:
- rows = utility.sql().where('id', slug_to_id(slug)).get(settings._SCHEMA_REDIRECTS)
- return len(rows) == 1
-
-def is_valid_slug(slug):
- """Attempts to saturate a slug, to check validity.
- Custom slugs are assumed to be valid, unless otherwise shown.
- """
- if slug[0] == settings._CUSTOM_CHAR:
- return True
-
- try:
- base62.saturate(slug)
- return True
- except (TypeError, ValueError) as e:
- logger.debug(e)
- return False
-
-def url_from_slug(slug):
- """Get the long URL attached to a slug.
- Returns long URL or False.
- """
- if not is_valid_slug(slug):
- return False
- if not slug_exists(slug):
- return False
- if slug[0] == settings._CUSTOM_CHAR:
- row = utility.sql().where('short', slug).get(settings._SCHEMA_REDIRECTS)[0]
- else:
- row = utility.sql().where('id', slug_to_id(slug)).get(settings._SCHEMA_REDIRECTS)[0]
- return row['long']
-
-#------------------
-# MODIFICATION
-#------------------
-
-def add_hit(slug, ip, ua, time=int(time.time())):
- """Add a hit relating to the given slug.
- Converts slug -> id before adding.
- """
- if not is_valid_slug(slug):
- return False
- id = slug_to_id(slug)
- data = {
- "url": id,
- "ip": ip,
- "time": time,
- "agent": ua.string if not ua.string == None else "",
- "agent_platform": ua.platform if not ua.platform == None else "",
- "agent_browser": ua.browser if not ua.browser == None else "",
- "agent_version": ua.version if not ua.version == None else "",
- "agent_language": ua.language if not ua.language == None else ""
- }
- return utility.sql().insert(settings._SCHEMA_CLICKS, data)
-
-def delete_slug(slug):
- if not is_valid_slug(slug) or not slug_exists(slug):
- return False
- id = slug_to_id(slug)
- rem_slug = utility.sql().where('id', id).delete(settings._SCHEMA_REDIRECTS)
- rem_click = utility.sql().where('url', id).delete(settings._SCHEMA_CLICKS)
- return rem_slug and rem_click
-
-#------------------
-# GENERATORS
-#------------------
-
-def make_delete_slug(length=64):
- """Generate a {length} long string which is
- used as the slug deletion key.
- """
- import string, random
- parts = string.uppercase + string.lowercase + string.digits
- return ''.join(random.choice(parts) for x in range(length))
View
156 chrd/daemon.py
@@ -1,156 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-# Taken from: www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python
-# It's licensed under Public Domain, so we're free to use it.
-
-import sys, os, time, atexit
-from signal import SIGTERM
-
-class Daemon:
- """
- A generic daemon class.
-
- Usage: subclass the Daemon class and override the run() method
- """
- def __init__(self, pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
- self.stdin = stdin
- self.stdout = stdout
- self.stderr = stderr
- self.pidfile = pidfile
-
- def daemonize(self):
- """
- do the UNIX double-fork magic, see Stevens' "Advanced
- Programming in the UNIX Environment" for details (ISBN 0201563177)
- http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
- """
- try:
- pid = os.fork()
- if pid > 0:
- # exit first parent
- sys.exit(0)
- except OSError, e:
- sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
- sys.exit(1)
-
- # decouple from parent environment
- os.chdir("/")
- os.setsid()
- os.umask(0)
-
- # do second fork
- try:
- pid = os.fork()
- if pid > 0:
- # exit from second parent
- sys.exit(0)
- except OSError, e:
- sys.stderr.write("fork #2 failed: %d (%s)\n" % (e.errno, e.strerror))
- sys.exit(1)
-
- # redirect standard file descriptors
- sys.stdout.flush()
- sys.stderr.flush()
- si = file(self.stdin, 'r')
- so = file(self.stdout, 'a+')
- se = file(self.stderr, 'a+', 0)
- os.dup2(si.fileno(), sys.stdin.fileno())
- os.dup2(so.fileno(), sys.stdout.fileno())
- os.dup2(se.fileno(), sys.stderr.fileno())
-
- # write pidfile
- atexit.register(self.delpid)
- pid = str(os.getpid())
- file(self.pidfile,'w+').write("%s\n" % pid)
-
- def delpid(self):
- os.remove(self.pidfile)
-
- def start(self, daemonize=True):
- """
- Start the daemon
- """
- # Check for a pidfile to see if the daemon already runs
- try:
- pf = file(self.pidfile,'r')
- pid = int(pf.read().strip())
- pf.close()
- except IOError:
- pid = None
-
- if pid:
- message = "pidfile %s already exist. Daemon already running?\n"
- sys.stderr.write(message % self.pidfile)
- sys.exit(1)
-
- if daemonize:
- # Start the daemon
- self.daemonize()
- self.run()
-
- def stop(self):
- """
- Stop the daemon
- """
- # Get the pid from the pidfile
- try:
- pf = file(self.pidfile,'r')
- pid = int(pf.read().strip())
- pf.close()
- except IOError:
- pid = None
-
- if not pid:
- message = "pidfile %s does not exist. Daemon not running?\n"
- sys.stderr.write(message % self.pidfile)
- return # not an error in a restart
-
- # Try killing the daemon process
- try:
- while 1:
- os.kill(pid, SIGTERM)
- time.sleep(0.1)
- except OSError, err:
- err = str(err)
- if err.find("No such process") > 0:
- if os.path.exists(self.pidfile):
- os.remove(self.pidfile)
- else:
- print str(err)
- sys.exit(1)
-
- def restart(self):
- """
- Restart the daemon
- """
- self.stop()
- self.start()
-
- def check(self):
- """
- Check if the daemon is currently running.
- Depends on /proc filesystem, so it might not work on all distros.
- Returns True/False
- """
- # Get the pid from the pidfile
- try:
- pf = file(self.pidfile,'r')
- pid = int(pf.read().strip())
- pf.close()
- except IOError:
- pid = None
-
- if not pid:
- return False
-
- try:
- return os.path.exists("/proc/{0}".format(pid))
- except OSError:
- return False
-
- def run(self):
- """
- You should override this method when you subclass Daemon. It will be called after the process has been
- daemonized by start() or restart().
- """
View
53 chrd/logger.py
@@ -1,53 +0,0 @@
-# -*- coding: utf-8 -*-
-import settings
-
-import os
-from time import strftime, gmtime
-
-_file_info = settings.log_dir + os.sep + "info.log"
-_file_debug = settings.log_dir + os.sep + "debug.log"
-_file_warning = settings.log_dir + os.sep + "warning.log"
-_file_error = settings.log_dir + os.sep + "error.log"
-
-if not os.path.exists(settings.log_dir):
- os.makedirs(settings.log_dir)
-
-
-def info(*args):
- args_ = fix(args)
- out(_file_info, ' '.join(args_))
-
-def debug(*args):
- args_ = fix(args)
- out(_file_debug, ' '.join(args_), print_=settings.debug)
-
-def warning(*args):
- args_ = fix(args)
- out(_file_warning, ' '.join(args_))
-
-def error(*args):
- args_ = fix(args)
- out(_file_error, ' '.join(args_))
-
-def fatal(*args):
- args_ = fix(args)
- out(_file_error, ' '.join(['!!FATAL!!'] + args_))
-
-def out(file_, text, time_=True, print_=True):
- if time_:
- text = '[{0}] {1}'.format(strftime("%m/%d/%Y %H:%M:%S", gmtime()), text)
- if print_:
- print text
- touch(file_)
- with open(file_, 'a+b') as f:
- f.write(text + "\n")
-
-def touch(file_):
- """Create a missing file.
- """
- if not os.path.exists(file_):
- with open(file_, 'w+'):
- pass
-
-def fix(args):
- return [str(arg) for arg in args]
View
330 chrd/pysql_wrapper.py
@@ -1,330 +0,0 @@
-# -*- coding: utf-8 -*-
-# Copyright (C) 2012 Christopher Carter <chris@gibsonsec.org>
-# Licensed under the MIT License.
-# Modified for use with chr url shortener.
-
-class pysql_wrapper:
- _instance = None
-
- def __init__(self, debug=False, **kwargs):
- # Are we gonna print various debugging messages?
- self.debug = debug
-
- # The internal database connection.
- # Can be either MySQL or sqlite, it doesnt matter.
- self._dbc = None
-
- # The database cursor. We'll use this to actually query stuff.
- self._cursor = None
-
- # The finalised query string to send to the dbc.
- self._query = ""
- self._query_type = ""
-
- # A dictionary of 'field' => 'value' which are used in the WHERE clause.
- self._where = {}
-
- # An internal counter of modified rows from the last statement.
- self._affected_rows = 0
-
- # Stuff we won't need unless we're using MySQL.
- self._db_host = None
- self._db_user = None
- self._db_pass = None
- self._db_name = None
- # Stuff we won't need unless we're using sqlite3
- self._db_path = None
-
- # Just so we know stuff worked after the connection.
- self._db_version = "SOMETHING WENT WRONG!"
- self._db_type = 'DEFAULT'
-
- # This is so you can easily add extra database types.
- self._db_connectors = {
- 'sqlite': self._connect_sqlite,
- 'mysql': self._connect_mysql
- }
-
- # Do some checks depending on what we're doing.
- if 'db_type' in kwargs:
- _db_type = kwargs['db_type']
- if _db_type.lower() == 'mysql':
- for db in ('db_host', 'db_user', 'db_pass', 'db_name'):
- if db not in kwargs:
- # If they miss something, we can't connect to MySQL.
- raise ValueError('No {0} was passed to pysql_wrapper.'.format(db))
- else:
- # We only want strings!
- if type(kwargs.get(db)) != str:
- raise TypeError('{0} was passed, but it isn\'t a string.')
- self._db_host = kwargs.get('db_host')
- self._db_user = kwargs.get('db_user')
- self._db_pass = kwargs.get('db_pass')
- self._db_name = kwargs.get('db_name')
- self._db_type = 'mysql'
- elif _db_type.lower() in ('sqlite', 'sqlite3'):
- self._db_type = 'sqlite'
- if 'db_path' not in kwargs:
- self._debug('Using sqlite and not given db_path.')
- raise ValueError('sqlite was selected, but db_path was not specified.')
- else:
- self._db_path = kwargs.get('db_path')
- if not isinstance(self._db_path, str):
- self._debug('Given', self._db_path, 'of type', type(self._db_path), 'which isn\'t a string.')
- raise TypeError("db_path was passed, but isn't a string.")
-
- try:
- # Try to grab the connector.
- connector = self._db_connectors[self._db_type]
- connector()
- except KeyError as e:
- self._debug('Given', self._db_type, 'as database_type, not supported.')
- raise Exception('The given database type is not supported.')
-
- # Let's knock it up a notch.. BAM!
- pysql_wrapper._instance = self
-
- def _debug(self, *stuff):
- if not self.debug:
- return
- print '[ ? ]', ' '.join(stuff)
-
- def __del__(self):
- # If this isn't called, it shouldn't really matter anyway.
- # If it is, let's tear down our connections.
- if self._dbc:
- self._dbc.close()
-
- def _sqlite_dict_factory(self, cursor, row):
- d = {}
- for idx, col in enumerate(cursor.description):
- d[col[0]] = row[idx]
- return d
-
- def _connect_sqlite(self, force=False):
- """Connect to the sqlite database.
-
- This function also grabs the cursor and updates the _db_version
- """
- import sqlite3
- self._dbc = sqlite3.connect(self._db_path)
- self._dbc.row_factory = self._sqlite_dict_factory
- self._cursor = self._dbc.cursor()
- self._cursor.execute('SELECT SQLITE_VERSION()')
- self._db_version = self._cursor.fetchone()
- self._db_type = 'sqlite'
-
- def _connect_mysql(self, force=False):
- """Connect to the MySQL database.
-
- This function also grabs the cursor and updates the _db_version
- """
- import MySQLdb
- self._dbc = MySQLdb.connect(self._db_host, self._db_user, self._db_pass, self._db_name)
- self._cursor = self._dbc.cursor(MySQLdb.cursors.DictCursor)
- self._cursor.execute('SELECT VERSION()')
- self._db_version = self._cursor.fetchone()
- self._db_type = 'mysql'
-
- def _reset(self):
- """Reset the given bits and pieces after each query.
- chr: kill the connection
- """
- self._where = {}
- self._query = ""
- self._query_type = ""
- self.__del__()
- return None
-
- def row(self):
- """Return last modified row id.
- """
- return self._cursor.lastrowid
-
- def where(self, field, value):
- """Add a conditional WHERE statement. You can chain multiple where() calls together.
-
- Example: pysql.where('id', 1).where('foo', 'bar')
- Param: 'field' The name of the database field.
- Param: 'value' The value of the database field.
- Return: Instance of self for chaining where() calls
- """
- self._where[field] = value
- return self
-
- def get(self, table_name, num_rows=False):
- """SELECT some data from a table.
-
- Example: pysql.get('table', 1) - Select one row
- Param: 'table_name' The name of the table to SELECT from.
- Param: 'num_rows' The (optional) amount of rows to LIMIT to.
- Return: The results of the SELECT.
- """
- self._query_type = 'select'
- self._query = "SELECT * FROM `{0}`".format(table_name)
- stmt, data = self._build_query(num_rows=num_rows)
- res = self._execute(stmt, data)
- self._reset()
- return res
-
- def insert(self, table_name, table_data):
- """INSERT data into a table.
-
- Example: pysql.insert('table', {'id': 1, 'foo': 'bar'})
- Param: 'table_name' The table to INSERT into.
- Param: 'table_data' A dictionary of key/value pairs to insert.
- Return: The results of the query.
- """
- self._query_type = 'insert'
- self._query = "INSERT INTO `{0}`".format(table_name)
- stmt, data = self._build_query(table_data=table_data)
- res = self._execute(stmt, data)
- if self._affected_rows > 0:
- res = True
- else:
- res = False
- self._reset()
- return res
-
- def update(self, table_name, table_data, num_rows = False):
- """UPDATE a table. where() must be called first.
-
- Example: pysql.where('id', 1).update('table', {'foo': 'baz'})
- Param: 'table_name' The name of the table to UPDATE.
- Param: 'table_data' The key/value pairs to update. (SET `KEY` = 'VALUE')
- Param: 'num_rows' The (optional) amount of rows to LIMIT to.
- return True/False, indicating success.
- """
- if len(self._where) == 0:
- return False
- self._query_type = 'update'
- self._query = "UPDATE `{0}` SET ".format(table_name)
- stmt, data = self._build_query(num_rows=num_rows, table_data=table_data)
- res = self._execute(stmt, data)
- if self._affected_rows > 0:
- res = True
- else:
- res = False
- self._reset()
- return res
-
- def delete(self, table_name, num_rows = False):
- """DELETE from a table. where() must be called first.
-
- Example: pysql.where('id', 1).delete('table')
- Param: 'table_name' The table to DELETE from.
- Param: 'num_rows' The (optional) amount of rows to LIMIT to.
- return True/False, indicating success.
- """
- if len(self._where) == 0:
- return False
- self._query_type = 'delete'
- self._query = "DELETE FROM `{0}`".format(table_name)
- stmt, data = self._build_query(num_rows=num_rows)
- res = self._execute(stmt, data)
- if self._affected_rows > 0:
- res = True
- else:
- res = False
- self._reset()
- return res
-
- def escape(self, string):
- return self._dbc.escape_string(string)
-
- def query(self, q):
- """Execute a raw query directly.
-
- Example: pysql.query('SELECT * FROM `posts` LIMIT 0, 15')
- Param: 'q' The query to execute.
- Return: The result of the query. Could be an array, True, False, anything, really.
- """
- self._query_type = 'manual'
- self._query = q
- res = self._execute(self._query, data=None)
- self._reset()
- return res
-
- def affected_rows(self):
- """Grab the amount of rows affected by the last query.
-
- Return: The amount of rows modified.
- """
- return self._cursor.rowcount
-
- def _execute(self, query, data=None):
- if data is not None:
- self._cursor.execute(query, data)
- else:
- self._cursor.execute(query)
- if self._db_type == 'sqlite':
- self._dbc.commit()
- res = self._cursor.fetchall()
- self._affected_rows = int(self._cursor.rowcount)
- return res
-
- def _format_str(self, thing):
- """Returns the format string for the thing.
-
- Due to how retarded MySQL is, this _HAS_ to be %s, or it won't work.
- """
- if self._db_type == 'sqlite':
- return '?'
- elif self._db_type == 'mysql':
- return '%s'
- else:
- # No idea, return ?.
- return '?'
-
- def _build_query(self, num_rows=False, table_data=False):
- return_data = ()
-
- # e.g. -> UPDATE `table` SET `this` = ?, `that` = ?, `foo` = ? WHERE `id` = ?;
-
- # If they've supplied where() statements
- if len(self._where) > 0:
- keys = self._where.keys()
- # If they've supplied table data:
- if type(table_data == dict):
- count = 1
- # If we're calling an UPDATE
- if self._query_type == 'update':
- for key, val in table_data.iteritems():
- format = self._format_str(type)
- if count == len(table_data):
- self._query += "`{0}` = {1}".format(key, format)
- else:
- self._query += "`{0}` = {1}, ".format(key, format)
- return_data = return_data + (val,)
- count += 1
- self._query += " WHERE "
- where_clause = []
- for key, val in self._where.iteritems():
- format = self._format_str(val)
- where_clause.append("`{0}` = {1}".format(key, format))
- return_data = return_data + (val,)
- self._query += ' AND '.join(where_clause)
-
- # If they've supplied table data.
- if type(table_data) == dict and self._query_type == 'insert':
- keys = table_data.keys()
- vals = table_data.values()
- num = len(table_data)
- for count, key in enumerate(keys):
- # Wrap column names in backticks.
- keys[count] = "`{0}`".format(key)
- self._query += " ({0}) ".format(', '.join(keys))
- # Append VALUES (?,?,?) however many we need.
- format = ""
- for count, val in enumerate(vals):
- format += '{0},'.format(self._format_str(val))
- format = format[:-1]
-
- self._query += "VALUES ({0})".format(format)
- for val in vals:
- return_data = return_data + (val,)
-
- # Do you want LIMIT with that, baby?!
- if num_rows:
- self._query += " LIMIT {0}".format(num_rows)
- return (self._query, return_data,)
View
39 chrd/root_check.py
@@ -1,39 +0,0 @@
-# -*- coding: utf-8 -*-
-import os, sys
-
-"""
- If they don't explicity say they want to use root, and
- are able to actually realise they *have* to type in 'Yes',
- they're probably just running it as root because they're dumb.
- Nothing done here requires root, why should it? Don't let that
- kind of shit happen.
-"""
-
-def root_check(no_daemon_key):
- explicity_use_root = False
- starting = 'start' in sys.argv or no_daemon_key in sys.argv
-
- if '--explicity-use-root' in sys.argv and starting:
- print "Don't run chr as root unless you really, super-duper have to."
- print "If it's so you can use port 80, think about installing a proxy such as nginx."
- print "If you're sure you do, you'll have to type 'Yes', with the capital and all."
- inp = raw_input('Do you really want to run chr as root? ')
- if inp not in ("Yes"):
- print 'Alrighty then!'
- sys.exit(0)
- print "Please, seriously consider why you need to run this as root."
- inp = raw_input('Are you certain? This is your last chance! Type \'Yes\' if you are: ')
- if not inp in ("Yes"):
- print 'Alrighty then!'
- sys.exit(0)
-
- print 'Alright, your loss.'
- explicity_use_root = True
-
- # If they're starting as root without --explicity-use-root:
- if os.getuid() == 0 and starting and not explicity_use_root:
- print 'Stop! You should never run anything as root unless you absolutely have to.'
- print 'This is not one of those occasions.'
- print 'chr will not function as root, please run it in usermode.'
- print 'If you\'re absolutely sure you want to run as root, use the --explicity-use-root flag.'
- sys.exit(2)
View
33 chrd/schema/schema.sql
@@ -1,33 +0,0 @@
--- THIS IS FOR sqlite3, NOT MySQL, SO DON'T TRY TO EVEN USE IT, BRO.
-
-CREATE TABLE IF NOT EXISTS `users` (
- `id` INTEGER PRIMARY KEY,
- `login` varchar(32) NOT NULL, -- Login name (lower case)
- `display` varchar(32) NOT NULL, -- Displayed login name
- `password` varchar(64) NOT NULL, -- sha256() hash of (salt + login.lower() + password + salt)
- `email` varchar(64) NOT NULL,
- `admin` tinyint(1) NOT NULL DEFAULT '0'
-);
-
-
--- To calculate hits: 'SELECT `id` FROM `clicks` WHERE `url` = <redirect_id>', then count.
-CREATE TABLE IF NOT EXISTS `redirects` (
- `id` INTEGER PRIMARY KEY,
- `long` TEXT NOT NULL,
- `short` TEXT NOT NULL, -- Added afterwards. This is the base62 encoding for the row id.
- `delete` varchar(64) NOT NULL, -- String to delete. /slug/delete/<`delete`>
- `user` INTEGER NOT NULL DEFAULT '0', -- Which user made this? 0 means non-logged in.
- `ip` varchar(15) NOT NULL -- raw ip of the submitter.
-);
-
-CREATE TABLE IF NOT EXISTS `clicks` (
- `id` INTEGER PRIMARY KEY,
- `url` INTEGER NOT NULL, -- id from redirect row
- `ip` varchar(15) NOT NULL, -- raw ip in form of 'W{1,3}.X{1,3}.Y{1,3}.Z{1,3}'
- `time` INTEGER NOT NULL, -- int(time.time()) of hit
- `agent` TEXT NOT NULL, -- Their user agent. Trimmed to a max of 256
- `agent_platform` TEXT NOT NULL, -- These are all based off of the
- `agent_browser` TEXT NOT NULL, -- flask request.user_agent variable.
- `agent_version` TEXT NOT NULL, -- It has the platform (Win, Mac), browser (Firefox, Safari)
- `agent_language` TEXT NOT NULL -- aswell as version and language.
-);
View
72 chrd/settings.py
@@ -1,72 +0,0 @@
-# -*- coding: utf-8 -*-
-# Settings file for chr url shortener.
-
-# Don't ever enable debugging in a situation
-# where the computer has a public IP.
-debug = False
-
-
-flask_debug = debug
-flask_host = '127.0.0.1'
-flask_port = 8080
-flask_base = "/"
-# Your shrunk URL formats. Use {slug} for
-# where the url slug goes.
-flask_url = "http://change.this/{slug}"
-flask_secret_key = 'SOMETHING!SECRET@AND#HARD$TO%GUESS'
-
-# Set this once, make it long and hard, and never change it.
-salt_password = 'SOMETHING!UNIQUE@AND#HARD$TO%GUESS^OR&ELSE'
-
-# Disable this if you're really security concious.
-# With it on, people will be able to get the IP of
-# your server. This is bad if you're behind CloudFlare
-# or some other load balancer.
-# With this disabled, urls are validated with a regex.
-validate_urls = True
-# The regex to validate urls if the above is False.
-# It's intentionally sparse, we don't need a full fledged regex.
-# DON'T CHANGE if you don't understand regexes.
-# Basically, this just handles urls like:
- # http://example.com
- # https://example.com
- # https://ssl.example.org
- # http://this.is.an.example.net/test.php
- # https://some-stupid.domain.co.nz/blah/is_a/word!
- # <stupidly long google images result that people always link>
-validate_regex = r'^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.!&()=:;?,-]*)\/?$'
-
-# Soft cap for URL length. So people can't waste space.
-# Besides, when is a URL ever *this* long?
-soft_url_cap = 512
-
-# Absolute directory for logging.
-log_dir = "/path/to/logs"
-
-
-# This has to be absolute or sqlite will complain.
-sql_path = "/path/to/chr.db"
-
-# reCAPTCHA settings.
-captcha_public_key = "YOUR_PUBLIC_API_KEY"
-captcha_private_key = "YOUR_PRIVATE_API_KEY"
-
-
-# API stuff, for when we finish this.
-api_enabled = False
-
-
-#--------------------------#
-# Constants (do not touch) #
-#--------------------------#
-
-# _SCHEMA_* are the table names.
-# This allows simple changement of table names without changing 500 classes.
-_SCHEMA_USERS = 'users'
-_SCHEMA_REDIRECTS = 'redirects'
-_SCHEMA_CLICKS = 'clicks'
-_SCHEMA_MAX_SLUG = 32 # soft limit, schema defines as TEXT.
-
-# Prefix for custom slugs.
-# If it's more than one character, you WILL break things.
-_CUSTOM_CHAR = '+'
View
30 chrd/utility.py
@@ -1,30 +0,0 @@
-# -*- coding: utf-8 -*-
-# chr's utility functions
-
-import settings
-from pysql_wrapper import pysql_wrapper
-
-def hash_pass(username, password, salt=settings.salt_password):
- import hashlib
- # This is so they can log in as Foo, FOO or fOo.
- username = username.lower()
- return hashlib.sha256(salt + password + username + salt).hexdigest()
-
-def sql():
- """Reuse this every time you need to do SQL stuff.
- sql().where('id', 1).get('table')
- """
- return pysql_wrapper(db_type='sqlite', db_path=settings.sql_path)
-
-def revision():
- # Change after modification, etc.
- return 'r7'
-
-class Struct:
- def __init__(self, **entries):
- self.__dict__.update(entries)
-
-def date_strip_day(date_):
- if not '/' in date_:
- return date_
- return date_.split('/')[1]
View
87 chrm.py
@@ -1,87 +0,0 @@
-# -*- coding: utf-8 -*-
-#!/usr/bin/env python
-import os, sys
-
-no_daemon_key = 'start-no-daemon'
-help_message = """python {script} [option] [flag] ...
-Options and flags:
-start \t\t: Start chr and daemonize the process.
-stop \t\t: Stop a running chr process.
-restart \t: Restart a running chr process. (Calls stop() and start())
-status \t\t: Check if a chr daemon is currently running.
-{non_daemon} : Start chr without daemonizing. This runs in your current shell.
-
--v, --version \t: Print the chr revision and exit.
--h, --help \t: Display this help message and exit."""
-
-try:
- # This script just checks if the user is using root.
- from chrd.root_check import root_check
- root_check(no_daemon_key)
-except ImportError as e:
- print 'Something went wrong importing root_check.'
- if os.getuid() == 0:
- print 'Sorry, but you can\'t run chr as root.'
- sys.exit(2)
- raise
-
-try:
- from chrd.daemon import Daemon
-except ImportError as e:
- print 'An error occured importing chr\'s daemon class.'
- print 'Try fixing your installation and try again.'
- raise
-
-class chrDaemon(Daemon):
- def run(self):
- from chrd.chrf import run
- run()
-
-if __name__ == '__main__':
- _two = len(sys.argv) < 2
- if len(sys.argv) < 2 or '-h' in sys.argv or '--help' in sys.argv:
- print help_message.format(script=sys.argv[0], non_daemon=no_daemon_key)
- sys.exit(2 if _two else 0) # if they -h'd, we're good, otherwise it's borke.
-
- args = {
- "stdout": "/dev/null",
- "stderr": "/dev/null"
- }
- if no_daemon_key in sys.argv:
- # Set stdout and stderr flags to /dev/stdout for the daemon
- # this makes life easier for figuring out what's going on.
- args = {"stdout": "/dev/stdout", "stderr": "/dev/stderr"}
-
- daemon = chrDaemon('/tmp/chr-daemon.pid', **args)
- for arg in sys.argv[1:]: # Obviously we don't want the name of the script
- arg = arg.lower()
- if arg in ('-v', '--version'):
- from chrd.utility import revision
- print 'chr', revision(), '-', 'Free as in freedom.'
- print 'https://github.com/PigBacon/chr', '-', 'MIT Licensed'
-
- elif arg in ('start'):
- daemon.start()
-
- elif arg in ('start-no-daemon'):
- daemon.start(daemonize=False)
-
- elif arg in ('stop'):
- daemon.stop()
-
- elif arg in ('restart'):
- daemon.restart()
-
- elif arg in ('status'):
- if daemon.check():
- print 'chr daemon currently running.'
- else:
- print 'chr daemon not running.'
- sys.exit(0)
-
- elif arg in ('--explicity-use-root'):
- continue
-
- else:
- print 'Unknown argument', arg
- sys.exit(2)
View
9 chru/__init__.py
@@ -0,0 +1,9 @@
+""" **chr** (coded under the name ``chru``) is a Python based URL shortening
+ service which uses Flask as a front end, and pysqlw as the SQL
+ backend, to interface with sqlite3.
+
+ It can shrink billions of unique URLs with less than 6 characters,
+ run in the background with no human interaction,
+ and it can fly like a bird -- or is that Super Man?
+"""
+from .version import __version__, __supported__
View
118 chru/chra.py
@@ -0,0 +1,118 @@
+# -*- coding: utf-8 -*-
+
+"""
+ chra is the default entry-point for chr.
+ Specifically, chru.chra:main is the entry point we use.
+"""
+
+from version import __version__, __supported__
+
+import argparse
+import os
+import sys
+
+import mattdaemon
+
+class daemon(mattdaemon.daemon):
+ def run(self, *args, **kwargs):
+ """ This method will simply import the web control,
+ set up the Flask app, and then run the app.
+ """
+ import web
+ import web.routes
+ web.routes.set_app(
+ debug=kwargs.get("debug"),
+ settings_file=kwargs.get("settings")[0],
+ log_file=kwargs.get("log"),
+ __supported__=__supported__
+ )
+ web.routes.app.run(host=web.s.flask_host, port=web.s.flask_port)
+
+def main():
+ """ This method is in charge of everything
+
+ It uses argparse to control the lovely arguments it handles,
+ and sets up -- and controls -- the chr daemon.
+ """
+ if "--make-config" in sys.argv:
+ import json
+ import web
+ print json.dumps(web.skeleton, indent=4)
+ return
+
+ parser = argparse.ArgumentParser(description="Take control of the chr url shortener.")
+ parser.add_argument("action",
+ help="Action to take on the server",
+ choices=["start", "stop", "restart", "status"],
+ metavar="action")
+
+ parser.add_argument("-v", "--version",
+ help="Print the chr version and exit",
+ action="version",
+ version=__version__)
+
+ parser.add_argument("-n", "--no-daemon",
+ help=argparse.SUPPRESS,
+ action="store_true")
+
+ parser.add_argument("-l", "--log",
+ help="Set the file that output is logged to (default: %(default)s)",
+ type=str,
+ nargs=1,
+ metavar="file",
+ default="/tmp/chr-daemon.log")
+
+ parser.add_argument("-p", "--pid",
+ help="Set the file the PID is saved to (default: %(default)s)",
+ type=str,
+ nargs=1,
+ metavar="file",
+ default="/tmp/chr-daemon.pid")
+
+ parser.add_argument("-s", "--settings",
+ help="Set the location of the settings.json file",
+ type=file,
+ nargs=1,
+ metavar="file")
+
+ # Not actually used via argparse.
+ parser.add_argument("--make-config",
+ help="Print the settings.json file skeleton to stdout",
+ action="store_true")
+ args = parser.parse_args()
+
+ daem_args = {
+ "stdin": "/dev/null",
+ "stdout": args.log,
+ "stderr": args.log
+ }
+ if args.no_daemon:
+ daem_args = {
+ "stdin": "/dev/stdin",
+ "stdout": "/dev/stdout",
+ "stderr": "/dev/stderr"
+ }
+ daem = daemon(args.pid, daemonize=not args.no_daemon, **daem_args)
+
+ if args.action == "start":
+ if args.settings == None:
+ print "You must pass in the settings.json file location (see --help for info)"
+ return
+
+ # -n flag given? -- enable debugging
+ debug = args.no_daemon
+ daem.start(debug=debug, log=args.log, settings=args.settings)
+ elif args.action == "stop":
+ daem.stop()
+ elif args.action == "restart":
+ daem.restart()
+ elif args.action == "status":
+ if daem.status():
+ print "chr currently running! :D"
+ else:
+ print "chr not running! D:"
+ return
+
+
+if __name__ == "__main__":
+ main()
View
2  chru/utility/__init__.py
@@ -0,0 +1,2 @@
+from .struct import struct
+import funcs
View
25 chrd/base62.py → chru/utility/base62.py
@@ -1,16 +1,19 @@
#!/usr/bin/env python
-#
-# Converts any integer into a base [BASE] number. I have chosen 62
-# as it is meant to represent the integers using all the alphanumeric
-# characters, [no special characters] = {0..9}, {A..Z}, {a..z}
-#
-# I plan on using this to shorten the representation of possibly long ids,
-# a la url shortenters
-#
-# saturate() takes the base 62 key, as a string, and turns it back into an integer
-# dehydrate() takes an integer and turns it into the base 62 string
-#
+"""
+ Note: Taken from the public domain (namely: Stack Overflow), so this
+ isn't my documentation.
+
+ Converts any integer into a base [BASE] number. I have chosen 62
+ as it is meant to represent the integers using all the alphanumeric
+ characters, [no special characters] = {0..9}, {A..Z}, {a..z}
+
+ I plan on using this to shorten the representation of possibly long ids,
+ a la url shortenters
+
+ saturate() takes the base 62 key, as a string, and turns it back into an integer
+ dehydrate() takes an integer and turns it into the base 62 string
+"""
import math
import sys
View
128 chru/utility/funcs.py
@@ -0,0 +1,128 @@
+# -*- coding: utf-8 -*-
+
+"""
+ This module just stores various functions which don't
+ really fit anywhere else in the app.
+"""
+
+import pysqlw
+
+import chru.web as web
+
+import requests
+import re
+import logging
+
+def sql():
+ """ Opens an sqlite connection, returning it for usage.
+ """
+ return pysqlw.pysqlw(db_type="sqlite", db_path=web.s.sql_path)
+
+def date_strip_day(date_):
+ """ Strips the day out of a date.. I think.
+ """
+ if not "/" in date_:
+ return date_
+ return date_.split("/")[1]
+
+def verify_sql():
+ """ Ensure the database file is made, and if not,
+ create it and populate it with the schema.
+ """
+ import os
+ if os.path.isfile(web.s.sql_path):
+ return True
+ with sql() as s:
+ s.wrapper.cursor.executescript(web.schema)
+ add_first = web.slug.url_to_slug("https://github.com/plausibility/chr", slug=web.s._CUSTOM_CHAR + "source")
+ if add_first:
+ return True
+ else:
+ return False
+
+def constant(f):
+ """ Simple decorator for const class variables.
+
+ For example:
+
+ .. code-block:: python
+
+ >>> class A:
+ ... @constant
+ ... foo = 9001
+ ...
+ >>> a = A()
+ >>> a.foo
+ 9001
+ >>> a.foo = 7
+ (...)
+ Some SyntaxError
+ (...)
+
+ :raises: SyntaxError
+ """
+ def fset(self, value):
+ raise SyntaxError("this is a constant variable")
+ def fget(self):
+ return f()
+ return property(fget, fset)
+
+def validate_url(url, timeout=1, fallback=True, regex=None, use_requests=True):
+ """ Checks whether or not a URL is valid.
+ The URL can go through two stages, depending
+ on the settings passed in.
+
+ 1. The URL can be validated by sending a HEAD request
+ with the requests module, and verifying that the reply
+ code (if any), is in a list of valid replies.
+ 2. The URL gets run against a regex, to verify that,
+ structurally, the URL is atleast valid.
+
+ If fallback is on, the URL will go through both steps (if
+ the first fails), to verify the validity.
+
+ Failure of both tests results in a validity failure, obviously.
+
+ :param url: the URL we're testing the validity of.
+ :type url: str
+ :param timeout: the timeout to pass to the HEAD request. this can stop exploits
+ :type timeout: int
+ :param fallback: should we fallback to the regex if the request fails?
+ :type fallback: bool
+ :param regex: the regex we'll compare against
+ :type regex: str
+ :param use_requests: should we use requests first?
+ :type use_requests: bool
+ :return: whether or not the URL is valid
+ :rtype: bool
+ """
+ if use_requests:
+ try:
+ logging.debug("Sending HEAD req to: %s", url)
+ head = requests.head(url, timeout=timeout)
+ logging.debug("Reply: %s", head.status_code)
+ # These are some codes we're 200 OK with. (geddit?)
+ valid_codes = [
+ 200, # OK
+ 204, # No Content
+ 301, # Moved Permanently
+ 302, # Found
+ 304, # Not Modified
+ 307, # Temporary Redirect (HTTP/1.1)
+ 308, # Permanent Redirect (HTTP/1.1)
+ 418, # I'm a teapot
+ ]
+ if head.status_code not in valid_codes:
+ raise requests.exceptions.HTTPError
+ return True
+ except requests.exceptions.RequestException as e:
+ logging.debug(e)
+ if fallback:
+ return validate_url(url, timeout=timeout, fallback=False, regex=regex, use_requests=False)
+ else:
+ return False
+ else:
+ if regex is None:
+ regex = web.s.validate_regex
+ logging.debug("Matching '%s' against '%s'", url, regex)
+ return re.match(regex, url, flags=re.I)
View
13 chru/utility/struct.py
@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+class struct:
+ """ This class is just a simple way to convert a
+ dictionary to an object, for easier interaction.
+
+ Instead of ``o['this']``, you can do ``o.this``;
+ Isn't it great?!
+ """
+ def __init__(self, **entries):
+ self.__dict__.update(entries)
+
+ def __repr__(self):
+ return str(self.__dict__)
View
2  chru/version.py
@@ -0,0 +1,2 @@
+__version__ = "2.0.0"
+__supported__ = ["2.0.0rc1", "2.0.0"]
View
1  chru/web/__init__.py
@@ -0,0 +1 @@
+from settings import s, schema, skeleton # say that three times fast.
View
339 chru/web/routes.py
@@ -0,0 +1,339 @@
+# -*- coding: utf-8 -*-
+
+"""
+ chru.web.routes handles the Flask app.
+
+ This module contains all of the ``@app.route`` calls,
+ and it sets up the module wide settings, so the rest of
+ the app can use them without problems.
+"""
+
+import json
+import re
+import time
+from datetime import datetime
+import logging
+
+from flask import Flask, render_template, session, abort, request, g
+from flask import redirect, url_for, flash, make_response, jsonify
+
+from recaptcha.client import captcha
+
+# ----- #
+import chru
+import chru.utility as utility
+import chru.web as web
+import chru.web.slug
+
+app = Flask(__name__)
+app.jinja_env.globals.update(sorted=sorted)
+app.jinja_env.globals.update(len=len)
+app.jinja_env.globals.update(date_strip_day=utility.funcs.date_strip_day)
+
+def set_app(debug=False, settings_file=None, log_file=None, __supported__=[]):
+ """ Sets up the finer details of the Flask app,
+ as well as the parsing in the settings file.
+
+ :param debug: should the app be set to debug mode?
+ :type debug: bool
+ :param settings_file: the file handler for the given settings file
+ :type settings_file: file handler
+ :param log_file: the file we're going to set :mod:`logging` to use
+ :type log_file: str
+ :param __supported__: the list of supported settings versions
+ :type __supported__: list
+ """
+ settings_dict = json.load(settings_file)
+ settings_file.close()
+ settings = utility.struct(**settings_dict)
+
+ web.s = settings
+ web.s.debug = debug
+
+ if "_version" not in settings_dict \
+ or web.s._version not in __supported__:
+ v = "unknown"
+ if "_version" in settings_dict:
+ v = settings_dict["_version"]
+ import sys
+ print "Given settings file (v: {0}) is not supported!".format(v)
+ print "Supported:", ", ".join(__supported__)
+ print "Please make a new one (--make-config > settings.json) and start again."
+ sys.exit(1)
+
+ utility.funcs.verify_sql()
+
+ app.debug = debug
+ app.secret_key = str(web.s.flask_secret_key)
+
+ logging.basicConfig(
+ format="%(asctime)s [%(levelname)s] %(message)s",
+ datefmt="%m/%d/%Y %I:%M:%S%p",
+ level=logging.DEBUG,
+ filename=log_file
+ )
+ logging.info("Successfully setup app and settings!")
+
+reserved_routes = ("index", "submit")
+
+@app.route("/index")
+@app.route("/")
+def index():
+ """ Simply renders the index page. """
+ return render_template("index.html", recaptcha_key=web.s.captcha_public_key)
+
+@app.route("/<slug>")
+def slug_redirect(slug):
+ """ Redirects to the appropriate slug, if it exists.
+
+ :param slug: the shortened url slug
+ :type slug: str
+ :return: a redirection (to index or the shortened url)
+ """
+ if not web.slug.is_valid(slug):
+ logging.debug("%s isn't valid", slug)
+ return redirect(url_for("index"))
+ if not web.slug.exists(slug):
+ logging.debug("%s doesn't exist", slug)
+ return redirect(url_for("index"))
+ url = web.slug.url_from_slug(slug)
+ web.slug.add_hit(slug, request.remote_addr, request.user_agent)
+ return redirect(url)
+
+@app.route("/<slug>/delete/")
+@app.route("/<slug>/delete")
+def slug_delete_nokey(slug):
+ """ Redirects back to the index if no deletion key is given. """
+ return redirect(url_for("index"))
+
+@app.route("/<slug>/delete/<delete>")
+def slug_delete(slug, delete):
+ """ Will attempt to delete a slug, if given a correct deletion key.
+
+ If an invalid key is given, simply redirects to the index.
+
+ :param slug: the shortened url slug to delete
+ :type slug: str
+ :param delete: the deletion key to use
+ :type delete: str
+ :return: a redirection to the index
+ """
+ if not web.slug.is_valid(slug) or not web.slug.exists(slug):
+ return redirect(url_for("index"))
+
+ row = web.slug.to_row(slug)
+ if delete != row["delete"]:
+ logging.debug("%s gave invalid delete code ('%s')", request.remote_addr, delete)
+ return redirect(url_for("index"))
+
+ if not web.slug.delete(slug):
+ logging.error("Unable to delete slug %s", slug)
+ flash("Something went wrong deleting {0}".format(web.s.flask_url.format(slug=row["short"])), "error")
+ else:
+ flash("Successfully deleted {0}".format(web.s.flask_url.format(slug=row["short"])), "success")
+ return redirect(url_for("index"))
+
+@app.route("/<slug>/stats")
+def slug_stats(slug):
+ """ Pulls up all related slug information, and provides it to
+ the statistics template. This gives information about the
+ amount of clicks, operating systems, browsers, when it
+ got the most clicks, and more.
+
+ If there are no clicks, this really won't return much
+ of value, as there is nothing!
+
+ :param slug: the shortened url slug to show stats for
+ :type slug: str
+ :return: the rendered stats page
+ """
+ if not web.slug.is_valid(slug) or not web.slug.exists(slug):
+ return redirect(url_for("index"))
+
+ row = web.slug.to_row(slug)
+ clicks = utility.funcs.sql().where("url", row["id"]).get(web.s._SCHEMA_CLICKS)
+
+ click_info = {
+ "platforms": {"unknown": 0},
+ "browsers": {"unknown": 0},
+ "pd": {} # clicks per day
+ }
+ month_ago = int(time.time()) - (60 * 60 * 24 * 30) # Last 30 days
+ month_ago_copy = month_ago
+
+ for x in xrange(1, 31):
+ # Pre-populate the per day clicks dictionary.
+ # This makes life much much easier for everything.
+ month_ago_copy += (60 * 60 * 24) # one day
+ strf = str(datetime.fromtimestamp(month_ago_copy).strftime("%m/%d"))
+ click_info["pd"][strf] = 0
+
+ for click in clicks:
+ if click["time"] > month_ago:
+ # If the click is less than a month old
+ strf = str(datetime.fromtimestamp(click["time"]).strftime("%m/%d"))
+ click_info["pd"][strf] += 1
+
+ if len(click["agent"]) == 0 \
+ or len(click["agent_platform"]) == 0 \
+ or len(click["agent_browser"]) == 0:
+ click_info["platforms"]["unknown"] += 1
+ click_info["browsers"]["unknown"] += 1
+ continue
+
+ platform_ = click["agent_platform"]
+ browser_ = click["agent_browser"]
+
+ if not platform_ in click_info["platforms"]:
+ click_info["platforms"][platform_] = 1
+ else:
+ click_info["platforms"][platform_] += 1
+
+ if not browser_ in click_info["browsers"]:
+ click_info["browsers"][browser_] = 1
+ else:
+ click_info["browsers"][browser_] += 1
+
+ # Formatting in directly without escapes, how horrible am I!
+ # But these are known to be *safe*, as they're set by us.
+ unique = utility.funcs.sql().query("SELECT COUNT(DISTINCT `{0}`) AS \"{1}\" FROM `{2}`".format("ip", "unq", web.s._SCHEMA_CLICKS))
+ unique = unique[0]["unq"]
+
+ rpt = len(clicks) - unique
+ if len(clicks) == 0:
+ rpt = 0
+
+ f_unique = unique
+ f_clicks = len(clicks) if len(clicks) != 0 else 1
+
+ hits = {
+ "all": len(clicks),
+ "unique": unique,
+ "return": rpt,
+ # (unique / all)% of visits only come once
+ "ratio": str(round(float(f_unique) / float(f_clicks), 2))[2:]
+ }
+
+ stats = {
+ "hits": hits,
+ "clicks": click_info,
+ "long": row["long"],
+ "long_clip": row["long"],
+ "short": row["short"],
+ "short_url": web.s.flask_url.format(slug=row["short"])
+ }
+ if len(row["long"]) > 30:
+ stats["long_clip"] = row["long"][:30] + "..."
+ return render_template("stats.html", stats=stats)
+
+@app.route("/submit", methods=["GET"])
+def submit_get():
+ """ Redirects GET requests to /submit to the index. """
+ return redirect(url_for("index"))
+
+@app.route("/submit", methods=["POST"])
+def submit():
+ """ Handles URL submissions, validating URLs,
+ creating the needed slug row, and providing
+ information to the client via JSON.
+
+ This will handle invalid captchas, overly
+ long URLs, invalid URLs (if validation is on),
+ and custom slug related problems.
+
+ :return: a JSON dictionary with submission info.
+ """
+ response = captcha.submit(
+ request.form["recaptcha_challenge_field"],
+ request.form["recaptcha_response_field"],
+ web.s.captcha_private_key,
+ request.remote_addr
+ )
+ # reCAPTCHA failed
+ if not response.is_valid:
+ return jsonify({
+ "error": "true",
+ "message": "The captcha typed was incorrect.",
+ "url": "",
+ "delete": "",
+ "given": request.form["chr_text_long"]
+ })
+
+ # URL too long
+ if len(request.form["chr_text_long"]) > web.s.soft_url_cap:
+ return jsonify({
+ "error": "true",
+ "message": "The URL you provided was actually <i>too</i> long to shrink!",
+ "url": "",
+ "delete": "",
+ "given": request.form["chr_text_long"]
+ })
+
+ # Invalid URL
+ if web.s.validate_urls:
+ valid = utility.funcs.validate_url(
+ url=request.form["chr_text_long"],
+ regex=web.s.validate_regex,
+ use_requests=web.s.validate_requests,
+ fallback=web.s.validate_fallback,
+ timeout=5
+ )
+ if not valid:
+ return jsonify({
+ "error": "true",
+ "message": "The URL provided was unable to be validated.",
+ "url": "",
+ "delete": "",
+ "given": request.form["chr_text_long"]
+ })
+
+ # Custom URL validation (slugs)
+ custom_slug = False
+ if len(request.form["chr_text_short"]) > 0:
+ # If this isn't False at the end,
+ # we display the error.
+ invalid_slug = False
+ slug_ = request.form["chr_text_short"]
+
+ if not re.match("^[\w-]+$", slug_):
+ invalid_slug = "That custom URL is invalid. Only alphanumeric characters, dashes, underscores, <i>please</i>!"
+ elif len(slug_) > web.s._SCHEMA_MAX_SLUG:
+ invalid_slug = "That custom URL is too long! Less than 32 characters, <i>please</i>!"
+ elif slug_.lower() in reserved_routes:
+ invalid_slug = "Sorry, that custom URL is reserved!"
+ elif web.slug.exists(web.s._CUSTOM_CHAR + slug_):
+ invalid_slug = "Sorry, that custom URL already exists!"
+
+ if invalid_slug:
+ return jsonify({
+ "error": "true",
+ "message": invalid_slug,
+ "url": "",
+ "delete": "",
+ "given": request.form["chr_text_long"]
+ })
+
+ custom_slug = web.s._CUSTOM_CHAR + slug_
+
+ # Yippee, it's valid!
+
+ slug_, id_, delete_, url_, old_ = web.slug.url_to_slug(
+ request.form["chr_text_long"],
+ ip=request.remote_addr,
+ slug=custom_slug
+ )
+ # xxxx -> http://wht.nt/slug/delete/xxxx
+ delete_ = web.s.flask_url.format(slug=slug_) + "/delete/" + delete_
+ delete_ = delete_ if "chr_check_delete" in request.form \
+ and request.form["chr_check_delete"] == "yes" \
+ and not old_ \
+ else ""
+
+ data = {
+ "error": "false",
+ "message": "Successfully shrunk your URL!" if not old_ else "URL already shrunk, here it is!",
+ "url": web.s.flask_url.format(slug=slug_),
+ "delete": delete_,
+ "given": url_
+ }
+ return jsonify(data)
View
82 chru/web/settings.py
@@ -0,0 +1,82 @@
+import collections
+import chru
+
+s = None
+
+# Under no circumstances should you touch this variable.
+schema = """
+-- THIS IS FOR sqlite3, NOT MySQL, SO DON'T TRY TO EVEN USE IT, BRO.
+
+CREATE TABLE IF NOT EXISTS `users` (
+ `id` INTEGER PRIMARY KEY,
+ `login` varchar(32) NOT NULL, -- Login name (lower case)
+ `display` varchar(32) NOT NULL, -- Displayed login name
+ `password` varchar(64) NOT NULL, -- sha256() hash of (salt + login.lower() + password + salt)
+ `email` varchar(64) NOT NULL,
+ `admin` tinyint(1) NOT NULL DEFAULT '0'
+);
+
+
+-- To calculate hits: 'SELECT `id` FROM `clicks` WHERE `url` = <redirect_id>', then count.
+CREATE TABLE IF NOT EXISTS `redirects` (
+ `id` INTEGER PRIMARY KEY,
+ `long` TEXT NOT NULL,
+ `short` TEXT NOT NULL, -- Added afterwards. This is the base62 encoding for the row id.
+ `delete` varchar(64) NOT NULL, -- String to delete. /slug/delete/<`delete`>
+ `user` INTEGER NOT NULL DEFAULT '0', -- Which user made this? 0 means non-logged in.
+ `ip` varchar(15) NOT NULL -- raw ip of the submitter.
+);
+
+CREATE TABLE IF NOT EXISTS `clicks` (
+ `id` INTEGER PRIMARY KEY,
+ `url` INTEGER NOT NULL, -- id from redirect row
+ `ip` varchar(15) NOT NULL, -- raw ip in form of 'W{1,3}.X{1,3}.Y{1,3}.Z{1,3}'
+ `time` INTEGER NOT NULL, -- int(time.time()) of hit
+ `agent` TEXT NOT NULL, -- Their user agent. Trimmed to a max of 256
+ `agent_platform` TEXT NOT NULL, -- These are all based off of the
+ `agent_browser` TEXT NOT NULL, -- flask request.user_agent variable.
+ `agent_version` TEXT NOT NULL, -- It has the platform (Win, Mac), browser (Firefox, Safari)
+ `agent_language` TEXT NOT NULL -- aswell as version and language.
+);
+
+-- NOT IMPLEMENTED, API keys.
+CREATE TABLE IF NOT EXISTS `api` (
+ `id` INTEGER PRIMARY KEY,
+ `key` TEXT NOT NULL, -- The API key.
+ `email` TEXT NOT NULL, -- The email of the key owner.
+ `name` TEXT NOT NULL, -- The name of the key owner.
+)
+"""
+
+# Hack to make --make-config easier, OrderedDict means there's less problems.
+# Reminder: change here, change in the docs!
+skeleton = collections.OrderedDict()
+skeleton["debug"] = False
+
+skeleton["sql_path"] = "/path/to/chr.db"
+skeleton["soft_url_cap"] = 512
+
+skeleton["flask_host"] = "127.0.0.1"
+skeleton["flask_port"] = 8080
+skeleton["flask_base"] = "/"
+skeleton["flask_url"] = "http://change.this/{slug}"
+skeleton["flask_secret_key"] = "UNIQUE_KEY"
+
+skeleton["captcha_public_key"] = "YOUR_PUBLIC_API_KEY"
+skeleton["captcha_private_key"] = "YOUR_PRIVATE_API_KEY"
+
+skeleton["salt_password"] = "UNIQUE_KEY"
+
+skeleton["validate_urls"] = True
+skeleton["validate_requests"] = True
+skeleton["validate_regex"] = r"^(https?:\/\/)?([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.!&()=:;?,\-%]*)\/?$"
+skeleton["validate_fallback"] = True
+
+skeleton["api_enabled"] = False
+
+skeleton["_SCHEMA_REDIRECTS"] = "redirects"
+skeleton["_SCHEMA_USERS"] = "users"
+skeleton["_SCHEMA_CLICKS"] = "clicks"
+skeleton["_SCHEMA_MAX_SLUG"] = 32
+skeleton["_CUSTOM_CHAR"] = "+"
+skeleton["_version"] = chru.__version__
View
257 chru/web/slug.py
@@ -0,0 +1,257 @@
+# -*- coding: utf-8 -*-
+
+"""
+ chru.web.slug provides an easy to use interface to the
+ shortened URL slugs, allowing creation, removal, editing, and more.
+
+ These functions depend on the SQL database, so without it they're useless.
+"""
+
+import time
+import re
+import logging
+
+import chru.utility as utility
+import chru.utility.base62 as base62
+import chru.web as web
+
+
+def url_to_slug(url, ip=False, slug=False):
+ """ Shrinks a URL to the slug equivalent.
+
+ No URL length limits are imposed, these should
+ be done **before** calling.
+
+ :param url: the long URL to shrink
+ :type url: str
+ :param ip: the IP address of the creator
+ :type ip: str or False
+ :param slug: the custom slug (if any) to use. If not given, one is created.
+ :type slug: str or False
+ :return: tuple with (shortened slug, row id, deletion key, long url, old or new url)
+ :rtype: tuple
+ """
+ old_slug = utility.funcs.sql().where("long", url).get(web.s._SCHEMA_REDIRECTS)
+ if len(old_slug) > 0:
+ # url already shortened, give that.
+ old_slug = old_slug[0]
+ return (old_slug["short"], old_slug["id"], old_slug["delete"], old_slug["long"], True,)
+ data = {
+ "long": url,
+ "short": "",
+ "delete": make_delete_slug(),
+ "ip": ip if ip else ""
+ }
+ sq = utility.funcs.sql()
+ row = sq.insert(web.s._SCHEMA_REDIRECTS, data)
+ if not row:
+ logging.debug("Unable to insert: %s", data)
+ return False
+ id = sq.wrapper.cursor.lastrowid
+ if not slug:
+ logging.debug("Got %s from slug add: %s", id, url)
+ slug = id_to_slug(id)
+ update = {
+ "short": slug
+ }
+ if utility.funcs.sql().where("id", id).update(web.s._SCHEMA_REDIRECTS, update):
+ slug = utility.funcs.sql().where("id", id).get(web.s._SCHEMA_REDIRECTS)[0]
+ return (slug["short"], slug["id"], slug["delete"], slug["long"], False,)
+ else:
+ return False
+
+### conversions
+
+def slug_to_id(slug):
+ """ Turns a short slug into the row ID equivalent, using base62.
+
+ For custom slugs, this is simply done by querying the database.
+
+ :param slug: the shortened slug to convert
+ :type slug: str
+ :return: the slug's ID, if any.
+ :rtype: int
+ :raises: ValueError
+ """
+ if slug[:len(web.s._CUSTOM_CHAR)] == web.s._CUSTOM_CHAR:
+ rows = utility.funcs.sql().where("short", slug).get(web.s._SCHEMA_REDIRECTS)
+ if len(rows) != 1:
+ return False
+ return rows[0]["id"]
+
+ if not is_valid(slug):
+ return False
+ if not isinstance(slug, basestring):
+ raise ValueError("Wanted str, got " + str(type(slug)))
+ return base62.saturate(slug)
+
+def id_to_slug(id):
+ """ Dehydrates an integer into the string equivalent.
+
+ :param id: the id to saturate
+ :type id: int
+ :return: the saturated slug, if any
+ :rtype: int
+ :raises: ValueError
+ """
+ if not isinstance(id, int):
+ raise ValueError("Wanted int, got " + str(type(id)))
+ return base62.dehydrate(id)
+
+def to_row(slug):
+ """ Returns all database row information relating to a slug.
+
+ Custom slug info is found by querying the database.
+
+ :param slug: the slug to pull the data for
+ :type slug: str
+ :return: the slug's database row (False if not found)
+ :rtype: dict or False
+ """
+ if not is_valid(slug):
+ return False
+
+ if slug[:len(web.s._CUSTOM_CHAR)] == web.s._CUSTOM_CHAR:
+ rows = utility.funcs.sql().where("short", slug).get(web.s._SCHEMA_REDIRECTS)
+ else:
+ rows = utility.funcs.sql().where("id", slug_to_id(slug)).get(web.s._SCHEMA_REDIRECTS)
+ return rows[0] if len(rows) == 1 else False
+
+### info
+
+def hits(slug):
+ """ Return the number of clicks a slug has had.
+ This is done by pulling all the clicks from the database,
+ then counting them.
+
+ :param slug: the slug to count clicks for
+ :type slug: str
+ :return: the number of clicks the slug has
+ :rtype: int
+ """
+ if not is_valid(slug):
+ return 0
+ id = slug_to_id(slug)
+ rows = utility.funcs.sql().where("url", id).get(web.s._SCHEMA_CLICKS)
+ return len(rows)
+
+def exists(slug):
+ """ Check if a given slug actually exists.
+ This will just query the database and verify that there is only one row.
+
+ :param slug: the slug to check the existance of
+ :type slug: str
+ :return: if the slug exists
+ :rtype: bool
+ """
+ if not is_valid(slug):
+ return False
+ if slug[:len(web.s._CUSTOM_CHAR)] == web.s._CUSTOM_CHAR:
+ rows = utility.funcs.sql().where("short", slug).get(web.s._SCHEMA_REDIRECTS)
+ else:
+ rows = utility.funcs.sql().where("id", slug_to_id(slug)).get(web.s._SCHEMA_REDIRECTS)
+ return len(rows) == 1
+
+def is_valid(slug):
+ """ Attempts to saturate a slug, to check validity.
+ Custom slugs are assumed to be valid, unless otherwise shown.
+
+ :param slug: the slug to check the validity of
+ :type slug: str
+ :return: whether the slug is valid
+ :rtype: bool
+ """
+ if slug[:len(web.s._CUSTOM_CHAR)] == web.s._CUSTOM_CHAR:
+ return True
+
+ try:
+ base62.saturate(slug)
+ return True
+ except (TypeError, ValueError) as e:
+ logging.debug(e)
+ return False
+
+def url_from_slug(slug):
+ """ Find the long URL a slug is hiding.
+ This is simply found by querying the database for the data.
+
+ :param slug: the slug to expand to a long url
+ :type slug: str
+ :return: the expanded URL (or False if invalid)
+ :rtype: str or bool
+ """
+ if not is_valid(slug):
+ return False
+ if not exists(slug):
+ return False
+ if slug[:len(web.s._CUSTOM_CHAR)] == web.s._CUSTOM_CHAR:
+ row = utility.funcs.sql().where("short", slug).get(web.s._SCHEMA_REDIRECTS)[0]
+ else:
+ row = utility.funcs.sql().where("id", slug_to_id(slug)).get(web.s._SCHEMA_REDIRECTS)[0]
+ return row["long"]
+
+### modification
+
+def add_hit(slug, ip, ua, time_=None):
+ """ Add a click for the given slug to the database.
+ Stores the user agent info, as well as IP address and timestamp.
+
+ :param slug: the slug to add a click for
+ :type slug: str
+ :param ip: the IP address of the clicking user
+ :type ip: str
+ :param ua: the Flask useragent object for the clicking user
+ :type ua: Flask useragent object
+ :param time\_: an optional unix timestamp to attach to the click (defaults to now)
+ :type time\_: int or None
+ :return: whether or not the insertation was successful
+ :rtype: bool
+ """
+ if not is_valid(slug):
+ return False
+ if time_ is None:
+ time_ = int(time.time())
+ id = slug_to_id(slug)
+ data = {
+ "url": id,
+ "ip": ip,
+ "time": time_,
+ # agent is max of 256 characters, for brevity.
+ "agent": ua.string[:256] if not ua.string == None else "",
+ "agent_platform": ua.platform if not ua.platform == None else "",
+ "agent_browser": ua.browser if not ua.browser == None else "",
+ "agent_version": ua.version if not ua.version == None else "",
+ "agent_language": ua.language if not ua.language == None else ""
+ }
+ return utility.funcs.sql().insert(web.s._SCHEMA_CLICKS, data)
+