Merge pull request #161 from Daniel-at-github/ReST-validation-error

Fix ReST validation errors and Makefile verbose

Makefile

@@ -28,18 +28,17 @@ test-schemas: install-deps
 	$(PY) $(TESTSDIR)/schemas.py -d $(BASEDIR) -s $(SCHEMASDIR) -v $(VERBOSE)
 
 test-ids-unique: install-deps
-	$(PY) $(TESTSDIR)/ids_unique.py -d $(BASEDIR)
+	$(PY) $(TESTSDIR)/ids_unique.py -d $(BASEDIR) -v $(VERBOSE)
 
 test-slugs-unique: install-deps
-	$(PY) $(TESTSDIR)/slugs_unique.py -d $(BASEDIR)
+	$(PY) $(TESTSDIR)/slugs_unique.py -d $(BASEDIR) -v $(VERBOSE)
 
 test-render-rest: install-deps
-	$(PY) $(TESTSDIR)/render_rest.py -d $(BASEDIR)
+	$(PY) $(TESTSDIR)/render_rest.py -d $(BASEDIR) -v $(VERBOSE)
 
 test-shape: install-deps
-	$(PY) $(TESTSDIR)/shape.py -d $(BASEDIR) -v 1
+	$(PY) $(TESTSDIR)/shape.py -d $(BASEDIR) -v $(VERBOSE)
 
 test: test-schemas test-ids-unique test-slugs-unique test-render-rest
 
 .PHONY: help test test-schemas test-ids-unique test-slugs-unique test-render-rest test-shape
-

djangocon-2012/videos/cryptography-for-django-applications.json

@@ -2,7 +2,7 @@
   "alias": "video/1381/cryptography-for-django-applications",
   "category": "DjangoCon 2012",
   "copyright_text": "Creative Commons Attribution license (reuse allowed",
-  "description": "Introduction\n============\n\nThe web is a hostile place, and isn't showing any signs of becoming less\nso. In order to mitigate this, many developers turn to cryptography.\nUnfortunately, cryptography can be complicated, and is easily\ncircumvented if not properly handled. This presentation will provide an\nintroduction to cryptographic tools available to Python/Django\napplications, appropriate use cases for each, proper usage, and\noperational concerns necessary to operate in a certified environment.\nFinally, we will also demonstrate a reusable application that wraps this\nall up, providing secure key-management capabilities to a running Django\nenvironment via the Django admin.\n\nWhy Encrypt?\n============\n\nRules of Encryption\n===================\n\n-  Don't do it if you don't need it.\n-  Don't write your own.\n-  Understand what you're doing if you do.\n\nWhen to encrypt?\n================\n\nUnderstand what you're protecting\n---------------------------------\n\n-  Data\n-  User records\n-  Code\n-  Systems\n\nUnderstand your attack vectors\n------------------------------\n\n-  Data (backups, revision control)\n-  Systems\n-  Application\n-  Transport\n-  Client\n\nUnderstand the types of encryption you might use:\n-------------------------------------------------\n\n-  Hashing\n\nPasswords are a special case. Use a key derivation function\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-  PBKDF2 \u2013 Upgrade to Django 1.4!\n\nAlgorithms\n~~~~~~~~~~\n\n-  MD5 - fine as a checksum. not fine as a cryptographic hash.\n-  SHA1 - fine as a checksum. becoming less fine as a cryptographic hash\n   every day\n-  SHA2 - so far so good. use as many bits as you can handle.\n\nSymmetric Encryption\n--------------------\n\n-  Fast\n-  Reversible\n\nAlgorithms\n~~~~~~~~~~\n\n-  Caesar Cipher (for fun puzzles)\n-  DES (don't use)\n-  AES (certified)\n-  Blowfish\n\nAsymmetric Encryption\n---------------------\n\n-  Slow\n-  One-way\n\nAlgorithms\n~~~~~~~~~~\n\n-  RSA\n-  DSA\n\nUses\n^^^^\n\nSigning\n'''''''\n\nWeb of Trust\n            \n\n-  PGP\n\nPKI\n   \n\nEncryption\n''''''''''\n\n-  PGP\n-  SSL\n-  TLS\n\nDoing it right\n==============\n\nUse known-good algorithms\n-------------------------\n\n-  AES-256\n-  SHA2\n-  RSA\n-  DSA\n\nUse known-good implementations\n------------------------------\n\n-  Open Source is good\n\nExtra Credit\n------------\n\n-  FIPS 140 certified implementations\n-  FIPS 140 / NIST configurations\n\nTransport (always use HTTPS)\n----------------------------\n\n-  Use good algorithms AES-256\n\nAt Rest (insecure servers or backups)\n-------------------------------------\n\n-  Understand the ramifications of key management\n\nExamples\n========\n\nHashing\n-------\n\n-  Use a key-derivation function\n\nDon't be linked-in\n~~~~~~~~~~~~~~~~~~\n\n-  Salt your hashes (with a secret).\n-  Salt and pepper your hashes if possible (with a known unique value)\n\nSSL\n---\n\n-  Forced connections\n-  Making the application aware\n-  Hardened cipher selection\n\nRobust PKI\n~~~~~~~~~~\n\n-  Client authentication\n-  SSL Test Page\n\nAsymmetric Encryption\n---------------------\n\nKey Management\n~~~~~~~~~~~~~~\n\n-  Using GPG Agent\n-  GPG Manager App\n\nPGP Files\n~~~~~~~~~\n\nSymmetric Encryption\n--------------------\n\nKey Management\n~~~~~~~~~~~~~~\n\n-  Use Asymmetric Encryption\n\nUse a unique Initialization Vector if possible\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-  LoopBack Devices\n\n",
+  "description": "Introduction\n============\n\nThe web is a hostile place, and isn't showing any signs of becoming less\nso. In order to mitigate this, many developers turn to cryptography.\nUnfortunately, cryptography can be complicated, and is easily\ncircumvented if not properly handled. This presentation will provide an\nintroduction to cryptographic tools available to Python/Django\napplications, appropriate use cases for each, proper usage, and\noperational concerns necessary to operate in a certified environment.\nFinally, we will also demonstrate a reusable application that wraps this\nall up, providing secure key-management capabilities to a running Django\nenvironment via the Django admin.\n\nWhy Encrypt?\n============\n\nRules of Encryption\n===================\n\n-  Don't do it if you don't need it.\n-  Don't write your own.\n-  Understand what you're doing if you do.\n\nWhen to encrypt?\n================\n\nUnderstand what you're protecting\n---------------------------------\n\n-  Data\n-  User records\n-  Code\n-  Systems\n\nUnderstand your attack vectors\n------------------------------\n\n-  Data (backups, revision control)\n-  Systems\n-  Application\n-  Transport\n-  Client\n\nUnderstand the types of encryption you might use:\n-------------------------------------------------\n\n-  Hashing\n\nPasswords are a special case. Use a key derivation function\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-  PBKDF2 \u2013 Upgrade to Django 1.4!\n\nHashing Algorithms\n~~~~~~~~~~~~~~~~~~\n\n-  MD5 - fine as a checksum. not fine as a cryptographic hash.\n-  SHA1 - fine as a checksum. becoming less fine as a cryptographic hash\n   every day\n-  SHA2 - so far so good. use as many bits as you can handle.\n\nSymmetric Encryption\n--------------------\n\n-  Fast\n-  Reversible\n\nAlgorithms (Symmetric)\n~~~~~~~~~~~~~~~~~~~~~~\n\n-  Caesar Cipher (for fun puzzles)\n-  DES (don't use)\n-  AES (certified)\n-  Blowfish\n\nAsymmetric Encryption\n---------------------\n\n-  Slow\n-  One-way\n\nAlgorithms (Asymmetric)\n~~~~~~~~~~~~~~~~~~~~~~~\n\n-  RSA\n-  DSA\n\nUses\n^^^^\n\nSigning\n'''''''\n\nWeb of Trust\n            \n\n-  PGP\n\nPKI\n   \n\nEncryption\n''''''''''\n\n-  PGP\n-  SSL\n-  TLS\n\nDoing it right\n==============\n\nUse known-good algorithms\n-------------------------\n\n-  AES-256\n-  SHA2\n-  RSA\n-  DSA\n\nUse known-good implementations\n------------------------------\n\n-  Open Source is good\n\nExtra Credit\n------------\n\n-  FIPS 140 certified implementations\n-  FIPS 140 / NIST configurations\n\nTransport (always use HTTPS)\n----------------------------\n\n-  Use good algorithms AES-256\n\nAt Rest (insecure servers or backups)\n-------------------------------------\n\n-  Understand the ramifications of key management\n\nExamples\n========\n\nHashing\n-------\n\n-  Use a key-derivation function\n\nDon't be linked-in\n~~~~~~~~~~~~~~~~~~\n\n-  Salt your hashes (with a secret).\n-  Salt and pepper your hashes if possible (with a known unique value)\n\nSSL\n---\n\n-  Forced connections\n-  Making the application aware\n-  Hardened cipher selection\n\nRobust PKI\n~~~~~~~~~~\n\n-  Client authentication\n-  SSL Test Page\n\nAsymmetric Encryption.\n----------------------\n\nKey Management\n~~~~~~~~~~~~~~\n\n-  Using GPG Agent\n-  GPG Manager App\n\nPGP Files\n~~~~~~~~~\n\nSymmetric Encryption.\n---------------------\n\nKey Management.\n~~~~~~~~~~~~~~~\n\n-  Use Asymmetric Encryption\n\nUse a unique Initialization Vector if possible\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n-  LoopBack Devices\n\n",
   "duration": null,
   "id": 1381,
   "language": "eng",

djangocon-2012/videos/using-celery-with-social-networks.json

@@ -2,7 +2,7 @@
   "alias": "video/1382/using-celery-with-social-networks",
   "category": "DjangoCon 2012",
   "copyright_text": "Creative Commons Attribution license (reuse allowed",
-  "description": "Twitter conditionally rate limits based on IP address rather than access\ntoken even when one is provided for some of its API calls. Facebook has\nat least 10 unique error messages to indicate a bad or expired access\ntoken (that I've found so far). LinkedIn's pagination has an occasional\noff-by-one bug resulting in an endless list of 1-user pages. Let's face\nit: interfacing with social networks is tricky. Celery helps, but to\nprovide stable, reliable, and fast social features for your website,\nyou'll need an arsenal of strategies and tools to get you the rest of\nthe way there.\n\nBy the end of this talk, you'll understand how to set up tasks to\nquickly serve users with massive networks by employing intelligent\ndistribution. You'll be able to design robust processes to handle\ninconsistencies or instabilities in 3rd party APIs. And you'll know how\nto have confidence that the work you intend to do gets done, regardless\nof external rate limits, pagination design, or API call dependency\nchains.\n\nThis talk is intended for people who have basic familiarity with celery\nand would like to learn more about how to take advantage of it for\nlarge, distributed task loads.\n\nOutline\n-------\n\nI. Intro\n\nA. 3rd party interfaces are hard\n\n::\n\n      * Speed\n\n        * Much slower than local data\n        * Users may still expect near-immediate results\n\n      * Rate limits\n\n        * Different rules for every service\n        * Need to handle reactive & proactive as some don't publish rates\n\n      * Instability\n\n        * Outages (yes, Facebook does go down)\n        * Random failures\n\nB. Why Celery?\n\n::\n\n      * Asynchronous\n      * Distributed\n      * Fault tolerant\n\nII.  Task Organization\n\n     A. Small, atomic tasks (1 API call per task) B. Minimal message\n     state\n\n     -  Primitive types only (no model instances!)\n     -  Defer as much data access to the task itself as possible\n\n     C. Create Task subclasses for common patterns D. Whenever possible,\n     make tasks idempotent\n\nIII. Task Distribution\n\n     A. Managing pagination\n\n     ::\n\n         * For a known set size\n\n           * Where limit/offset is supported, launch all page tasks simlutaneously\n           * Otherwise, 1 page launches the next as soon as the next cursor is obtained\n\n         * For an unknown set size\n\n           * Set max simultaneous pages\n           * Task is terminal if blank, otherwise launches page w/ offset + max pages\n\n         * Setting page size is an art, not a science\n\n           * Minimize the number of api calls when possible\n           * Avoid long-running tasks by setting a timeout ceiling\n           * Avoid the temptation to pass API data to dependent tasks\n\n     B. Tracking task dependencies (\"Done?\" is difficult for distributed\n     systems)\n\n     ::\n\n         * Use an external backend to store a dependency tree\n         * Subclass ResultSet to evaluate the task state of the tree\n         * Requires ignore_result=False\n\nIV.  Rate Limiting\n\n     A. Problems\n\n     -  Celery's rate limiting doesn't do what you think it does\n     -  3rd party rate limits depend on many factors\n\n     B. Solution\n\n     -  For services with known rate limits:\n\n        -  Use an external backend to store rate limit counters\n        -  Increment counters based on rate limit factors per api call\n\n     -  For services with unknown rate limits:\n\n        -  Use an external backend to store rate limit backoff counters\n        -  Ramp up / ratchet down call rate by power law as api calls\n           fail/succeed\n\nV.   Failover\n\nA. Problems\n\n::\n\n      * Celery's countdown doesn't do what you think it does\n      * 3rd parties can fail in lots of \"interesting\" ways\n\nB. Solution\n\n::\n\n      * Implement native RabbitMQ alternative to countdown\n      * Create task base classes per social network to handle error conditions\n\nVI.   Multiple queues\n\n      A. Better control over task priority management & resource\n      distribution B. Not all social accounts are created equal\n      (handling whales & spikes) C. When you can't stream updates, use a\n      trickle queue\n\nVII.  Celerybeat considered harmful\n\n      A. Periodic task persistence gets out of sync with code B. Just 1\n      more process to manage C. Cron: it's just. not. that. hard.\n\nVIII. Debugging\n\n      A. Don't use \"always eager\" B. Logging, logging, logging C. Unit\n      tests are good, but integration tests save lives\n\nIXIV. Gotchas\n\n      A. Open socket prevents Celery soft timeout B. Celery soft timeout\n      doesn't retry the task C. If result state is not known, Celery\n      reports \"PENDING\"\n\n\n",
+  "description": "Twitter conditionally rate limits based on IP address rather than access\ntoken even when one is provided for some of its API calls. Facebook has\nat least 10 unique error messages to indicate a bad or expired access\ntoken (that I've found so far). LinkedIn's pagination has an occasional\noff-by-one bug resulting in an endless list of 1-user pages. Let's face\nit: interfacing with social networks is tricky. Celery helps, but to\nprovide stable, reliable, and fast social features for your website,\nyou'll need an arsenal of strategies and tools to get you the rest of\nthe way there.\n\nBy the end of this talk, you'll understand how to set up tasks to\nquickly serve users with massive networks by employing intelligent\ndistribution. You'll be able to design robust processes to handle\ninconsistencies or instabilities in 3rd party APIs. And you'll know how\nto have confidence that the work you intend to do gets done, regardless\nof external rate limits, pagination design, or API call dependency\nchains.\n\nThis talk is intended for people who have basic familiarity with celery\nand would like to learn more about how to take advantage of it for\nlarge, distributed task loads.\n\nOutline\n=======\n\n#. Intro\n\n#. 3rd party interfaces are hard\n\n   -  Speed\n\n      -  Much slower than local data\n      -  Users may still expect near-immediate results\n\n   -  Rate limits\n\n      -  Different rules for every service\n      -  Need to handle reactive & proactive as some don't publish\n         rates\n\n   -  Instability\n\n      -  Outages (yes, Facebook does go down)\n      -  Random failures\n   -\n\n#. Why Celery?\n\n   -  Asynchronous\n   -  Distributed\n   -  Fault tolerant\n\n#. Task Organization\n\n   #. Small, atomic tasks (1 API call per task) B. Minimal message state\n\n      -  Primitive types only (no model instances!)\n      -  Defer as much data access to the task itself as possible\n\n   #. Create Task subclasses for common patterns D. Whenever possible,make tasks idempotent\n\n#. Task Distribution\n\n   #. Managing pagination\n\n          -  For a known set size\n\n             -  Where limit/offset is supported, launch all page tasks\n                simlutaneously\n             -  Otherwise, 1 page launches the next as soon as the next\n                cursor is obtained\n\n          -  For an unknown set size\n\n             -  Set max simultaneous pages\n             -  Task is terminal if blank, otherwise launches page w/\n                offset + max pages\n\n          -  Setting page size is an art, not a science\n\n             -  Minimize the number of api calls when possible\n             -  Avoid long-running tasks by setting a timeout ceiling\n             -  Avoid the temptation to pass API data to dependent tasks\n          -\n\n   #. Tracking task dependencies (\"Done?\" is difficult for distributed systems)\n\n       -  Use an external backend to store a dependency tree\n       -  Subclass ResultSet to evaluate the task state of the tree\n       -  Requires ignore\\_result=False\n\n#. Rate Limiting\n\n   #. Problems\n\n      -  Celery's rate limiting doesn't do what you think it does\n      -  3rd party rate limits depend on many factors\n\n   #. Solution\n\n      -  For services with known rate limits:\n    \n         -  Use an external backend to store rate limit counters\n         -  Increment counters based on rate limit factors per api call\n    \n      -  For services with unknown rate limits:\n    \n         -  Use an external backend to store rate limit backoff counters\n         -  Ramp up / ratchet down call rate by power law as api callsfail/succeed\n   #.\n    \n#. Failover\n\n#. Problems\n\n   -  Celery's countdown doesn't do what you think it does\n   -  3rd parties can fail in lots of \"interesting\" ways\n\n#. Solution\n\n   -  Implement native RabbitMQ alternative to countdown\n   -  Create task base classes per social network to handle error\n      conditions\n\n#. Multiple queues\n\n   #. Better control over task priority management & resource distribution\n   #. Not all social accounts are created equal (handling whales & spikes)\n   #. When you can't stream updates, use a trickle queue\n\n#. Celerybeat considered harmful\n\n   #. Periodic task persistence gets out of sync with code \n   #. Just 1 more process to manage\n   #. Cron: it's just. not. that. hard.\n\n#. Debugging\n\n   #. Don't use \"always eager\" \n   #. Logging, logging, logging \n   #. Unit tests are good, but integration tests save lives\n\n#. Gotchas\n\n    #. Open socket prevents Celery soft timeout \n    #. Celery soft timeout doesn't retry the task\n    #. If result state is not known, Celery reports \"PENDING\"\n",
   "duration": null,
   "id": 1382,
   "language": "eng",