2.0.0 (CTFd#741)

* Fix user and admin panel user/team graphs
* Closes CTFd#682
* Unify login and logout under specific functions
* Closes CTFd#659
* Rename Challenges.hidden to Challenges.state
* Start to clean up API and front end integration starting with profile updating
* Slightly cleaner code
* Clean API to respond with success, data, and status codes
* Simpler COUNTRIES_LIST and update profile to use COUNTRIES_LIST
* Lookup country code in users page. Update front end calls to get API data properly
* Fix some API endpoints and fix JS to process new responses
* Update config.py to support new values
* Closes CTFd#635
* Update some code to handle user types, add email domain whitelisting
* Write a logging wrapper
* Use logging wrapper for submissions
* Close CTFd#656
* Break up config.html to make it easier to maintain
* Fix logging, domain_whitelist, and config
* Improving views.py, starting to add Announcements
* Starting announcements front end
* Make it easier to see large images, clean up some more REST API differences
* Closes CTFd#668
* Update Proxyfix config to REVERSE_PROXY
* Add announcements front end
* Move creation/edit modals into seperate files. Start moving user updating into their admin profile pages.
* Update font-awesome to 5.4.1
* Switch to user-edit icon
* Update the update_check function to send up more anonymous data for statistics purposes.
* Start work on CTFd#640
* Add the user action modals and update API to fix responses
* Fix admin teams page
* Add challenge requirements
* Implement anonymous locked challenges
* Team editting from admin panel
* Switch from simple cache to filesystem cache
* Implements a Cache backed server side session (CTFd#658) and fixes Users editting endpoint
* Add our messaging for docs
* Closes CTFd#700
* Remove invalid import
* Move challenge enditting around a whole lot and probably break a bunch of things
* Show challenge names in prerequisites instead of challenge IDs
* Closes CTFd#661
* Change user templates to use url_for
* Remove extra function
* Rewrite admin panel to use url_for
* Fix events to work under subdirectories
* Start cleaning up config panel
* Fix filesystem uploader; deprecate view_challenges_unregistered, view_scoreboard_if_authed, prevent_registration, view_after_ctf; implement new visibility decorators
* Remove workshop mode, fix some glitches with the new visibility settings
* Fix ctf_logo on core theme
* Fix setup errors
* Removing default from get_config b/c of memoization issues and getting some tests working
* Relax email regex validation rule (CTFd#693)
* Update to pycodestyle and fix new lint errors
* Add a ctf_id to update_check
* Change challenge plugin layout. Rename mailgun configs to be more descriptive (Closes CTFd#702)
* Detect if people try to set routes with '/' to simplify CTFd#690
* Closes CTFd#690
* Clean up some code
* Clean up challenge submit to rate limit
* Fix js version compatability issue
* Close some TODOs
* Hide challenges if not authenticated
* Make set_config reset the cache for those config values
* Return 404 on empty challenges for /api/v1/<challenge_id>/solves
* Fix setting boolean configs
* Properly change account config settings
* Move datetimes to isoformat (Closes CTFd#703)
* Remove all .isoformat() calls because it isn't UTC aware (ends in Z). Switch to isoformat function & filter
* Make /v1/submissions endpoint work for admin submission creation
* Make oauth_id unique for Users and Teams
* Move challenge submission endpoint and implement mark solved. Fix some isoformat issues.
* Only show team's missing challenges if in team mode
* Adding support for Hints & Unlocks
* Update challenge submission url
* Fix encoding functions in Python3
* Fix hexencode in Python3
* Added functional tests for challenges API for non-admin users (CTFd#705)
* Set hint default type to be standard
* Fix some JS issues. Closes CTFd#704
* Implement session.regenerate on top of the CachingSessionInterface
* Challenge challenge attempt responses from numbers to strings
* Fix password updating for UserSchema
* Remove leftover challenge submission code
* Remove old migrations :(, resolve challenge requirements not loading correctly, move migration functions
*  Added functional tests for challenges/hints/admin API (CTFd#710)
* Fix helpers and re-add JSONLite
* Install MySQL 5.7
* Try more mysql
* Update password for mysql
* Fixing issuse in Users.get_solves
* Add new import/export code
* Switch to CTFdSerializer for Python 3
* Re-implement import exports and add a very flaky test
* Redesign submissions API response
* Get export to roundtrip in tests
* Int score b/c Decimal is not JSON serializeable
* Remove unused route methods
* Fix POST /api/v1/configs and start adding admin tests
* Add user_id and team_id to top/10
* Fix admin creating Teams
* Fix Team website validation
* Change admins_only to reply with a 403 if the request is JSON
* Organize admin tests and fix authed_only to return 403 on unauthed
* Adding check_account_visibility, check_score_visibility for /api/v1/teams/<team_id>/(solves|awards|fails)
* Fix teams/me endpoints again
* Fix users/me endpoints to return 403 if unauthed
* Fix Python 3 config API
* Add fetch and promise polyfills. (CTFd#712)
* Add exec to docker-entrypoint.sh (CTFd#713)
* Display import_ctf Exceptions via repr (CTFd#651)
- Wraps exceptions on `/admin/import` returned to users in a `repr()`, making debugging easier.
* Add error messages to the admin panel, fix schemas for users, start working on UI for imports/exports
* Make unauthed challenge submission attempt return 403 instead of 302, Fix user deletion, fix associated tests, remove TODOs
* Remove old means of creating solves
* Remove most of the content from teams.js and users.js
* Remove extra code from /challenges.js
* Fix POST'ing & PATCH'ing pages
* Make (users|teams)/fails return only count to users. Fix public score graphs to factor in awards
* Fix admin side scoregraphs. Fix Awardschemas for admins
* Add requirements to db migration
* Adding some team decorators
* Fix require_team_mode decorator
* Make verified emails decorator return 403 on JSON requests
* Redo initial revision
* Add SQLiteJSON back
* Adding ratelimit to /redirect and removing POST from /oauth
* Fix PATCH tags
* Actually fix PATCH tags
* Simplify 500.html
* Added tests for challenges, awards, files, flags, hints ... (CTFd#723)
* Added tests for challenges, awards, files, flags, hints, notifications, pages, submissions, tags
* Fix user data validation functions, Fix hidden challenges and include test
* Add a locked state to attempt
* OAuth teams get verified, use logging functions in redirect route
* Removing extra print call
* Update requirements.txt
* Fix possible AttributeError
* Start work on CTFd#716
* Closes CTFd#717
* Fix issue patching teams
* Rename .j2 to .html, implement preview for challenges if admin
* Move admin/challenge.html to admin/challenges/challenge.html
* Remove old modals
* Add Reset CTF button (CTFd#639)
* Add Reset link to config.html
* Delete Tracking
* files handler should return a 404 on files it cant find
* Denote official teams (CTFd#729), make scoregraph fill to zero
* Remove old javascript files, make some challenge elements refresh by reloading
* Fix team editting modals to work more reliably
* Fix rendering of CTF paused
* Remove hide_scores funtion and roll it into scores visibility
* Log to stdout/stderr by default (CTFd#719)
* Fix user searching
* Remove searching for users/teams by country
* Add badges to admin team and user pages, implement user banning (CTFd#643)
* Remove shell.py, clean up admin team.html, add tests for banned users, teams
* Start cleaning up dynamic_challenges to meet new challenge type plugin format
* Remove POST method from teams.public
* Add credentials: 'same-origin' to all fetch calls (CTFd#734)
* Add challenge preview, add challenge deletion, fix file deletions when deleting challenges
* Fix imports UI (CTFd#735)
* Show prerequisites before adding a blank one (CTFd#738), Refresh all challenges after a submission (CTFd#739)
* Admins can see hidden challenges
* Fix some UI elements, fix loading location hash, set version to be 2.0.0
* Clean up some challenge plugin pages
* Add default for flag type
* Fix Python3 bytes/str issues
* Add in MLC urls and support user mode for oauth
* Fix seeing user graphs when scores are hidden, clean up setup.html, add links to MLC oauth
* Add state parameter support
* Use URLSafeTimedSerializer wrapper for sending token based emails
* setting APPLICATION_ROOT from env var (CTFd#732)
* Rearrange config.py and update README
* Updating README

.travis.yml

@@ -2,19 +2,30 @@ language: python
 services:
     - mysql
     - postgresql
+addons:
+  apt:
+    sources:
+    - mysql-5.7-trusty
+    packages:
+    - mysql-server
+    - mysql-client
 env:
-    - TESTING_DATABASE_URL='mysql+pymysql://root@localhost/ctfd'
+    - TESTING_DATABASE_URL='mysql+pymysql://root:password@localhost/ctfd'
     - TESTING_DATABASE_URL='sqlite://'
     - TESTING_DATABASE_URL='postgres://postgres@localhost/ctfd'
 python:
     - 2.7
     - 3.6
+before_install:
+    - sudo mysql -e "use mysql; update user set authentication_string=PASSWORD('password') where User='root'; update user set plugin='mysql_native_password';FLUSH PRIVILEGES;"
+    - sudo mysql_upgrade -u root -ppassword
+    - sudo service mysql restart
 install:
     - pip install -r development.txt
 before_script:
     - psql -c 'create database ctfd;' -U postgres
 script:
-    - pep8 --ignore E501,E712 CTFd/ tests/
+    - pycodestyle --ignore E501,E712,E402 CTFd/ tests/
     - nosetests -v -d --with-randomly
 after_success:
     - codecov

CTFd/__init__.py

@@ -3,27 +3,35 @@
 
 from distutils.version import StrictVersion
 from flask import Flask
+from werkzeug.contrib.fixers import ProxyFix
 from jinja2 import FileSystemLoader
 from jinja2.sandbox import SandboxedEnvironment
 from sqlalchemy.engine.url import make_url
 from sqlalchemy_utils import database_exists, create_database
 from six.moves import input
 
-from CTFd.utils import cache, migrate, migrate_upgrade, migrate_stamp, update_check
 from CTFd import utils
+from CTFd.utils.migrations import migrations, migrate, upgrade, stamp, create_database
+from CTFd.utils.sessions import CachingSessionInterface
+from CTFd.utils.updates import update_check
+from CTFd.utils.initialization import init_request_processors, init_template_filters, init_template_globals
+from CTFd.utils.events import socketio
+from CTFd.plugins import init_plugins
 
 # Hack to support Unicode in Python 2 properly
 if sys.version_info[0] < 3:
     reload(sys)
     sys.setdefaultencoding("utf-8")
 
-__version__ = '1.2.0'
+__version__ = '2.0.0'
 
 
 class CTFdFlask(Flask):
     def __init__(self, *args, **kwargs):
         """Overriden Jinja constructor setting a custom jinja_environment"""
         self.jinja_environment = SandboxedBaseEnvironment
+        self.jinja_environment.cache = None
+        self.session_interface = CachingSessionInterface(key_prefix='session')
         Flask.__init__(self, *args, **kwargs)
 
     def create_jinja_environment(self):
@@ -78,7 +86,7 @@ def confirm_upgrade():
 
 
 def run_upgrade():
-    migrate_upgrade()
+    upgrade()
     utils.set_config('ctf_version', __version__)
 
 
@@ -90,21 +98,9 @@ def create_app(config='CTFd.config.Config'):
         theme_loader = ThemeLoader(os.path.join(app.root_path, 'themes'), followlinks=True)
         app.jinja_loader = theme_loader
 
-        from CTFd.models import db, Teams, Solves, Challenges, WrongKeys, Keys, Tags, Files, Tracking
+        from CTFd.models import db, Teams, Solves, Challenges, Fails, Flags, Tags, Files, Tracking
 
-        url = make_url(app.config['SQLALCHEMY_DATABASE_URI'])
-        if url.drivername == 'postgres':
-            url.drivername = 'postgresql'
-
-        if url.drivername.startswith('mysql'):
-            url.query['charset'] = 'utf8mb4'
-
-        # Creates database if the database database does not exist
-        if not database_exists(url):
-            if url.drivername.startswith('mysql'):
-                create_database(url, encoding='utf8mb4')
-            else:
-                create_database(url)
+        url = create_database()
 
         # This allows any changes to the SQLALCHEMY_DATABASE_URI to get pushed back in
         # This is mostly so we can force MySQL's charset
@@ -114,32 +110,37 @@ def create_app(config='CTFd.config.Config'):
         db.init_app(app)
 
         # Register Flask-Migrate
-        migrate.init_app(app, db)
+        migrations.init_app(app, db)
 
         # Alembic sqlite support is lacking so we should just create_all anyway
         if url.drivername.startswith('sqlite'):
             db.create_all()
+            stamp()
         else:
-            if len(db.engine.table_names()) == 0:
-                # This creates tables instead of db.create_all()
-                # Allows migrations to happen properly
-                migrate_upgrade()
-            elif 'alembic_version' not in db.engine.table_names():
-                # There is no alembic_version because CTFd is from before it had migrations
-                # Stamp it to the base migration
-                if confirm_upgrade():
-                    migrate_stamp(revision='cb3cfcc47e2f')
-                    run_upgrade()
-                else:
-                    exit()
+            # This creates tables instead of db.create_all()
+            # Allows migrations to happen properly
+            upgrade()
+
+        from CTFd.models import ma
+
+        ma.init_app(app)
 
         app.db = db
         app.VERSION = __version__
 
+        from CTFd.cache import cache
+
         cache.init_app(app)
         app.cache = cache
 
-        update_check(force=True)
+        # If you have multiple workers you must have a shared cache
+        socketio.init_app(
+            app,
+            message_queue=app.config.get('CACHE_REDIS_URL')
+        )
+
+        if app.config.get('REVERSE_PROXY'):
+            app.wsgi_app = ProxyFix(app.wsgi_app)
 
         version = utils.get_config('ctf_version')
 
@@ -156,31 +157,39 @@ def create_app(config='CTFd.config.Config'):
         if not utils.get_config('ctf_theme'):
             utils.set_config('ctf_theme', 'core')
 
+        update_check(force=True)
+
+        init_request_processors(app)
+        init_template_filters(app)
+        init_template_globals(app)
+
+        # Importing here allows tests to use sensible names (e.g. api instead of api_bp)
         from CTFd.views import views
+        from CTFd.teams import teams
+        from CTFd.users import users
         from CTFd.challenges import challenges
         from CTFd.scoreboard import scoreboard
         from CTFd.auth import auth
-        from CTFd.admin import admin, admin_statistics, admin_challenges, admin_pages, admin_scoreboard, admin_keys, admin_teams
-        from CTFd.utils import init_utils, init_errors, init_logs
-
-        init_utils(app)
-        init_errors(app)
-        init_logs(app)
+        from CTFd.admin import admin
+        from CTFd.api import api
+        from CTFd.events import events
+        from CTFd.errors import page_not_found, forbidden, general_error, gateway_error
 
         app.register_blueprint(views)
+        app.register_blueprint(teams)
+        app.register_blueprint(users)
         app.register_blueprint(challenges)
         app.register_blueprint(scoreboard)
         app.register_blueprint(auth)
+        app.register_blueprint(api)
+        app.register_blueprint(events)
 
         app.register_blueprint(admin)
-        app.register_blueprint(admin_statistics)
-        app.register_blueprint(admin_challenges)
-        app.register_blueprint(admin_teams)
-        app.register_blueprint(admin_scoreboard)
-        app.register_blueprint(admin_keys)
-        app.register_blueprint(admin_pages)
-
-        from CTFd.plugins import init_plugins
+
+        app.register_error_handler(404, page_not_found)
+        app.register_error_handler(403, forbidden)
+        app.register_error_handler(500, general_error)
+        app.register_error_handler(502, gateway_error)
 
         init_plugins(app)