Merge pull request #1 from Yupeek/develop

merge develop into master for 1.0.0

.travis.yml

@@ -5,13 +5,15 @@ sudo: false
 language: python
 python:
   - "2.7"
+  - "3.3"
   - "3.4"
   - "3.5"
 
 # Django versions for matrix
 env:
   - DJANGO_VERSION='>=1.8,<1.9'
   - DJANGO_VERSION='>=1.9,<1.10'
+  - DJANGO_VERSION='>=1.10,<1.11'
 
 # Command to install dependencies, e.g. pip install -r requirements.txt --use-mirrors
 install:
@@ -22,6 +24,9 @@ install:
   - if [[ $TRAVIS_PYTHON_VERSION == '3.2' ]]; then travis_retry pip install 'coverage<4.0'; fi
   - pip install -r requirements.txt
 
+before_script:
+  - isort --recursive --check-only --diff dynamic_logging testproject
+  - flake8 dynamic_logging testproject
 
 # Command to run tests, e.g. python setup.py test
 script:
@@ -34,5 +39,7 @@ after_success:
 
 matrix:
   exclude:
-    - python: "3.5"
-      env: DJANGO_VERSION='>=1.8,<1.9'
+    - python: "3.3"
+      env: DJANGO_VERSION='>=1.9,<1.10'
+    - python: "3.3"
+      env: DJANGO_VERSION='>=1.10,<1.11'

MANIFEST.in

@@ -3,3 +3,5 @@ include README.rst
 include CONTRIBUTING.rst
 include requirements.txt
 include test_requirements.txt
+recursive-include dynamic_logging/static *.js *.css
+recursive-include dynamic_logging/templates *.html

README.rst

@@ -2,16 +2,15 @@
 django-dynamic-loging
 =====================
 
-allow to change the logging configuration for running production website.
-you found a bug, and the current stacktrace is not enouth ? first, you should
-change the settings LOGGING to make it more verbose (with something like LEVEL: 'DEBUG')
+allow to change the logging configuration for running production website from the admin interface.
+
+all you need to do is :
+
+1. create a logging config [Config model] from a use friendly form
+2. create a timelaps [Trigger model] in which your config is valid (from start_date to end_date, or forever)
+3. enjoy your logging, since you saved the trigger, the app will do the stuff to run it now if it's already
+   valide, or at the date/time you enabled it
 
-but, the bad thing, is that you must:
-1. connect to your production server
-2. change the settings in live
-3. make sure no synthax error
-4. restart the service (with downtime)
-5. do not forget to rollback after some time to prevent performance issues.
 
 stable branche
 
@@ -49,6 +48,45 @@ development status
      :alt: Requirements Status
 
 
+the old way was :
+
+you found a bug, and the current stacktrace is not enough ? first, you should
+change the settings LOGGING to make it more verbose (with something like LEVEL: 'DEBUG')
+
+but, the bad thing, is that you must:
+1. connect to your production server
+2. change the settings in live
+3. make sure no synthax error
+4. restart the service (with downtime)
+5. do not forget to rollback after some time to prevent performance issues.
+
+
+
+
+Overview
+--------
+
+
+with django-dynamic-logging, you can update at runtime the logging configuration, including :
+
+- update handlers levels and filter, but nothing else (for security purpose)
+- create/delete/update loggers. this include the level, handlers, filters and propagate flag.
+
+may logging configuration can exists in the database, but only one can be active. the leatest trigger (start_date) take
+precedence and will activate his config.
+
+ie: you want to set the app «myproject.import» in debug mode to for this night: you set the trigger, and it will
+enable the debug only for this night. at day, the default logging config will run
+
+**screenshots**
+
+.. image:: https://github.com/Yupeek/django-dynamic-logging/raw/develop/testproject/static/screenshot/DL_home.png
+     :alt: home
+
+.. image:: https://github.com/Yupeek/django-dynamic-logging/raw/develop/testproject/static/screenshot/create_config.png
+     :alt: home
+
+
 Installation
 ------------
 
@@ -68,8 +106,82 @@ the supported versions is the same as current django
 - python 2.7, 3.4, 3.5
 - django 1.8, 1.9, 1.10
 
-configuration
+configuration in sources
+------------------------
+
+1. add `dynamic-logging` to your INSTALLED_APPS
+
+and that's all
+
+configuration for running system
+--------------------------------
+
+1. go to your admin, and create a Config
+2. create the Trigger that will enable it whenever you want.
+
+
+.. _propagation:
+
+propagation of new config
+-------------------------
+
+each time a config or trigger is updated/deleted/created, the dynamic_logging system must recalculate the new config.
+but to work, it must be aware of the fact the something was updated. to make it available, there is 3 possibility.
+in mono-processing, where the logging config is global to all thread, it's not a issues, but in multi-process (like with
+gunicorn setup) or even multi-server, we must propagate the info that one running instance has just changed something in
+the config.
+
+for doing this, there is 4 Propagator shiped with dynamic_logging:
+
+- ``ThreadSignalPropagator``: the default one, it work in real-time in a mono-server, mono-process setup. it may not be possible
+  in real production to have this setup.
+- ``DummyPropagator``: nothing happen whene a config is updated. all the triggers and next trigger application is computed only
+  at startup time
+- ``TimerPropagator``: it check a modification in the config each `interval` seconds. this work, but is ineficient.
+- ``AmqpPropagator``: the best choice for production, but it require a running Amqp message queue broker (tested upon RabbitMQ).
+  it take in config the url of the server, and will connect each running instance to it. each time an instance update the config,
+  all instance will be triggered and will reload theire config in near realtime.
+
+
+to change the propagator, you can use the folowing settings:
+
+.. code-block:: python
+
+    DYNAMIC_LOGGING = {
+        "upgrade_propagator": {'class': "dynamic_logging.propagator.AmqpPropagator",
+                               'config': {'url': 'amqp://guest:guest@localhost:5672/%2F'}}
+    }
+
+specials cases
+--------------
+
+django-dynamic-logging handle some specials cases for you by default.
+
+- if you update a config or a trigger it will compute the current config and the next one on all running
+  instance of your website (see propagation_)
+
+- if you enable the DEBUG (or lesser) level on django.db.backends, it will change the settings of your
+  databases connection to make sure the CursorDebugWrapper is used and will call the debug for all query.
+  if not, you will not see any query by default.
+
+you can override or add some special cases by adding your own special cases in
+`dynamic_logging.signals.AutoSignalsHandler.extra_signals`.
+
+settings
+--------
+
+you can add into your settings a DYNAMIC_LOGGING dict with the folowing key to customise the dynamic logger behavior
+
+- signals_auto: the list of special logging handlers. currently only db_debug is enabled
+- config_upgrade_propagator: the class that is charged to trigger a scheduler reload for all running instances of the website.
+  see propagation_
+
+
+what's next ?
 -------------
 
-add `dynamic-logging` to your INSTALLED_APPS
 
+some of the next feature can be:
+
+- live logging browser (via websocket)
+- push/pull configuration from/to othes servers (via amqp)

dynamic_logging/__init__.py

@@ -1,2 +1,2 @@
-__VERSION__ = '1.0.0'
+__version__ = '1.0.0'
 default_app_config = 'dynamic_logging.apps.DynamicLoggingConfig'

dynamic_logging/admin.py

@@ -1 +1,76 @@
 # -*- coding: utf-8 -*-
+from django.contrib import admin
+from django.core.urlresolvers import reverse
+from django.db import models
+from django.template.defaultfilters import safe
+from django.utils.translation import ugettext_lazy as _
+
+from dynamic_logging.scheduler import main_scheduler
+from dynamic_logging.widgets import JsonLoggerWidget
+
+from .models import Config, Trigger
+
+
+@admin.register(Config)
+class ConfigAdmin(admin.ModelAdmin):
+    list_display = ['name', 'config_is_running', 'link_to_triggers', 'add_trigger']
+    formfield_overrides = {
+        models.TextField: {'label': 'settings', 'widget': JsonLoggerWidget},
+    }
+
+    def config_is_running(self, obj):
+        return main_scheduler.current_trigger.config == obj
+
+    config_is_running.boolean = True
+    config_is_running.short_description = _('config is running')
+
+    def link_to_triggers(self, obj):
+        return safe('<a href="%s?config=%d">%d trigger(s)</a>' % (
+            reverse('admin:dynamic_logging_trigger_changelist'),
+            obj.pk,
+            obj.triggers.count()
+        ))
+
+    def get_changeform_initial_data(self, request):
+        return {'config_json': Config.default().config_json}
+
+    def add_trigger(self, obj):
+        return safe('<a href="%s?config=%d">add trigger</a>' % (reverse('admin:dynamic_logging_trigger_add'), obj.pk))
+
+    class Media:
+        css = {'all': ('admin/css/dynamic_logging.css',
+                       'admin/css/forms.css')}
+
+        js = ('admin/js/collapse.min.js', )
+
+    def changelist_view(self, request, extra_context=None):
+        extra_context = extra_context or {}
+        extra_context['current_trigger'] = main_scheduler.current_trigger
+        extra_context['next_trigger'] = main_scheduler.next_timer and main_scheduler.next_timer.trigger
+        # add extra data for handlers in each loggers
+        loggers = list(main_scheduler.current_trigger.config.config.get('loggers', {}).values())
+        if main_scheduler.next_timer:
+            extra_context['next_trigger'] = main_scheduler.next_timer.trigger
+            loggers += list(main_scheduler.next_timer.trigger.config.config.get('loggers', {}).values())
+
+        return super(ConfigAdmin, self).changelist_view(request, extra_context)
+
+
+@admin.register(Trigger)
+class TriggerAdmin(admin.ModelAdmin):
+    list_display = ['name', 'start_date', 'end_date', 'is_active', 'config_is_running', 'link_to_config']
+    date_hierarchy = 'start_date'
+    list_filter = ['is_active', 'start_date', 'end_date', 'config']
+    list_editable = ['is_active']
+
+    def link_to_config(self, obj):
+        return safe('<a href="%s">%s</a>' % (
+            reverse('admin:dynamic_logging_config_change', args=(obj.config_id, )),
+            obj.config.name
+        ))
+
+    def config_is_running(self, obj):
+        return main_scheduler.current_trigger == obj
+
+    config_is_running.boolean = True
+    config_is_running.short_description = _('config is running')

dynamic_logging/apps.py

@@ -4,25 +4,44 @@
 import logging
 
 from django.apps.config import AppConfig
-from django.db.models.signals import post_delete, post_save
+from django.core.signals import setting_changed
 from django.db.utils import OperationalError
 
+from dynamic_logging.settings import get_setting
+from dynamic_logging.signals import AutoSignalsHandler
+
 logger = logging.getLogger(__name__)
 
 
 class DynamicLoggingConfig(AppConfig):
     name = 'dynamic_logging'
+    auto_signal_handler = AutoSignalsHandler()
+
+    def __init__(self, *args, **kwargs):
+        self.propagator = None
+        super(DynamicLoggingConfig, self).__init__(*args, **kwargs)
+
+    def on_settings_changed(self, sender, setting, *args, **kwargs):
+        if setting == 'DYNAMIC_LOGGING':
+            self.setup_propagator()
+
+    def setup_propagator(self):
+        from dynamic_logging.propagator import Propagator
+        if self.propagator is not None:
+            self.propagator.teardown()
+        self.propagator = Propagator.get_current()
+        self.propagator.setup()
 
     def ready(self):
         # import at ready time to prevent model loading before app ready
         from dynamic_logging.scheduler import main_scheduler
         try:
-            main_scheduler.reload()
-        except OperationalError:
+            main_scheduler.reload(2)  # 2 sec to prevent unit-tests to load the production database
+        except OperationalError:  # pragma: nocover
             pass  # no trigger table exists atm. we don't care since there is no Trigger to pull.
         # setup signals for Trigger changes. it will reload the current trigger and next one
-        from .signals import reload_timers_on_trigger_change
-        Trigger = self.get_model('Trigger')
 
-        post_save.connect(reload_timers_on_trigger_change, sender=Trigger)
-        post_delete.connect(reload_timers_on_trigger_change, sender=Trigger)
+        self.auto_signal_handler.apply(get_setting('signals_auto'))
+        self.setup_propagator()
+
+        setting_changed.connect(self.on_settings_changed)

dynamic_logging/handlers.py

@@ -2,14 +2,28 @@
 from __future__ import absolute_import, print_function, unicode_literals
 
 import logging
+import threading
 from collections import defaultdict
+from contextlib import contextmanager
 
 logger = logging.getLogger(__name__)
 
 
 class MockHandler(logging.Handler):
     """Mock logging handler to check for expected logs."""
-    messages = defaultdict(list)
+
+    _messages_by_thread = defaultdict(list)
+
+    @classmethod
+    @contextmanager
+    def capture(cls):
+        current = defaultdict(list)
+        cls._messages_by_thread[threading.current_thread()].append(current)
+        try:
+            yield current
+        finally:
+            cls._messages_by_thread[threading.current_thread()].remove(current)
 
     def emit(self, record):
-        self.messages[record.levelname.lower()].append(record.getMessage())
+        for messages_list in self._messages_by_thread[threading.current_thread()]:
+            messages_list[record.levelname.lower()].append(record.getMessage())

dynamic_logging/migrations/0001_initial.py

@@ -16,19 +16,22 @@ class Migration(migrations.Migration):
         migrations.CreateModel(
             name='Config',
             fields=[
-                ('id', models.AutoField(verbose_name='ID', auto_created=True, primary_key=True, serialize=False)),
+                ('id', models.AutoField(auto_created=True, serialize=False, primary_key=True, verbose_name='ID')),
                 ('name', models.CharField(max_length=255)),
-                ('config_json', models.TextField()),
+                ('config_json', models.TextField(default='{}', validators=[dynamic_logging.models.json_value])),
+                ('last_update', models.DateTimeField(auto_now=True)),
             ],
         ),
         migrations.CreateModel(
             name='Trigger',
             fields=[
-                ('id', models.AutoField(verbose_name='ID', auto_created=True, primary_key=True, serialize=False)),
+                ('id', models.AutoField(auto_created=True, serialize=False, primary_key=True, verbose_name='ID')),
                 ('name', models.CharField(max_length=255)),
-                ('start_date', models.DateTimeField(null=True, default=django.utils.timezone.now)),
-                ('end_date', models.DateTimeField(null=True, default=dynamic_logging.models.now_plus_2hours)),
-                ('config', models.ForeignKey(to='dynamic_logging.Config', related_name='triggers')),
+                ('is_active', models.BooleanField(default=True)),
+                ('start_date', models.DateTimeField(default=django.utils.timezone.now, null=True, blank=True)),
+                ('end_date', models.DateTimeField(default=dynamic_logging.models.now_plus_2hours, null=True, blank=True)),
+                ('last_update', models.DateTimeField(auto_now=True)),
+                ('config', models.ForeignKey(related_name='triggers', to='dynamic_logging.Config')),
             ],
             options={
                 'get_latest_by': 'start_date',