Structured_Logging.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'logging' from '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py'>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reload logging for the Notebook\n",
    "\n",
    "import importlib\n",
    "import logging\n",
    "\n",
    "importlib.reload(logging)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "# Configure root logger behaviour to output all log levels to stdout\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(levelname)s - %(message)s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Structured logging in Python\n",
    "\n",
    "### Rob Van Gennip\n",
    "\n",
    "rob.vangennip.ca"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Good afternoon\n",
    "\n",
    "Thank you for joining me here today to talk about structured logging in Python\n",
    "\n",
    "I am Rob Van Gennip and I will try to end a couple minutes early so you can get ahead of the lunch line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# About Me\n",
    "\n",
    "- Engineering Lead at Wave\n",
    "- Over 3 years of using Python with Django\n",
    "- Focus on distributed systems and tooling\n",
    "- Previously > 5 years of C# and native Windows development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Based in Toronto\n",
    "\n",
    "Provides a suite of financial services and online software for small businesses, such as payroll, accounting, invoicing, and online payments \n",
    "\n",
    "Name transition in progress from Wave Accounting to Wave Financial to reflect we do more than accounting\n",
    "\n",
    "We are hiring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Topics\n",
    "\n",
    "- Log Levels\n",
    "- Logging Classes\n",
    "- Structured Logging\n",
    "- Tips"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Log Levels\n",
    "\n",
    "- Debug\n",
    "- Info\n",
    "- Warning\n",
    "- Error\n",
    "- Critical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Debug\n",
    "\n",
    "Detailed internal state for diagnosing problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DEBUG - Calculated sum of 3 and 4 as 7\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import logging\n",
    "\n",
    "logger = logging.getLogger()\n",
    "\n",
    "def sum(x, y):\n",
    "    result = x + y\n",
    "    logger.debug(f'Calculated sum of {x} and {y} as {result}')\n",
    "    return result\n",
    "\n",
    "sum(3, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Tracking an algorithm’s internal state\n",
    "\n",
    "Can help with tracing function calls\n",
    "\n",
    "Use instead of print function for developer output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Info\n",
    "\n",
    "Confirmation things are working as expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - User logged out\n"
     ]
    }
   ],
   "source": [
    "logger.info('User logged out')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Understand what is happening in the application flow\n",
    "\n",
    "A couple per operation, perhaps related to handling request or server state changed\n",
    "\n",
    "Server started; event processed; user logged in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Warning\n",
    "\n",
    "Something unexpected happened or indicative of a problem for the near future"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING - Unable to create short link\n"
     ]
    }
   ],
   "source": [
    "logger.warning('Unable to create short link')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Something important but not an error\n",
    "\n",
    "Disk space low"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Error\n",
    "\n",
    "Something is wrong, not able to perform an operation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR - Failed to update user\n"
     ]
    }
   ],
   "source": [
    "logger.error('Failed to update user', exc_info=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Failure to read or create a database record\n",
    "\n",
    "For exceptions, should capture all tracebacks \n",
    "\n",
    "exc_info: indicates if exception information is added to the logging message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR - Cannot divide by zero\n",
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-7-9203fb8052f4>\", line 5, in <module>\n",
      "    divide_by_zero()\n",
      "  File \"<ipython-input-7-9203fb8052f4>\", line 2, in divide_by_zero\n",
      "    x = 1/0\n",
      "ZeroDivisionError: division by zero\n"
     ]
    }
   ],
   "source": [
    "def divide_by_zero():\n",
    "    x = 1/0\n",
    "\n",
    "try:\n",
    "    divide_by_zero()\n",
    "except ZeroDivisionError:\n",
    "    # Equivalent to logger.error('Cannot divide by zero', exc_info=True)\n",
    "    logger.exception('Cannot divide by zero')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "`exception` should only be called from an exception handler\n",
    "\n",
    "exc_info accepts:\n",
    "- boolean\n",
    "- an exception instance\n",
    "- an exception tuple in the format returned by sys.exc_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Critical\n",
    "\n",
    "Something really bad happened, may be unable to continue running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CRITICAL - An unhandled exception was thrown by the application\n"
     ]
    }
   ],
   "source": [
    "logger.critical('An unhandled exception was thrown by the application')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Out of memory\n",
    "\n",
    "Disk is full"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Logging Classes\n",
    "\n",
    "- Loggers\n",
    "- Handlers\n",
    "- Filters\n",
    "- Formatters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Four key classes defined by the `logging` module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "importlib.reload(logging)\n",
    "\n",
    "# Include logger name in output\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format='%(levelname)s - %(name)s - %(message)s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Logger\n",
    "\n",
    "- Interface that application code directly uses\n",
    "- Named typically as a dot-separated hierarchy like `'a'`, `'a.b'` or `'a.b.c.d'`\n",
    "- Hierarchical and descendants of the root logger\n",
    "- Retrieve an instance with `logging.getLogger(name)`\n",
    "- Retrieving a logger with the same name returns a reference to the same instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Logger never instantiated directly, but always through `getLogger`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - root - Hello PyCon Canada!\n"
     ]
    }
   ],
   "source": [
    "root_logger = logging.getLogger()  # no name provided\n",
    "\n",
    "root_logger.info('Hello PyCon Canada!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - branch.leaf - Hello PyCon Canada!\n"
     ]
    }
   ],
   "source": [
    "leaf_logger = logging.getLogger('branch.leaf')\n",
    "\n",
    "leaf_logger.info('Hello PyCon Canada!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - __main__ - Hello PyCon Canada!\n"
     ]
    }
   ],
   "source": [
    "# Use __name__ to get the fully-qualified name of the module\n",
    "# Example: api/user.py → 'api.user'\n",
    "# __name__ is '__main__' in an interactive prompt, script, or standard input\n",
    "\n",
    "default_logger = logging.getLogger(__name__)  # recommended\n",
    "\n",
    "default_logger.info('Hello PyCon Canada!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Stick to the standard logger name\n",
    "\n",
    "When moving files around, be aware this will impact the logger used\n",
    "\n",
    "To keep backwards compatibility functionality, hardcode the name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Handler\n",
    "\n",
    "- Sends the log records to a destination\n",
    "- `StreamHandler`: output to streams such as `stdout`, `stderr`\n",
    "- `NullHandler`: no output\n",
    "- `FileHandler`: output to a disk file\n",
    "- `RotatingFileHandler`: rotates at max file size\n",
    "- `TimedRotatingFileHandler`: interval based rotation\n",
    "- `SmtpHandler`: send an email per record"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "`RotatingFileHandler`: old log files by append the extensions .1, .2 to the file name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Filter\n",
    "\n",
    "- Fine grained mechanism for determining which log records to output\n",
    "- Can be applied on loggers and handlers\n",
    "- Returns `True` if the record is to be processed\n",
    "- Can modify the log record "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Adapted from https://docs.python.org/3/howto/logging-cookbook.html\n",
    "from random import choice\n",
    "\n",
    "\n",
    "class ContextInjectionFilter(logging.Filter):\n",
    "    IPS = ['123.231.231.123', '127.0.0.1', '192.168.0.1']\n",
    "\n",
    "    def filter(self, record):\n",
    "        record.ip = choice(ContextFilter.IPS)\n",
    "        return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Formatter\n",
    "\n",
    "- Specify how content of log records are transformed for output\n",
    "- Format should at least include current time, level, name, and message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### JSON Formatter\n",
    "\n",
    "- Output each record as JSON object \n",
    "    - Includes each format string argument as key/value\n",
    "- JSON is a widely supported format\n",
    "- Conveniently machine parsable\n",
    "- Stop writing custom log parsers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'logging' from '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/logging/__init__.py'>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "importlib.reload(logging)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: python-json-logger in /usr/local/lib/python3.6/site-packages\r\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from python-json-logger)\r\n"
     ]
    }
   ],
   "source": [
    "! pip3 install python-json-logger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "LOGGING = {\n",
    "    'version': 1,\n",
    "    'loggers': {\n",
    "        'branch.leaf': {\n",
    "            'level': 'DEBUG',\n",
    "            'handlers': ['request_console', 'json_console'],\n",
    "            'propagate': False,  # do not propagate up hierarchy to 'branch'\n",
    "        },\n",
    "        'branch': {\n",
    "            'level': 'WARNING',  # set the minimum threshold\n",
    "            'handlers': ['request_console', 'json_console'],\n",
    "            'propagate': False,\n",
    "        },\n",
    "    },\n",
    "    'root': {\n",
    "        'handlers': ['request_console', 'json_console'],\n",
    "        'level': 'ERROR',\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Lets put it all together using a dictionary configuration\n",
    "\n",
    "Configuration can determine what is output\n",
    "\n",
    "Each log operation will be handled twice, once per handler\n",
    "\n",
    "Not every logger needs to be configured, goes up hierarchy to root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "LOGGING = {\n",
    "    **LOGGING,\n",
    "    'handlers': {\n",
    "        'request_console': {\n",
    "            'class': 'logging.StreamHandler',\n",
    "            'formatter': 'request',\n",
    "            'stream': 'ext://sys.stdout',\n",
    "        },\n",
    "        'json_console': {\n",
    "            'class': 'logging.StreamHandler',\n",
    "            'formatter': 'json',\n",
    "            'stream': 'ext://sys.stdout',\n",
    "        },\n",
    "    },\n",
    "    'formatters': {\n",
    "        'request': {\n",
    "            # request_id must always be provided to logger in `extra`\n",
    "            'format': '%(levelname)s - %(request_id)s - %(message)s',\n",
    "        },\n",
    "        'json': {  # https://pypi.python.org/pypi/python-json-logger\n",
    "            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter',\n",
    "            'format': '%(levelname)s %(message)s',\n",
    "        },\n",
    "    },\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "If `request_id` not provided, a `KeyError` will be raised by the format string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO - 90fca134-4468-41dc-bb26-4d087d61857b - User deleted\n",
      "{\"levelname\": \"INFO\", \"message\": \"User deleted\", \"request_id\": \"90fca134-4468-41dc-bb26-4d087d61857b\", \"duration\": 42}\n"
     ]
    }
   ],
   "source": [
    "import logging.config\n",
    "logging.config.dictConfig(LOGGING)  # configure logging settings\n",
    "\n",
    "leaf_logger = logging.getLogger('branch.leaf')      # filter level >= DEBUG\n",
    "flower_logger = logging.getLogger('branch.flower')  # filter level >= WARNING\n",
    "\n",
    "data = {\n",
    "    'request_id': '90fca134-4468-41dc-bb26-4d087d61857b',\n",
    "    'duration': 42,\n",
    "}\n",
    "\n",
    "leaf_logger.info('User deleted', extra=data)\n",
    "flower_logger.info('User deleted', extra=data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Make use of `extra` parameter\n",
    "- plain: `request_id` output\n",
    "- plain: `duration` not in output, not in format string\n",
    "- json: `extra` is merged with keys in format string\n",
    "\n",
    "Add as much context as you need\n",
    "\n",
    "It doesn’t really take any longer to log custom properties as you write your logging\n",
    "\n",
    "The extra properties can provide more details that make it easier to troubleshoot application problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Structured Logging\n",
    "\n",
    "- Record key/value pairs\n",
    "- Output is readable by humans and computers\n",
    "- Build context around operations\n",
    "- Eliminate any guesswork, let tools format and adapt to the data\n",
    "- Not specific to JSON, logfmt alternative:\n",
    "```\n",
    "level=info msg=\"Stopping all fetchers\"\n",
    "    tag=stopping_fetchers id=ConsumerFetcherManager-1382721708341\n",
    "    module=kafka.consumer.ConsumerFetcherManager\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "A general problem with log files is they are unstructured text data, making it unnecessarily hard to query them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Use Cases\n",
    "\n",
    "- Debugging and searching across every component of the application\n",
    "    - Record unique identifiers like request or user id\n",
    "- Monitoring the infrastructure’s health and security\n",
    "    - Looking at frequency and volume, trying to detect anomalies\n",
    "- Generate meaningful reports and fuel data products\n",
    "    - E.g. server access logs to do some summarization and aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "A good centralized log management system can enable searching by custom fields\n",
    "\n",
    "Quickly narrow down problems for a specific client\n",
    "\n",
    "Carry these ids through multiple touch points to track transactions through the system and follow them across machines, networks, and services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Infrastructure\n",
    "\n",
    "### Open Source\n",
    "- **E**lasticsearch: storage and search engine\n",
    "- **L**ogstash: filtering and forwarding\n",
    "- **K**ibana: analysis and visualization\n",
    "\n",
    "### Commercial\n",
    "- Sumo Logic\n",
    "- Splunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "TODO: diagram\n",
    "\n",
    "- Forwarder\n",
    "- Collector/Filter\n",
    "- Storage"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div class=\"cell border-box-sizing text_cell rendered\">\n",
    "  <div class=\"prompt input_prompt\"></div>\n",
    "  <div class=\"inner_cell\">\n",
    "    <div class=\"text_cell_render border-box-sizing rendered_html\">\n",
    "      <h2 id=\"Kibana-Demo\"><a href=\"https://demo.elastic.co/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(apache2.access.url,apache2.access.response_code,apache2.access.user_agent.name,apache2.access.user_agent.os),description:'',filters:!(),index:'filebeat-*',interval:auto,options:(darkTheme:!f),panels:!((col:1,id:f89a1660-6836-11e7-bd1c-eb5e5ad48f8b,panelIndex:1,row:6,size_x:12,size_y:3,type:visualization),(col:6,id:'2a762d00-6836-11e7-bd1c-eb5e5ad48f8b',panelIndex:2,row:1,size_x:7,size_y:2,type:visualization),(col:1,id:'472fdf70-6833-11e7-bd1c-eb5e5ad48f8b',panelIndex:3,row:9,size_x:6,size_y:5,type:visualization),(col:4,id:'48060c70-6839-11e7-bd1c-eb5e5ad48f8b',panelIndex:4,row:1,size_x:2,size_y:2,type:visualization),(col:1,id:'5322de50-683c-11e7-bd1c-eb5e5ad48f8b',panelIndex:6,row:1,size_x:3,size_y:2,type:visualization),(col:10,id:'0a1fcf00-683d-11e7-bd1c-eb5e5ad48f8b',panelIndex:7,row:3,size_x:3,size_y:3,type:visualization),(col:7,id:'3261d270-6855-11e7-bd1c-eb5e5ad48f8b',panelIndex:8,row:9,size_x:6,size_y:5,type:visualization),(col:1,id:d53d5a40-69be-11e7-bd1c-eb5e5ad48f8b,panelIndex:9,row:3,size_x:9,size_y:3,type:visualization)),query:(match_all:()),sort:!(_score,desc),timeRestore:!t,title:'Unique+Visitor+Dashboard',uiState:(P-1:(vis:(defaultColors:('0+-+150':'rgb(247,251,255)','150+-+300':'rgb(107,174,214)','300+-+500':'rgb(8,48,107)'))),P-3:(mapCenter:!(17.97873309555617,-2.109375),mapZoom:2),P-4:(vis:(defaultColors:('0+-+100':'rgb(0,104,55)'))),P-7:(vis:(params:(sort:(columnIndex:!n,direction:!n)))),P-9:(vis:(defaultColors:('0+-+3':'rgb(247,251,255)','12+-+14':'rgb(148,196,223)','14+-+17':'rgb(107,174,214)','17+-+20':'rgb(74,152,201)','20+-+23':'rgb(46,126,188)','23+-+26':'rgb(23,100,171)','26+-+28':'rgb(8,74,145)','3+-+6':'rgb(227,238,249)','6+-+9':'rgb(208,225,242)','9+-+12':'rgb(182,212,233)')))),viewMode:view)\" data-preview-link>https://demo.elastic.co/app/kibana#/discover</a><a class=\"anchor-link\" href=\"#Kibana-Demo\">&#182;</a></h2>\n",
    "    </div>\n",
    "  </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Add columns with \"Table icon\"\n",
    "- `apache2.access.url`\n",
    "- `apache2.access.response_code`\n",
    "- `apache2.access.user_agent.name`\n",
    "- `apache2.access.user_agent.os`\n",
    "\n",
    "Find matching results with \"Magnifying icon\": \"Windows 7\"\n",
    "\n",
    "Visualize results with \"Selected Fields\" sidebar: 3rd row\n",
    "\n",
    "Filter by a remote IP address"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Tips\n",
    "\n",
    "- Be concise and descriptive\n",
    "- Include a human readable message as key/value pair\n",
    "- Include extra context data (especially for exceptions)\n",
    "- Do not log passwords or any personally identifiable information\n",
    "- For logging in a package, the application should dictate the logging configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "A logging statement should contain both data and description\n",
    "\n",
    "Trying to figure out why an exception happened is infinitely easier if you know more details about who the user was and the input parameters\n",
    "\n",
    "It is important to have complete logs and to trust them"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}