Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Switch to config-file based logging #56

Merged
merged 5 commits into from

3 participants

@bclermont

Diamond currently assume that logging is always /var/log/diamond/diamond.log

What if a user want to:

  • run as WARN level? only options are DEBUG and INFO
  • have a custom collectors plugin and want to run only this collector as DEBUG and kept the rest as ERROR on a production server deployed from a debian/redhat package. (well you can always change code, if it's allowed)
  • run diamond on a flash disk based appliance, with limited or no writeable space
  • send all output to a centralized logging system, such as graylog2
  • handle logging to syslogd

moving python logging configuration to the config file let you do this and offload logging configuration to logging python , instead of dealing with command line parameters.

the only thing I don't like is that logging.config expect to have [handlers] section, which is also used by diamond.
it's only for keys=
I tried to have a 2nd section [handlers] with only keys= but confibobj don't accept that, while ConfigParser do.

Sample of configuration to have log sent to both /var/log/diamond/diamond.log AND local syslog daemon:

[handlers]
keys = rotated_file,syslog

[loggers]
keys = root

[formatters]
keys = syslog,default

[logger_root]
level = INFO
handlers = syslog,rotated_file

[handler_syslog]
class = handlers.SysLogHandler
level = DEBUG
formatter = syslog
args = ('/dev/log',)

[formatter_syslog]
format = %(asctime)-15s diamond[%(process)d] %(message)s
datefmt = %b %d %H:%M:%S

[handler_rotated_file]
class = handlers.TimedRotatingFileHandler
level = DEBUG
formatter = default
# rotate at midnight, each day and keep 7 days
args = ('/var/log/diamond/diamond.log', 'midnight', 1, 7)

[formatter_default]
format = [%(asctime)s] [%(threadName)s] %(message)s
datefmt =
@bclermont

ArchiveHandler still initialize it's logging handler itself.

But that too can be moved to the config file.

bin/diamond
((46 lines not shown))
+ logging.config.fileConfig(options.configfile,
+ disable_existing_loggers=False)
+ else:
+ # python <= 2.5 does not have disable_existing_loggers
+ # default was to always disable them, in our case we want to
+ # keep any logger created by handlers
+ disabled = {}
+ for key, value in logging.root.manager.loggerDict.items():
+ try:
+ disabled[key] = value.disabled
+ except AttributeError:
+ # This is OK, we're just not dealing with a Logger
+ pass
+ logging.config.fileConfig(options.configfile)
+ for key, disabled in disabled.items():
+ logging.root.manager.loggerDict[key] = True

this should maybe be logging.root.manager.loggerDict[key].disabled = False?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
bin/diamond
((45 lines not shown))
+ if sys.version_info >= (2,6):
+ logging.config.fileConfig(options.configfile,
+ disable_existing_loggers=False)
+ else:
+ # python <= 2.5 does not have disable_existing_loggers
+ # default was to always disable them, in our case we want to
+ # keep any logger created by handlers
+ disabled = {}
+ for key, value in logging.root.manager.loggerDict.items():
+ try:
+ disabled[key] = value.disabled
+ except AttributeError:
+ # This is OK, we're just not dealing with a Logger
+ pass
+ logging.config.fileConfig(options.configfile)
+ for key, disabled in disabled.items():

this disabled loop variable overrides the disabled dict on L:89

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@kormoc kormoc merged commit 2b3dfdf into from
@kormoc
Collaborator

Looks great! Thanks guys!

@CBarraford CBarraford referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
@bclermont bclermont referenced this pull request in saltstack/salt
Closed

Syslog Logging Timeformat is non-standard #3970

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 25, 2012
  1. switch to logging.config to handle log file and verbosity

    Bruno Clermont authored
  2. fix default configuration

    Bruno Clermont authored
  3. set logger to debug when running in foreground

    Bruno Clermont authored
Commits on Apr 26, 2012
  1. @Lacrymology
  2. Merge pull request #1 from Lacrymology/master

    Bruno Clermont authored
    bugfix on your modifications
This page is out of date. Refresh to see the latest.
Showing with 62 additions and 36 deletions.
  1. +22 −30 bin/diamond
  2. +40 −6 conf/diamond.conf.example
View
52 bin/diamond
@@ -19,16 +19,14 @@ from diamond.server import Server
import optparse
import signal
-import logging.handlers
+import logging.config
def main():
# Initialize Options
parser = optparse.OptionParser()
parser.add_option("-c", "--configfile", dest="configfile", default="/etc/diamond/diamond.conf", help="config file")
- parser.add_option("-l", "--logfile", dest="logfile", default=None, help="log file")
parser.add_option("-f", "--foreground", dest="foreground", default=False, action="store_true", help="run in foreground")
parser.add_option("-p", "--pidfile", dest="pidfile", default=None, help="pid file")
- parser.add_option("-v", "--verbose", dest="verbose", default=False, action="store_true", help="verbose")
parser.add_option("-r", "--run", dest="collector", default=None, help="run a given collector once and exit")
# Parse Command Line Args
@@ -67,40 +65,34 @@ def main():
# Initialize Logging
log = logging.getLogger('diamond')
- log.setLevel(logging.INFO)
- # Configure Logging Format
- formatter = logging.Formatter('[%(asctime)s] [%(threadName)s] %(message)s')
-
- # Configure Log Stream Handler
+ # Running in foreground, configure Log Stream Handler with debug severity
if options.foreground:
+ log.setLevel(logging.DEBUG)
+ # Configure Logging Format
+ formatter = logging.Formatter('[%(asctime)s] [%(threadName)s] %(message)s')
+ # handler
streamHandler = logging.StreamHandler(sys.stdout)
streamHandler.setFormatter(formatter)
streamHandler.setLevel(logging.DEBUG)
log.addHandler(streamHandler)
-
- # Got Log File Handler log file
- if not options.logfile:
- options.logfile = str(config['server']['log_file'])
-
- # Attempt to Log File Directory
- logdir = os.path.abspath(os.path.dirname(options.logfile))
- if not os.path.exists(logdir):
- os.mkdir(logdir)
-
- # Configure Log File handler
- if config['server'].has_key('log_file_days'):
- fileHandler = logging.handlers.TimedRotatingFileHandler(options.logfile, 'midnight', 1, backupCount=int(config['server']['log_file_days']))
else:
- fileHandler = logging.FileHandler(options.logfile, 'w')
-
- fileHandler.setFormatter(formatter)
- fileHandler.setLevel(logging.DEBUG)
- log.addHandler(fileHandler)
-
- # Configure Logging Verbosity
- if options.verbose:
- log.setLevel(logging.DEBUG)
+ # if running in background, configure loggers from config file
+ try:
+ if sys.version_info >= (2,6):
+ logging.config.fileConfig(options.configfile,
+ disable_existing_loggers=False)
+ else:
+ # python <= 2.5 does not have disable_existing_loggers
+ # default was to always disable them, in our case we want to
+ # keep any logger created by handlers
+ logging.config.fileConfig(options.configfile)
+ for logger in logging.root.manager.loggerDict.values():
+ logger.disabled = 0
+ except Exception, e:
+ sys.stderr.write("Error occurs when initialize logging: ")
+ sys.stderr.write(str(e))
+ sys.stderr.write(os.linesep)
# Log
log.info('Changed UID: %d (%s) GID: %d (%s).' % (os.getuid(), config['server']['user'], os.getgid(), config['server']['group']))
View
46 conf/diamond.conf.example
@@ -17,12 +17,6 @@ user =
# Leave empty to use the current group
group =
-# Log file
-log_file = /var/log/diamond/diamond.log
-
-# Number of days to keep log files
-log_file_days = 7
-
# Pid file
pid_file = /var/run/diamond.pid
@@ -39,6 +33,9 @@ collectors_reload_interval = 3600
### Options for handlers
[handlers]
+# daemon logging handler(s)
+keys = rotated_file
+
### Defaults options for all Handlers
[[default]]
@@ -130,3 +127,40 @@ method = Sequential
# Default numeric output
byte_unit = byte
+
+################################################################################
+### Options for logging
+# for more information on file format syntax:
+# http://docs.python.org/library/logging.config.html#configuration-file-format
+
+[loggers]
+
+keys = root
+
+# handlers are higher in this config file, in:
+# [handlers]
+# keys = ...
+
+[formatters]
+
+keys = default
+
+[logger_root]
+
+# to increase verbosity, set DEBUG
+level = INFO
+handlers = rotated_file
+
+[handler_rotated_file]
+
+class = handlers.TimedRotatingFileHandler
+level = DEBUG
+formatter = default
+# rotate at midnight, each day and keep 7 days
+args = ('/var/log/diamond/diamond.log', 'midnight', 1, 7)
+
+[formatter_default]
+
+format = [%(asctime)s] [%(threadName)s] %(message)s
+datefmt =
+
Something went wrong with that request. Please try again.