Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for Logstash v1 event schema (and improved timestamp handling) #9539

Merged
merged 5 commits into from Jan 3, 2014
Merged

Support for Logstash v1 event schema (and improved timestamp handling) #9539

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6b18f9f
No need for fancy timezone handling, just send a UTC timestamp. Also,
jcollie Jan 2, 2014
c63ced6
Add 'salt' to the list of tags.
jcollie Jan 2, 2014
3535dbc
add a formatter for v1 logstash events
jcollie Jan 2, 2014
3a88b54
update fields for v1
jcollie Jan 2, 2014
d87a4be
update logstash docs
jcollie Jan 2, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
132 changes: 108 additions & 24 deletions salt/log/handlers/logstash_mod.py
View file
Open in desktop
Expand Up @@ -11,17 +11,18 @@
UDP Logging Handler
-------------------

In order to setup the datagram handler for `Logstash`_, please define on
the salt configuration file:
For versions of `Logstash`_ before 1.2.0:

In the salt configuration file:

.. code-block:: yaml

logstash_udp_handler:
host: 127.0.0.1
port: 9999
version: 0


On the `Logstash`_ configuration file you need something like:
In the `Logstash`_ configuration file:

.. code-block:: text

Expand All @@ -32,23 +33,45 @@
}
}

For version 1.2.0 of `Logstash`_ and newer:

In the salt configuration file:

.. code-block:: yaml

logstash_udp_handler:
host: 127.0.0.1
port: 9999
version: 1

In the `Logstash`_ configuration file:

.. code-block:: text

input {
udp {
port => 9999
codec => json
}
}

Please read the `UDP input`_ configuration page for additional information.


ZeroMQ Logging Handler
----------------------

In order to setup the ZMQ handler for `Logstash`_, please define on the
salt configuration file:
For versions of `Logstash`_ before 1.2.0:

In the salt configuration file:

.. code-block:: yaml

logstash_zmq_handler:
address: tcp://127.0.0.1:2021
version: 0


On the `Logstash`_ configuration file you need something like:
In the `Logstash`_ configuration file:

.. code-block:: text

Expand All @@ -63,6 +86,27 @@
}
}

For version 1.2.0 of `Logstash`_ and newer:

In the salt configuration file:

.. code-block:: yaml

logstash_zmq_handler:
address: tcp://127.0.0.1:2021
version: 1

In the `Logstash`_ configuration file:

.. code-block:: text

input {
zeromq {
topology => "pubsub"
address => "tcp://0.0.0.0:2021"
codec => json
}
}

Please read the `ZeroMQ input`_ configuration page for additional
information.
Expand Down Expand Up @@ -123,13 +167,6 @@
from salt.log.setup import LOG_LEVELS
from salt.log.mixins import NewStyleClassMixIn

# Import 3rd-party libs
try:
from pytz import utc as _UTC
HAS_PYTZ = True
except ImportError:
HAS_PYTZ = False

log = logging.getLogger(__name__)

# Define the module's virtual name
Expand All @@ -151,18 +188,20 @@ def __virtual__():

def setup_handlers():
host = port = address = None
logstash_formatter = LogstashFormatter()

if 'logstash_udp_handler' in __opts__:
host = __opts__['logstash_udp_handler'].get('host', None)
port = __opts__['logstash_udp_handler'].get('port', None)
version = __opts__['logstash_udp_handler'].get('version', 0)

if host is None and port is None:
log.debug(
'The required \'logstash_udp_handler\' configuration keys, '
'\'host\' and/or \'port\', are not properly configured. Not '
'configuring the logstash UDP logging handler.'
)
else:
logstash_formatter = LogstashFormatter(version = version)
udp_handler = DatagramLogstashHandler(host, port)
udp_handler.setFormatter(logstash_formatter)
udp_handler.setLevel(
Expand All @@ -184,6 +223,7 @@ def setup_handlers():
if 'logstash_zmq_handler' in __opts__:
address = __opts__['logstash_zmq_handler'].get('address', None)
zmq_hwm = __opts__['logstash_zmq_handler'].get('hwm', 1000)
version = __opts__['logstash_zmq_handler'].get('version', 0)

if address is None:
log.debug(
Expand All @@ -192,6 +232,7 @@ def setup_handlers():
'configuring the logstash ZMQ logging handler.'
)
else:
logstash_formatter = LogstashFormatter(version = version)
zmq_handler = ZMQLogstashHander(address, zmq_hwm=zmq_hwm)
zmq_handler.setFormatter(logstash_formatter)
zmq_handler.setLevel(
Expand All @@ -215,20 +256,20 @@ def setup_handlers():


class LogstashFormatter(logging.Formatter, NewStyleClassMixIn):
def __init__(self, msg_type='logstash', msg_path='logstash'):
def __init__(self, msg_type='logstash', msg_path='logstash', version=0):
self.msg_path = msg_path
self.msg_type = msg_type
self.version = version
self.format = getattr(self, 'format_v{0}'.format(version))
super(LogstashFormatter, self).__init__(fmt=None, datefmt=None)

def formatTime(self, record, datefmt=None):
timestamp = datetime.datetime.utcfromtimestamp(record.created)
if HAS_PYTZ:
return _UTC.localize(timestamp).isoformat()
return '{0}+00:00'.format(timestamp.isoformat())
return datetime.datetime.utcfromtimestamp(record.created).isoformat()[:-3] + 'Z'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you really sure? See #9140

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, since the timestamp is already in UTC, all that pytz is doing for you is a fancy way to add "+00:00" to the end of the string:

>>> t = time.time()
>>> d=datetime.datetime.utcfromtimestamp(t)
>>> d
datetime.datetime(2014, 1, 3, 2, 25, 16, 197281)
>>> l = pytz.utc.localize(d)
>>> l
datetime.datetime(2014, 1, 3, 2, 25, 16, 197281, tzinfo=<UTC>)
>>> l.isoformat()
'2014-01-03T02:25:16.197281+00:00'
>>> d.isoformat()
'2014-01-03T02:25:16.197281'
>>> datetime.datetime.utcfromtimestamp(t).isoformat()[:-3]+'Z'
'2014-01-03T02:25:16.197Z'

So, I think that it's just simpler to skip pytz. Also, as mentioned in the comments, 1.3.2 has a bug where it won't accept timestamps with more than millisecond precision.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be comfortable with something like the existing fallback which appends +00:00?

It seemed that it would need less configuration on the logstash side...

return '{0}+00:00'-format(datetime.datetime.utcfromtimestamp(record.created).isoformat()[:-3])

We'd still drop pytz.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Z and +00:00 are equivalent, but other than saving a few bytes on the wire it doesn't matter to me, it won't make a difference on the logstash side.

http://en.wikipedia.org/wiki/ISO_8601#UTC

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Darn!

Yes, correct, the Z is the same 😄


def format(self, record):
def format_v0(self, record):
host = socket.getfqdn()
message_dict = {
'@timestamp': self.formatTime(record),
'@fields': {
'levelname': record.levelname,
'logger': record.name,
Expand All @@ -247,8 +288,7 @@ def format(self, record):
),
'@source_host': host,
'@source_path': self.msg_path,
'@tags': [],
'@timestamp': self.formatTime(record),
'@tags': ['salt'],
'@type': self.msg_type,
}

Expand Down Expand Up @@ -278,6 +318,50 @@ def format(self, record):
message_dict['@fields'][key] = repr(value)
return json.dumps(message_dict)

def format_v1(self, record):
message_dict = {
'@version': 1,
'@timestamp': self.formatTime(record),
'host': socket.getfqdn(),
'levelname': record.levelname,
'logger': record.name,
'lineno': record.lineno,
'pathname': record.pathname,
'process': record.process,
'threadName': record.threadName,
'funcName': record.funcName,
'processName': record.processName,
'message': record.getMessage(),
'tags': ['salt'],
'type': self.msg_type
}

if record.exc_info:
message_dict['exc_info'] = self.formatException(
record.exc_info
)

# Add any extra attributes to the message field
for key, value in record.__dict__.items():
if key in ('args', 'asctime', 'created', 'exc_info', 'exc_text',
'filename', 'funcName', 'id', 'levelname', 'levelno',
'lineno', 'module', 'msecs', 'msecs', 'message', 'msg',
'name', 'pathname', 'process', 'processName',
'relativeCreated', 'thread', 'threadName'):
# These are already handled above or not handled at all
continue

if value is None:
message_dict[key] = value
continue

if isinstance(value, (string_types, bool, dict, float, int, list)):
message_dict[key] = value
continue

message_dict[key] = repr(value)
return json.dumps(message_dict)


class DatagramLogstashHandler(logging.handlers.DatagramHandler):
'''
Expand Down