Truncate large dictionaries is local vars #632
Description
Summary
If you have large dicts in local vars, the messages to be sent to the server can be too large for the server to accept.
There should be an option to truncate dicts to a maximum length.
Is your feature request related to a problem? Please describe.
When capturing exceptions using client.capture_exception() the stack trace, including local variables, is sent to the server. The size of the message sent to the server depends on the size of the local variables. When the message is larger than a configurable limit, the server rejects it with a 400 error. This can lead to errors being silent for weeks (we learned this the hard way).
When a local variable is a list or a string, it is truncated
apm-agent-python/elasticapm/utils/encoding.py
Lines 185 to 189 in 6113dc2
This prevents large lists from being sent and reduces the message size.
When a local var is a dict, it is not truncated. There is a TODO comment about it in the shorten function above. But that function never gets a dict as input. Instead, dicts are recursively scanned before even calling the shorten function.
The recursive scanning seems to happen in the varmap function here
apm-agent-python/elasticapm/utils/__init__.py
Lines 63 to 64 in 6113dc2
This leads to huge messages being sent to server when there's an exception in a function with huge dicts.
Describe the solution you'd like
Ideally, for our use case at least, dicts would be shortened as it happens with lists or strings.
We have achieved this by monkey patching the var map function like this
import elasticapm.utils
# Monkey patch APM to only report the 20 first items of dicts
def first_dict_elements(var, max_length):
stop = max_length if len(var) <= max_length else max_length - 1
ret = {}
length = 0
for k, v in elasticapm.utils.compat.iteritems(var):
ret[k] = v
length += 1
if length >= stop:
if len(var) > length:
ret['...'] = "(%d more elements)" % (len(var) - length,)
break
return ret
def varmap(func, var, context=None, name=None):
"""
Executes ``func(key_name, value)`` on all values,
recursively discovering dict and list scoped
values.
"""
if context is None:
context = set()
objid = id(var)
if objid in context:
return func(name, "<...>")
context.add(objid)
if isinstance(var, dict):
cropped = first_dict_elements(var, 20)
ret = dict((k, varmap(func, v, context, k)) for k, v in elasticapm.utils.compat.iteritems(cropped))
elif isinstance(var, (list, tuple)):
ret = func(name, [varmap(func, f, context, name) for f in var])
else:
ret = func(name, var)
context.remove(objid)
return ret
elasticapm.utils.varmap = varmapThis truncates all dicts to 20 elements and adds a last element with the number of skipped elements.
Note that there is some ugly length counting needed because the varmap function is called twice for each dict so we sometimes process dicts that have already been truncated and getting the number of skipped elements becomes tricky.
Would a solution like this be implementable by the library?
Describe alternatives you've considered
There might be config options available to achieve this that we did not see. Or maybe is possible to provide some pre- or post-processor of the stack trace to achieve the same. We did not investigate that route.