dictionary changed size during iteration

[starenka]

I get this error pretty often in TransactionsMetricSet:

RuntimeError: dictionary changed size during iteration
  File "elasticapm/utils/threading.py", line 79, in run
    rval = self._function(*self._args, **self._kwargs)
  File "elasticapm/metrics/base_metrics.py", line 97, in collect
    for data in metricset.collect():
  File "elasticapm/metrics/base_metrics.py", line 215, in collect
    for (name, labels), t in compat.iteritems(self._timers):

Environment (please complete the following information)

• OS: Debian stable
• Python version: 2.7.13
• Framework and version [e.g. Django 2.1]: 1.11.20
• APM Server version: 7.6.0
• Agent version: 5.3.3

[beniwohli]

@starenka that's interesting. It probably happens because while we iterate over the metrics in the background thread, the foreground thread (the one that is handling Django requests) is adding new metrics data.

There are two ways to fix this: we could either take the lock before iterating, or turns the items iterator into a list before iterating over it.

Taking a lock would probably the cleanest way, but it'll block the main thread, even if it is for a very short amount of time. OTOH, turning the iterator into a list would consume a little bit more memory, but wouldn't block the main thread.

@basepi WDYT?

[basepi]

I guess I could go either way. I'm loathe to block the main thread -- I feel like I'd rather (briefly) lose a little memory during the operation. But I don't think either option is particularly bad.