Elastic apm runtime error

[pusnik]

We've seen this issue when running elastic APM on the following stack:

Python 2.7.11
Django 1.8.19
Elastic apm 3.0.0

Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/django/http/response.py", line 309, in close signals.request_finished.send(sender=self._handler_class) File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 189, in send response = receiver(signal=self, sender=sender, **named) File "/usr/local/lib/python2.7/site-packages/elasticapm/contrib/django/apps.py", line 50, in <lambda> lambda sender, **kwargs: client.end_transaction() if _should_start_transaction(client) else None, File "/usr/local/lib/python2.7/site-packages/elasticapm/base.py", line 260, in end_transaction self._collect_transactions() File "/usr/local/lib/python2.7/site-packages/elasticapm/base.py", line 298, in _collect_transactions self._stop_send_timer() File "/usr/local/lib/python2.7/site-packages/elasticapm/base.py", line 323, in _stop_send_timer self._send_timer.join() File "/usr/local/lib/python2.7/threading.py", line 929, in join raise RuntimeError("cannot join thread before it is started") RuntimeError: cannot join thread before it is started
This happens in a production environment or on the test server where we simulate higher rps on the server. On the staging environment with lower rps rate apm runs normally.
It doesn't happen immediately. We get for about 1-2 minutes of records before if crashes.

Do you recommend changing any setting to have it running more stable?

[beniwohli]

Hi @pusnik. Thanks for opening the issue!

We've seen this issue before, but only once or twice in our test environment. What's really puzzling is that we're checking if the thread is alive before calling .join():

apm-agent-python/elasticapm/base.py

Lines 320 to 323 in e8b134d

	def _stop_send_timer(self):
	if self._send_timer and self._send_timer.is_alive() and not self._send_timer == threading.current_thread():
	self._send_timer.cancel()
	self._send_timer.join()

Only thing I can imagine is that calling .cancel() sometimes makes the thread die immediately, so that the .join() right after fails. As cancel() only sets an Event, we can probably move it out of the if clause, that would fix it if this is indeed the root cause. Basically:

def _stop_send_timer(self):
    if self._send_timer and not self._send_timer == threading.current_thread():
        self._send_timer.cancel()
        if self._send_timer.is_alive():
            self._send_timer.join()

I pushed this change to a branch. Would you mind giving this a test run in your test environment under high load to see if it fixes it? You can install it like this:

pip install git+https://github.com/beniwohli/apm-agent-python.git@possible-fix-276

[pusnik]

Hi @beniwohli

Thank you for the quick response. I have deployed it to the test environment. Tests are pending.
Will let you know as soon as I know something.

Best,

Grega

[pusnik]

@beniwohli it seems that the RuntimeError is gone, at least I can't see it anymore in logs. Will monitor them.
Nevertheless the metrics are still not visible on our Kibana at high load, but this is due to our server not having enough resources. We'll be scaling on Monday.
You can close the ticket and I'll reopen if needed.

Thank's again.

[beniwohli]

Thanks for giving it a test run @pusnik! I will release this with a couple other bug fixes later today.