Services have degraded performance sometimes

[rmeissner]

Problem Owner

@Uxio0

What is the Problem

Currently our services experience degraded performance related because of different reasons. #2 is specific to the DB related issues but there are also other reasons that need to be identified. Some examples are

• Overloaded ethereum nodes
• Unresponsive pods

This causes many issues (similar to the ones listed in #2):

• Slow response times or even server outages
• Increased maintenance costs

Why is it relevant?

Slow response times or even server outages

To empower our interfaces it is important to provide a reliable and fast information source.

Increased maintenance costs

The high maintenance overhead and costs make it harder to scale the services to new networks.

How can we track the problem? (What is the KPI?)

• Response time / Error rate / Uptime

Solutions

• Separate the celery tasks into different queues that should have different workers safe-transaction-service#620
• Fix memory usage on Celery worker safe-transaction-service#657
• Optimize database queries safe-transaction-service#508

Related Issues

• Service team is missing access to infrastructure resources #5
• Services use a lot of Database resource #2
• Initial request response time is very high #7
• Service team needs to fix most issues manually #6
• Kibana is very slow for analyzing larger timeframes #10
• There is an overflow of service related logs #11