Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ELB & updating Metabase => deadlock #6264

Closed
kelset opened this issue Oct 25, 2017 · 1 comment
Closed

ELB & updating Metabase => deadlock #6264

kelset opened this issue Oct 25, 2017 · 1 comment
Labels
Operation/Elastic Beanstalk .Performance Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness Type:Bug Product defects

Comments

@kelset
Copy link

kelset commented Oct 25, 2017

Hey all,
I'll start right away with the details to contextualize:

Your browser and the version: Chromium Version 61.0.3158.0 (Developer Build) (64-bit) (but also happens with other browsers)
Your operating system: MacOS X 10.12.6
Your databases: mongoDB hosted on Mongo Atlas
Metabase version: 0.25.2
Metabase hosting environment: EC2 Server on AWS (via ELB) (machine: t2.small) (os: 64bit Amazon Linux 2017.03 v2.6.0 running Docker 1.12.6)
Metabase internal database: postgres

Basically, with the configuration above, we were mostly "fine", from time to time Metabase would slow down but overall it was working. We decided to try upgrading both Metabase and the os, from:

Metabase 0.25.2 -> 0.26.2
os: 64bit Amazon Linux 2017.03 v2.6.0 running Docker 1.12.6 -> 64bit Amazon Linux 2017.03 v2.7.4 running Docker 17.03.2-ce

But once those two were done, our instance basically deadlocked in less than an hour. Only way to put if back up online was to restart the instance; since it kept happening we had to rollback.
We first rolled back the os version to 64bit Amazon Linux 2017.03 v2.6.0 running Docker 1.12.6, which "slowed down" the degradation of performances (= no deadlocks in 1h) but the ELB instance keeps going to health state "warning".

It looks like it's performance degradation is a recurring issue ( #6209 #6071 #5607 #5106), but none of these other issues have any hints on how to fix aside from rolling back - I've opened this new issue to give more details about my specific situation & because it's kinda hard for me to tell if the causes are the same (and hopefully @attekei @camsaul @salsakran can help me out to pinpoint the issue/ suggest a fix).

@salsakran salsakran added Type:Bug Product defects .Performance Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness Operation/Elastic Beanstalk labels Dec 4, 2017
@salsakran
Copy link
Contributor

@kelset we haven't gotten any other reports of a deadlock happening with beanstalk, and have made big improvements in how thread are handed and memory consumption since this was opened.

I'm closing this issue unless you (or someone else) has seen this recently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Operation/Elastic Beanstalk .Performance Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness Type:Bug Product defects
Projects
None yet
Development

No branches or pull requests

3 participants