Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High CPU usage after upgrade to 0.26.1 #6209

Open
rbhalla opened this issue Oct 19, 2017 · 11 comments

Comments

@rbhalla
Copy link

commented Oct 19, 2017

Running on ECS (using docker image), connecting to Redshift.

CPU spikes at 45 past the hour, and reduces at ~20 past the hour. Restarting the task doesn't seem to do much, for now I've increased CPU allocation, but ideally would like to resolve any problem.

Not sure if this is related to the deadlock issue raised in another issue. If it is, feel free to close.

Let me know if you need anymore specific information.

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

@firiz

This comment has been minimized.

Copy link

commented Oct 20, 2017

I'am running jar version on Ubuntu 16 and this is happening to me as well.

@Eilliar

This comment has been minimized.

Copy link

commented Oct 23, 2017

Yeah, it happens to me too. The process consumes 100% of CPU and metabase completely freezes. One can't access the interface, it keeps loading forever. So I have to restart the process.

Some infos:
Running metabase on a t2.medium instance in AWS, Ubuntu 16.04 with openjdk version "1.8.0_131". Also starting the jar with -Xmx1024M.

When this happens, the log does not give any clue on the cause, it just show some errors from Slack integration (but this is turned off in the admin panel) and stops logging until I restart the process.

@rbhalla

This comment has been minimized.

Copy link
Author

commented Oct 26, 2017

screen shot 2017-10-26 at 17 03 07

Upgraded to 0.26.2. Seemed better at the beginning but now in alarm state for longer periods of time. Seems like a buildup of something being processed? Complete guess but I'll try and dig in a bit more.

@rbhalla

This comment has been minimized.

Copy link
Author

commented Oct 30, 2017

For anyone stumbling upon this issue, disabling xray "solved" this problem.

@rbhalla

This comment has been minimized.

Copy link
Author

commented Nov 14, 2017

Resurrecting this old thread again, apologies.

After monitoring it for a couple of weeks now past that, disabling xrays did reduce how long the spikes persisted, but didn't completely eliminate hourly spikes. Again this only seems to be something happening on the latest version (now 0.26.2)

screen shot 2017-11-14 at 09 47 30

@camsaul

This comment has been minimized.

Copy link
Member

commented Nov 15, 2017

@sbelak is this the X-ray sync stuff that's causing these performance spikes?

@sbelak

This comment has been minimized.

Copy link
Contributor

commented Nov 15, 2017

No. X-rays are purely on demand/frontend triggered. If it's as nicely periodic as this, it's something else.

@rbhalla

This comment has been minimized.

Copy link
Author

commented Nov 15, 2017

I've been digging more into metabase's settings and it seems like turning off hourly scanning for filter values drastically reduces the peaks hourly. I'm guessing the remaining smaller peaks are related to the database syncing that happens.

screen shot 2017-11-15 at 07 35 26

I did initially tie some of the improvement to xrays, so I'm going to switch those back on and confirm that wasn't actually affecting anything.

@nkconnor

This comment has been minimized.

Copy link

commented Nov 15, 2017

Yes, turning off the scanning has fixed this issue for us as well. I hope this is addressed in a patch as the scans could easily take down or cripple a production database after upgrade

@camsaul camsaul added Metadata/Sync and removed X-Rays/ labels Nov 15, 2017

@hexleyOlive

This comment has been minimized.

Copy link

commented Jan 8, 2018

Same issues, after upgrade Metabase from 0.25 to the latest 0.27.2, DATABASE SYNCING
and SCANNING FOR FILTER VALUES have created performance impact and high activities on SQL Server.

DATABASE SYNCING take around 1hr and around 30% of SQL CPU (I've setup one time Daily).
SCANNING FOR FILTER VALUES take 2hrs and around 50% of SQL CPU (I've setup one time Weekly).

DATABASE SYNCING should be "lightweight process", but is very (too) high to only checks updates to database schema.

@abronte

This comment has been minimized.

Copy link

commented Jul 23, 2018

I see the same issue on 0.29.3.

Other details:

  • syncing configured for 6am daily
  • scanning for filter values set to manual
  • x-rays disabled
  • metabase CPU pegged at 100% seems to happen randomly
  • Presto/hive and postgres backends configured. Don't see CPU issues with them
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.