New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory quota exceeded on Heroku #3360
Comments
@tlrobinson it sounds like we need to be passing a map heap size param to the JVM when running Metabase, e.g. Ideally we'd want to set that number based on the whether it's a 1x or 2x dyno, if there's some way that could be determined |
@camsaul That's what I thought as well based on Heroku docs (https://devcenter.heroku.com/articles/java-support#adjusting-environment-for-a-dyno-size). JAVA_TOOL_OPTIONS config variable are set to "-Xmx350m -Xss512k". (I'm currently back on the standard 1x with 512MB RAM) |
@lorenzosantos, I didn't realize Heroku let you set that. It should be helpful in debugging this 👍 . Can you try doing adding these options to -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC That will tell the GC to collect classes when they're no longer needed. Clojure tends to create lots of one-off classes which can cause memory issues on low-memory instances. If that doesn't work my next suggestion is to try lowering the max heap size a bit: -Xmx300m Metabase should still run fine with that amount of memory. It seems like for whatever reason Heroku might be telling the JVM it can use more memory than it's actually allowed to use. Let me know if either one works. |
Seeing this as well on 0.19.3 on hobby/hobby-basic, using the more restrictive JVM options:
|
@camsaul Tried adding those options but it didn't help. A few minutes after startup memory usage was already exceeding the quota. Going to try lowering the max heap size to -Xmx300m now. |
@camsaul Lowered the max heap size to -Xmx300m didn't fix the issue either. Do you have any other ideas? |
@lorenzosantos it seems like for whatever reason the @tlrobinson are we setting custom JVM options when running Metabase on Herkou? |
I looked at this a little more, and it does seem like the memory options are being respected. The metabase-deploy project added custom settings in bin/start, then removed them in favor of the jvm buildpack (see metabase/metabase-deploy#5). The heroku-buildpack-jvm-common sets memory per dyno size, or picks up custom settings, in
My next step might be to try deploying the main project directly, although I can't see why that would make a difference if both metabase and metabase-deploy stable branches are 0.19.3, and the JVM options are being set already. |
ping @tlrobinson |
Still happening with metabase-deploy currently on v0.20.2. Didn't look through the changelog for applicable commits, just FYI. |
Just discovered that if instead of In testing |
This can be fixed with metabase/metabase-deploy#5 |
Currently still experiencing this on 0.26 with Pro 1x. There doesn't seem to be a resolution clearly stated anywhere. Is this still an ongoing issue? |
This appears to be ongoing. We're currently on a standard 1x dyno, so presumably hitting this code: ...which, I think is forcing the limit to 248... and leaving me no option to lower it. If I set JAVA_OPTS to
Given how bin/start is currently written any ideas how I can lower the memory limits to see if these R14 errors go away? |
After struggling with a 1X instance, I've upgraded to a 2x instance but continued having the same problems. The errors (and subsequent crashing) tend to crop up sometime within a day of restarting the instance.... so, not completely broken, but also not truly usable. I tried reducing the available memory with these settings: I'm now trying Stay tuned. Any help appreciated. |
I'd recommend something like If you still get R14 errors with |
I can't recall, but I think I tried settings even lower than -Xmx500m. That all said, I've moved us off heroku as it just isn't cutting it - so I don't have a way to continue troubleshooting this thread. |
This comment has been minimized.
This comment has been minimized.
Has anyone had a solution for this? I've just run into it but it seems the JVM is self determining memory limits |
@te-chris the What dyno size are you using? You might also want to try these options as described in Dev Center:
Lots of JVM users have had luck with that. |
@jkutner Those two options are the way to go, but they are completely ignored: Inside the metabase container:
For some reason metabase is started with the flag I created a pull request for it since I tested it and everything looks good to me: #8459 |
I am getting OutofMemory error after upgrading to 0.31. It was working fine until I upgraded the metabase version. The app is deployed in Heroku, in free tier. I've been using this for quite some time now. Please let me know how to fix the same [preferably without upgrading to a paid version in Heroku] Here is the error from the log, hope it helps. |
I think an upgrade to Java 11 would help. I'm not sure if that's possible, but other apps have been reporting a much better memory profile. |
I tried upgrading to Java 11, that didn't work either. The issue started once I upgraded to the latest version. Is there a way I can downgrade to the previous version? |
We're working on a fix to reduce memory usage. Stay tuned. |
@leenasn You should be able to |
@tlrobinson I tried that, but no luck :(. Noticed one thing in the log that its still downloading 0.31.0 version of the Metabase. Pasting the log for your reference. Am I missing anything or doing anything wrong?
|
@leenasn Sorry, my instructions were wrong, our Heroku deploys use a custom buildpack so that they're faster. You need to change the buildpack like this:
However, people have reported memory problems with 0.30.4 as well so I'd recommend going back to whatever version you were on previously until we fix the memory issue (likely this week) You'll also need to trigger a new deploy by adding a commit and pushing:
|
@tlrobinson the above also didn't work out, because when I changed the buildpack to use a specific version, it stopped downloading the Jar. I forked the buildpack and changed the version to 0.30.4 and then deployed and it worked. And for me, 0.30.4 is working fine without memory issues. Thank you for the support, looking forward to the new version with the fix for memory issues. |
Fixed by metabase/metabase-deploy#22 Try deploying again and it should work pick up the new deploy script (I think) in which case it should be working without issue |
@camsaul I tried it. But still the same result, OutofMemory error. I tried with free, hobby and Std 1X dynos. It works in the free dyno for version 0.30.4. Hope this helps. |
@camsaul this hasn't been fixed. i get the same issues since i updated metabase-deploy and update to v0.31 and now v0.31.1. v0.30.4 worked fine before. even the PL instance doesn't fix it. OOM basically happens few minutes after every reboot. |
@camsaul thanks for opening it up again. do you have an ETA on this? if we need to deal with it for another week. i'd rather start looking for alternative hosting. |
@leenasn @kritop I pushed a change to
If that doesn't work please post the exact OOM logs you're seeing. |
Closing as fixed by metabase/metabase-deploy@c8147a7 If anyone is still having issues after trying @tlrobinson's fix please chime in |
I tried it and now am not getting OOM for 0.31 version. Yaay!! Thank you @tlrobinson @camsaul |
Same here, #3360 (comment) did the trick perfectly and all runs on the free Heroku container again. Thanks! |
same here!
…On Thu, 29 Nov 2018 at 11:51, Menno Holtkamp ***@***.***> wrote:
Same here, #3360 (comment)
<#3360 (comment)>
did the trick perfectly and all runs on the free Heroku container again.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3360 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AB8LIq9n997J1BhQvpSsByxfg4R8qLtkks5uz7wggaJpZM4J9Thu>
.
--
Christoph André Müller
CTO
coffeecircle.com | Richtig. Gut. Wir ♥ Kaffee.
T: +49 176 61018310
E: christoph@coffeecircle.com
Newsletter <https://www.coffeecircle.com/de/e/newsletter> | Facebook
<https://www.facebook.com/coffeecircle> | Youtube
<https://www.youtube.com/coffeecircle> | Instagram
<https://www.instagram.com/coffeecircle> | LinkedIn
<https://www.linkedin.com/company/circle-products-gmbh/>| Jobs
<https://www.coffeecircle.com/de/e/jobs-und-praktika>
Sitz der Gesellschaft: Lindower Str. 18, 13347 Berlin | Amtsgericht
Charlottenburg, HRB 128185 B | Geschäftsführer: Martin Elwert | USt-IdNr.
DE272941751
|
I've just updated and am still getting this:
Is there anything else I needed to do? |
I have seen this error in the log, but ignored it as it never affected the performance of the application so far :) |
Heroku's docs say |
Hi When I try to clone the metadata as explained in one of the comment above, it doesn't allow me. It throws error below. Can anyone help me? Thanks! To https://git.heroku.com/raj-movie-recommendationsystem.git |
Running v0.19.3 on Heroku - 2x (1GB) Dyno. (Used the one click install)
Metabase exceeds memory usage on both the 1x (512MB) Dyno and the 2x (1GB) Dyno. Memory usage immediately climbs after startup. Consistent R14 errors.
Screenshot of metabase DB on heroku:
I've connected only one Redshift database. With roughly 300,000 records. One dashboard and a few saved questions.
Seemed like this issue was previously fixed with a version update (#2704). Any thoughts/suggestions?
The text was updated successfully, but these errors were encountered: