Configuration defaults - memory usage

[jaywink]

I'd like to change our defaults to make sure diaspora* uses the least amount of memory necessary in a basic production install. We can then adapt the install guides with some hints on how to raise performance.

Most pods however start small and someone running diaspora* on a small 1-2GB VM should be able to do so with default settings, not have to witness crashes and tune down.

Have I understood correctly that for minimal memory output, the following changes could be made?

sidekiq.concurrency 5 -> 1
sidekiq.backtrace true -> false
server.unicorn_worker 2 -> 1

What about settings.typhoeus_concurrency? Any other settings that would help pin down the memory footprint that Rails likes to suck up? :)

Also this would probably help with database connections not running out like many people are reporting, since by default sidekiq.concurrency would be lower. Still, it wouldn't hurt to bump database connection pool? Would that affect memory and I wonder what is a typical system like with connection amounts.. Whatever we use seems very low.

[jhass]

I guess there's no point in reasoning about these, my reasoning is what the current values came out of. Somebody has to try with different values how they affect memory usage and performance.

[jaywink]

Sure I'd be glad to run some real field tests first - just wanted to open discussion as sick of constantly answering the OOM calls from new podmins running on <2 GB servers.. :)

[jhass]

I run a 2GB server with lots of stuff besides diaspora: https://usage.aeshna.de/aeshna.de/aeshna.aeshna.de/memory.html

[lazzarello]

I can back up this thread and agree with @jaywink I run a pod with exactly 2GB of memory and I have regular OOM errors. I realize that the "it works for me" situation is tempting to fall back on and @jhass also happens to be a very active committer on the D* source so perhaps I can help lend a third opinion to this discussion, which I think needs some action. In my opinion the defaults for a production Diaspora pod are too expensive for an average cloud server. As it stands Diaspora is the most resource intensive volunteer VM I run and also requires the most maintenance.

It seems that the Sidekiq authors have some best practices advice for memory optimization, which I'd like to look into myself. There's this vague mention that Typhoeus may cause some unexpected side effects. There's also some recommendations about application code changes to limit the size of collections that get the perform_async method called.

Anyhoo, I do think Diaspora uses too many resources to be considered "accessible" for a modest Linux enthusiast to run their own pod. It's also a bit high for those of us who already donate server space to many projects.