Improve memory footprint of queue workers #32

timabbott · 2015-09-26T18:30:53Z

Since it seems there's a good amount of interest in running Zulip in smaller environments, it's worth doing a bit of optimization on the Zulip memory footprint. I think this will also be pretty valuable for running the Zulip development environment on a laptop.

I think probably the simplest version of this implementation would be a way to run all the queue workers in a single process that just watches all the queues; that should be great for development and smaller servers, and easy to make work (no need to even make it multithreaded).

Part 2 would be to make that multithreaded.

timabbott · 2015-09-26T18:52:46Z

The places in the code to start looking at this are zerver/worker/queue_processors.py and zerver/management/commands/process_queue.py.

To start with getting this working in the development environment, you can patch tools/run-dev.py to run things with your new tooling.

dosire · 2015-09-27T19:46:44Z

Would be very cool to make this multithreaded, the 4GB memory requirement for a production server is very high.

timabbott · 2015-12-14T06:19:22Z

#367 will make the Zulip workers multithreaded in the development environment; there will be a bit more work to do this in production since we'll probably want to offer both configuration options...

dosire · 2015-12-15T01:45:13Z

@timabbott cool! I'm not sure why anyone would want non-multithreaded workers, or is it temporarily until multithreading is stable?

timabbott · 2015-12-15T02:00:29Z

Well there are two considerations:

debuggability -- it will likely be easier to debug some things in the multi-process configuration than the multithreaded configuration.
stability -- currently if a worker crashes, the rest of them keep running happily; with multithreading, it's possible for a worker crashing to bring them all down.

Both of these seem like things that shouldn't be a problem in the long term, but likely are important enough that I'd imagine we want to support both options for a transitional period...

dosire · 2015-12-15T02:02:14Z

Totally agree about the transitional period.

This change drops the memory used for Python processes run by Zulip in development from about 1GB to 300MB on my laptop. On the front of safety, http://pika.readthedocs.org/en/latest/faq.html explains "Pika does not have any notion of threading in the code. If you want to use Pika with threading, make sure you have a Pika connection per thread, created in that thread. It is not safe to share one Pika connection across threads.". Since this code only connects to rabbitmq inside the individual threads, I believe this should be safe. Progress towards zulip#32.

This is a follow-up to #32.

…es API. The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurence in a topic. This list is passed back in the `topic_links` parameter of the get-messages API response. topic_links before vs now: before: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Tests amended.

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links before vs now: before: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Tests amended.

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurrence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links before vs now: before: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Tests amended.

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurrence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links earlier vs now: earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Tests amended.

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurrence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links earlier vs now: earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Also, adjusted the recipient_row template to use the url parameter of the topic_links to set as the href. Tests amended.

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurrence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links earlier vs now: earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com", 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Similarly, the topic_links local echo logic in the frontend now returns back an object. Tests amended. Fixes: zulip#17109

The linkifier code now includes both the shortened text and the expanded URL, sorted by the order of the occurrence in a topic. This list is passed back in the `topic_links` parameter of the /messages and the /events APIs. topic_links earlier vs now: earlier: ['https://www.google.com', 'https://github.com/zulip/zulip/32'] now: [{'url': 'https://www.google.com', 'text': 'https://www.google/com}, {'url': 'https://github.com/zulip/zulip/32', 'text': 'zulip#32'}] Similarly, the topic_links local echo logic in the frontend now returns back an object. Fixes: zulip#17109.

timabbott mentioned this issue Sep 26, 2015

Decrease Zulip server minimum memory footprint to better support smaller installations and embedded devices. #34

Closed

timabbott added the enhancement label Sep 27, 2015

timabbott mentioned this issue Oct 1, 2015

Ubuntu 14.04, Digitalocean - VM kernel panic #51

Closed

timabbott added the area: production label Oct 14, 2016

rht mentioned this issue May 28, 2017

[PROD] Run queue processes multithreaded in production if system memory <4GB. #5098

Closed

timabbott closed this as completed in 940cf9d Jun 3, 2017

timabbott added a commit that referenced this issue Jun 4, 2017

docs: Update installation docs for new RAM requirements.

e8166ee

This is a follow-up to #32.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve memory footprint of queue workers #32

Improve memory footprint of queue workers #32

timabbott commented Sep 26, 2015

timabbott commented Sep 26, 2015

dosire commented Sep 27, 2015

timabbott commented Dec 14, 2015

dosire commented Dec 15, 2015

timabbott commented Dec 15, 2015

dosire commented Dec 15, 2015

Improve memory footprint of queue workers #32

Improve memory footprint of queue workers #32

Comments

timabbott commented Sep 26, 2015

timabbott commented Sep 26, 2015

dosire commented Sep 27, 2015

timabbott commented Dec 14, 2015

dosire commented Dec 15, 2015

timabbott commented Dec 15, 2015

dosire commented Dec 15, 2015