Benchmark Mcrouter vs Memcached vs no caching

[geerlingguy]

One thing that would be nice to see is whether the use of this operator would result in better performance for your applications (noting that performance is only one of the two metrics mcrouter tries to optimize, the other being availability).

I'm thinking maybe of running a benchmark on a CMS like Wordpress or Drupal with front end caching disabled, in the following scenarios:

1. Without any kind of caching layer.
2. With memcached, one pod/instance.
3. With mcrouter in front of 4 pods using a 'replicated' pool.
4. With mcrouter in front of 4 pods using a 'sharded' pool.

Could use something like wrk or ab for basic HTTP tests, but maybe some better benchmarking tool?

[geerlingguy]

Can also use memaslap: http://docs.libmemcached.org/bin/memaslap.html

[geerlingguy]

So when trying to test memslap, I found that installing libmemcached via brew (brew install libmemcached), I only got the latest stable version, 1.0.18, which was released in 2014.

When I used it's version of memslap I got little helpful information:

$ memslap -s 192.168.64.13:30534
    Threads connecting to servers 1
    Took 5.464 seconds to load data

Also, even now and then I'd hit weird hiccups like:

clients/execute.cc:32 Failure on 1071 insert ((0x7fe485e408c0) A TIMEOUT OCCURRED, No active_fd were found,  host: 192.168.64.13:30534 -> libmemcached/io.cc:259) of xpPSFNO4ZoikYMxmM7X3

So I ditched trying to benchmark with memslap. Also found this: facebook/mcrouter#91

So could also look into using twemperf.

[geerlingguy]

So... been trying to get wordpress to be happy with Mcrouter. I tried out the Memcached Redux plugin, but even though it seemed to connect and use the cluster properly, it resulted in performance more than halving, from 55 req/s to 20 req/s after I placed it's object-cache.php file in place and defined my cluster IP/port.

So then I tried W3TC, which is like a porsche compared to the fiat of the redux plugin... but it has a fancy connectivity test which fails when I connect to mcrouter, but passes when I connect directly to one of the memcached pods.

So... I think I give up on wordpress. At least for now.

Also, I noticed this issue during my searching: autopilotpattern/wordpress#28 — I hope they have better luck than I did :P

[geerlingguy]

Drupal instructions, using Drupal VM:

1. Download Drupal VM.

2. Create file local.config.yml in the Drupal VM directory, with contents:

   drupal_composer_dependencies:
     - "drupal/devel:^1.2"
     - "drupal/memcache:^2.0"
   installed_extras:
     - memcached
   

3. Run vagrant up

4. Visit http://drupalvm.test and log in with admin/admin

5. Visit the 'Extend' page and enable both 'Memcache' and 'Memcache Admin' modules.

6. Visit the Memcached statistics report page: http://drupalvm.test/admin/reports/memcache

At this point, it should show it's connecting to a local memcached instance on port 11211, but there is nothing being stored; this is because Drupal's memcached module requires some extra settings file configuration before it starts storing data in memcached.

1. Edit Drupal's settings.php file:

   1. Log into the VM with vagrant ssh
   2. Change permissions on the settings.php file: sudo chmod 644 sites/default/settings.php
   3. Edit the file: sudo vim sites/default/settings.php

2. Add the following on the end of the file:

   $settings['memcache']['servers'] = ['127.0.0.1:11211' => 'default'];
   $settings['memcache']['bins'] = ['default' => 'default'];
   $settings['memcache']['key_prefix'] = '';
   $settings['cache']['default'] = 'cache.backend.memcache';
   

3. Refresh the Memcached statistics report page (http://drupalvm.test/admin/reports/memcache)

You should now see some sets and gets in the report. This means the local memcached instance is working, and Drupal is able to store and retrieve cache data.

[geerlingguy]

Now let's switch to Mcrouter: sudo vim sites/default/settings.php, and then change the $settings['memcache']['servers'] key to 192.168.64.14:30588 (or however you can reach your local Mcrouter instance; I added a NodePort service so I could access it through a minikube IP and port).

If you refresh the statistics report page you'll get a notice of "No default statistics for this bin."—however, the bottom of the page shows that values are still being retrieved (all the gets) from Memcached. Also, after some usage, you should start getting full statistics on the stats page again.

You can run stats servers against mcrouter (via telnet) to see how many objects are notfound and stored on the servers. If you're using a replicated setup, then the stored amount should be exactly the same on all the servers, for example:

stats servers
STAT mcrouter-memcached-0.mcrouter-memcached.default.svc.cluster.local:11211:ascii:plain:notcompressed-1000 avg_latency_us:400.321 pending_reqs:0 inflight_reqs:0 avg_retrans_ratio:0 max_retrans_ratio:0 min_retrans_ratio:0 up:1; deleted:4 found:910 notfound:262 stored:984
STAT mcrouter-memcached-1.mcrouter-memcached.default.svc.cluster.local:11211:ascii:plain:notcompressed-1000 avg_latency_us:400.025 pending_reqs:0 inflight_reqs:0 avg_retrans_ratio:0 max_retrans_ratio:0 min_retrans_ratio:0 up:1; deleted:4 found:895 notfound:236 stored:984
STAT mcrouter-memcached-2.mcrouter-memcached.default.svc.cluster.local:11211:ascii:plain:notcompressed-1000 avg_latency_us:303.361 pending_reqs:0 inflight_reqs:0 avg_retrans_ratio:0 max_retrans_ratio:0 min_retrans_ratio:0 up:1; deleted:4 found:851 notfound:281 stored:984
END

[geerlingguy]

Some stats:

Setup	MySQL location	wrk req/s	ab req/s
Mcrouter, replicated in K8s	local in VM	85.95	6.62
Mcrouter, replicated in K8s	K8s	TODO	TODO
Mcrouter, sharded in K8s	local in VM	99.54	7.03
Mcrouter, sharded in K8s	K8s	51.74	6.34
Memcached, direct in K8s	local in VM	90.25	5.73
Memcached, direct in K8s	K8s	TODO	TODO
Memcached, direct in VM	local in VM	136.14	13.39
Memcached, direct in VM	K8s	49.85	8.20
No caching	local in VM	174.85	13.42
No caching	K8s	60.15	8.13

wrk: wrk -c4 -d20 -t4 http://drupalvm.test (tests anonymous access to the home page)
ab: ab -n 250 -c 10 -C "SESSxyz=XYZ" http://drupalvm.test/ (tests authenticated access to the home page)

All tests run four times, with first test discarded (for cache warmup); latter three are averaged.

[geerlingguy]

After running a number of tests, it looks like things are a bit slower when running through the network between VMs and into K8s, in both the case of direct memcached access and memcached-through-mcrouter.

So one additional test I want to try is running a MySQL instance inside K8s (just use something like https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/#deploy-mysql), and see how well that performs.

[geerlingguy]

Interesting... I discovered the reason why wrk commands were taking an extra 60 seconds or so—I had two entries for the local server in my /etc/hosts file, and it seems like it was timing out on the first entry then finally moving on to the second. Don't do that :D

[geerlingguy]

So... minikube started eating the dirt after about the 90th benchmark run. I got some good data, but std deviation was kind of all over the place, so I'm going to try to get a tiny bit more data using twemperf and wrap this up.

[geerlingguy]

Using the command:

kubectl run -it --rm mcperf --image=quay.io/redhat/mcperf --restart=Never -- -s mcrouter -p 5000 --linger=0 --timeout=5 --conn-rate=1000 --call-rate=1000 --num-calls=10 --num-conns=1000 --sizes=u1,16

Memcached setup	Requests per second
Mcrouter, replicated	9851
Mcrouter, sharded	9884
Single memcached instance, direct	9883

[geerlingguy]

Closing issue; summarized things in blog post: https://www.jeffgeerling.com/blog/2019/mcrouter-operator-demonstration-k8s-operator-sdk-usage-ansible

[diptomondal007]

@geerlingguy quay.io/redhat/mcperf this image doesn't exist anymore

[geerlingguy]

Shucks :(

Might have to create a separate one.