Add memcached support as an alternative to the file-based cache.

[GoogleCodeExporter]

Some basic support has been implemented and is under review.

Original issue reported on code.google.com by jmara...@google.com on 20 Jul 2012 at 7:51

• Blocking: Store the re-written file to another nodes #474
• Blocked on: LoadFromFile + memcached -> meta-data cache timestamp problems #488, load-test memcached integration failure modes #522

[GoogleCodeExporter]

Original comment by jmara...@google.com on 20 Jul 2012 at 8:28

• Now blocking: Store the re-written file to another nodes #474

[GoogleCodeExporter]

http://code.google.com/p/modpagespeed/source/detail?r=1704 contains the first 
round of changes needed to add memcached support.

we don't have get performance metrics; actually as it stands right now we have 
bad performance metrics, but there is hope.

Original comment by jmara...@google.com on 20 Jul 2012 at 8:30

[GoogleCodeExporter]

http://code.google.com/p/modpagespeed/source/detail?r=1707 adds an option 
"ModPagespeedMemcachedServers" to mod_pagespeed.  Note that this support for 
this appears functional but is completely untuned.  In particular:

1. No testing for multiple memcached servers is done yet, though support for 
this
   appears to be built into the underlying apr_memcache* calls in apr_util.
2. The performance under load using a single local memcached server is inferior 
to
   what you get with a tmpfs, although this can be probably be improved by batching
   requests and other techniques.
3. The size of objects sent to memcached is limited to about 1M, so images 
larger
   than this cannot be optimized at all in the current revision.  To address this
   we may need to still enable the file-cache for large objects.

Original comment by jmara...@google.com on 24 Jul 2012 at 1:36

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

http://code.google.com/p/modpagespeed/source/detail?r=1723 gets memcached 
system tests working in the Prefork MPM (previous testing was in Worker).

Original comment by jmara...@google.com on 25 Jul 2012 at 8:11

[GoogleCodeExporter]

Progress report.  As of 
http://code.google.com/p/modpagespeed/source/detail?r=1777 I am reasonably 
confident that this implementation is correct.  Prior to that change there was 
a small chance of incorrect data on lookup due to md5 collision; now we store 
the key with the data and use that to verify.

It could use more tuning but is not too bad.  In my loadtests I was thinking it 
was performing worse than tmpfs but that was because the default memcached max 
size is 64Mb and we had way more data than that in our load-test scripts.  I 
bumped up the capacity to 1G by running "memcached -m 1000" and now it performs 
almost as well as tmpfs, but provide the benefit of  network scalability.

We don't have doc published yet, but if you build from trunk and put in 

   ModPagespeedMemcachedServers host1:port1,host2:port2,...

then you can give this a try.


Still to come:  proper doc, and try to get more performance using multiget.

Original comment by jmara...@google.com on 11 Aug 2012 at 3:03

[GoogleCodeExporter]

To fully support memcached we need to solve Issue 488, which observes that with 
ModPagespeedLoadFromFile we are storing local filesystem timestamps into our 
metadata cache.

Original comment by jmara...@google.com on 5 Sep 2012 at 1:31

• Now blocked on: LoadFromFile + memcached -> meta-data cache timestamp problems #488

[GoogleCodeExporter]

Original comment by jmara...@google.com on 12 Oct 2012 at 1:24

• Now blocked on: load-test memcached integration failure modes #522

[GoogleCodeExporter]

Original comment by j...@google.com on 26 Oct 2012 at 4:58

• Added labels: Milestone-v23

[GoogleCodeExporter]

memcached support is now load-tested, failure-tested, and fully integrated into 
the trunk.

For any & all listening, please build from trunk & give it a try!

Original comment by jmara...@google.com on 1 Nov 2012 at 6:04

• Added labels: release-note

[GoogleCodeExporter]

Original comment by jmara...@google.com on 1 Nov 2012 at 6:04

• Changed state: Fixed