[Cache] memcache fix for J3.6.2, memcache(-d) performance improvements, less connections, add support for 'platform specific caching' #11565

csthomas · 2016-08-11T23:44:22Z

Pull Request for Issue (Not yet)

Summary

Cache storage memcache from J3.6.2 is not working.
This PR fix that.

I also add the same changes to memcached.

This is part of another (more optimized) PR #10565.

Summary of Changes

[UPDATED]

Fix compression issue.
Before only first instance of memcache can set compression to true on getConnection method.
Next instances did not call getConnection and then did not use compression at all.
Move $this->_compress = ... to consctructor.

https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aR47

Check static::$_db === null before isSupported().
If connection exists then skip call isSupported and do not create useless connections.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL55
Fix persistent connection. Remove test connection which break persistent.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL79
Remove race condition. We do not need to initiate empty list.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL89
Joomla uses only one memcache server. Remove replace and use only set method.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL194
Fix remove method. Search in all list instead of in first item only.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL236
Do not test connection in isSupported method https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL311

In global configuration we should have an option to set another custom port if default does not work.

Do not lock twice. Do not call lockindex for lock/'unlock' method.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL334
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aL395
Fix locklooped.
https://github.com/joomla/joomla-cms/pull/11565/files#diff-8104780960bee59ef8a377ffd818797aR375

https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/cache/controller/callback.php#L99

Override _getCacheId($id, $group) for memcache(-d).
If Platform specific caching in Global Configuration->Cache Settings is enabled
then mobile (user agent) user without that patch could not delete/see all cache data.

Testing Instructions

On Joomla 3.6.2 or newer try to use Cache handler MemcachE and MemcacheD
Test Joomla with memcachE and MemcacheD handler on J 3.6.2 (without and with patch)
Check Backend->Cache Clear page.

(Optional)
Check how many connection (in memcached stats) generates joomla before patch and after patch.
After patch there should be less connections.

Other method to test.

#11565 (comment)

photodude · 2016-08-12T01:09:35Z

Looks like this PR reveals a memcache issue with our storage controller and HHVM
Fatal error: [] operator not supported for strings in /libraries/joomla/cache/storage/memcache.php on line 194

Look like HHVM doesn't like " assigning values to the array by omitting the key, resulting in an empty pair of brackets ([])."

$arr[key] = value;
$arr[] = value;
// key may be an integer or string
// value may be any value of any type

If $arr doesn't exist yet, it will be created, so this is also an alternative way to create an array. This practice is however discouraged because if $arr already contains some value (e.g. string from request variable) then this value will stay in the place and [] may actually stand for string access operator. It is always better to initialize variable by a direct assignment.

I believe we can fix this by changing line 194 from $index[] = $tmparr; to array_push($index, $tmparr);

Fix broken memcache storage on J3.6.2, refresh code joomla#11565

csthomas · 2016-08-12T07:05:42Z

I can change it but I think that HHVM do not use unserialize on $index.

$index all time should be array or false if not set.
Memcache implicitly serializes array to string and stores that string in memory.
May be HHVM do not use unserialize after get variable and we have to do it explicitly.

photodude · 2016-08-12T13:05:20Z

From what I can tell the following line is returning a string rather than an array or false.

$index = static::$_db->get($this->_hash . '-index');

csthomas · 2016-08-12T18:20:29Z

I can not confirm that on my local hhvm.

with hhvm 3.14.4 server from (deb http://dl.hhvm.com/ubuntu xenial main):

hhvm -m server -d hhvm.server.port=9000 -d hhvm.server.source_root=/home/xxx/public_html/demo362/

URLs:

works.

I have got only:
"Warning: Division by zero in /home/tomash/public_html/demo362/plugins/system/debug/debug.php on line 589"

photodude · 2016-08-12T20:14:48Z

I'm just going by the conditions necessary to reproduce the error as found on Travis CI.

You can see the mock code at this link https://3v4l.org/HLGlZ

You can change $index to an array, null or to false and there is no error. But if $index is a string, the array modify opperator with throw the error.

The only conditions for $index to be a string in the cache MEMCACHE store() code is for the following line to return a string

$index = static::$_db->get($this->_hash . '-index');

It's possible that it's only returning a string under our PHP unit tests.

csthomas · 2016-08-12T21:26:58Z

IMHO there is bug in travis tests, or php-memcache is too old.
I use this one 3.0.9 from:
https://launchpad.net/ubuntu/xenial/+source/php-memcache

Example which I do not understand:

if [[ $INSTALL_MEMCACHE == "yes" ]]; then phpenv config-add build/travis/phpenv/memcached.ini; fi

Why there is memcacheD extension loaded instead memcachE?

csthomas · 2016-08-12T22:04:19Z

Please correct me if I'm wrong.

Why did not travis see that memcachE can not connect to server after applied PR #9622.
Human tests missing memcachE too.

photodude · 2016-08-12T23:04:24Z

This is telling Travis CI to adjust the PHP environment if the PHP version needs to make special settings for MEMCACHE

if [[ $INSTALL_MEMCACHE == "yes" ]]; then phpenv config-add build/travis/phpenv/memcached.ini; fi

It's specific to Travis and the container settings and only sets up MEMCACHE in the PHP environment. It's also not required for some PHP versions. In some versions both MEMCACHE and MEMCACHED are supported.

Currently both MEMCACHE and MEMCACHED are supported, even though MEMCACHE is outdated and will go away at some future date. (HHVM only supports MEMCACHE and PHP 7 only supports MEMCACHED)
I would agree there is a bug if the MEMCACHE connection wasn't being made and php unit Travis CI was still passing the unit tests.

photodude · 2016-08-12T23:46:53Z

Looking deeper into what is happening at this line.

$index = static::$_db->get($this->_hash . '-index');

Memcache::get returns a string unless it was passed an array of keys.

http://php.net/manual/en/memcache.get.php

This is why $index is a string when the array modifier opperator tries to append $tmparr to the value. Causing the error.

csthomas · 2016-08-13T12:53:35Z

Currently both MEMCACHE and MEMCACHED are supported, even though MEMCACHE is outdated and will go away at some future date. (HHVM only supports MEMCACHE and PHP 7 only supports MEMCACHED)

I thought that too but when I checked yesterday MEMCACHE is still working on php7.0 - you can test it on ubuntu 16.04 LTS.

if [[ $INSTALL_MEMCACHE == "yes" ]]; then phpenv config-add build/travis/phpenv/memcached.ini; fi

IMHO this is a bug/typo, it should be:

if [[ $INSTALL_MEMCACHE == "yes" ]]; then phpenv config-add build/travis/phpenv/memcache.ini; fi

I created a PR for that #11576.

mbabker · 2016-08-13T13:19:46Z

libraries/joomla/cache/storage/memcache.php

@@ -31,15 +31,15 @@ class JCacheStorageMemcache extends JCacheStorage
 	 * @var    boolean
 	 * @since  11.1
 	 */
-	protected $_persistent = false;
+	protected static $_persistent = false;


There's no need to make these static. Actually I'd say that's a step backward. The only reason the Memcache instance is static is because Joomla still doesn't believe in proper dependency injection but prefers global singletons.

Let me explain my point of view.

Before start set compression for memcache to true in Global configuration;

Example:

Component create instance of cache with some options like lifetime=600

Cache instance get instance of storage:

$hash = md5(serialize($this->_options)); if (isset(self::$_handler[$hash])) { return self::$_handler[$hash]; } self::$_handler[$hash] = JCacheStorage::getInstance($this->_options['storage'], $this->_options);

first storage instance is created, compression=1, yes?

Next joomla get modules.

Module has different lifetime=900 then options is different then joomla create another instance
of storage.

Now we have to storages of memcache - do you agree?.

First storage (before patch) has set compression=1 because getConnection() is called

Second storage instance (before patch) has compressio=0 because this does not need to create another connection, (because use existed static::$db)

What we have before patch? Only first instance of storage uses compression.

If you understand above, then
should I change it and each time of call __construct() set

$this->compression = JFactory::getConfig()->get('memcache_compress', false) ? MEMCACHE_COMPRESSED : 0;

Is it better?

This probably won't happen before 4.0 but I'd rather see all the adapters
have the internal connection injected versus us having such a static
singleton based API. We should be able to create a single Memcache(d)
connection and drop that into both the cache and session as an example.
But back on track...

So right now because the connection is static there's little value in these
options being non static if they only affect the connection params.
However if they can affect other behaviors they should be configurable
per-instance. Granted I don't see how this is actually useful with our
current structure but imagine a scenario where different cache data might
be stored on different Memcache(d) instances for whatever reason, each
connection should be configurable basically.

It just feels wrong that whatever ends up being the first call to these
class' constructors dictates the connection for the full request without
getting into some PHP magic (i.e. Reflection or overloading the autoloader
and creating custom classes).

On Saturday, August 13, 2016, Tomasz Narloch notifications@github.com
wrote:

In libraries/joomla/cache/storage/memcache.php
#11565 (comment):

@@ -31,15 +31,15 @@ class JCacheStorageMemcache extends JCacheStorage
* @var boolean
* @SInCE 11.1
*/

protected $_persistent = false;

protected static $_persistent = false;

Let me explain my point of view.

Before start set compression for memcache to true in Global configuration;

Example:

Component create instance of cache with some options like lifetime=600

Cache instance get instance of storage:

$hash = md5(serialize($this->_options)); if (isset(self::$_handler[$hash])) { return self::$_handler[$hash]; } self::$_handler[$hash] = JCacheStorage::getInstance($this->_options['storage'], $this->_options);

first storage instance is created, compression=1, yes?

Next joomla get modules.

Module has different lifetime=900 then options is different then joomla
create another instance
of storage.

Now we have to storages of memcache - do you agree?.

First storage (before patch) has set compression=1 because
getConnection() is called

Second storage instance (before patch) has compressio=0 because this
does not need to create another connection, (because use existed
static::$db)

What we have before patch? Only first instance of storage uses compression.

If you understand above, then
should I change it and each time of call __construct() set

$this->compression = JFactory::getConfig()->get('memcache_compress', false) ? MEMCACHE_COMPRESSED : 0;

Is it better?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/joomla/joomla-cms/pull/11565/files/4241cbcde6f909b613017fdd4dac8ea9d1f64ad2#r74691810,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAWfoUTZolV7LHqcIg9DyR2hsIfSwLGAks5qfihcgaJpZM4JiqOc
.

csthomas · 2016-08-13T21:38:54Z

@mbabker I changed problematic stuff.
I also removed protected $_persistent = false; because it is not using now.
If it is not B/C then I can revert it, but it is useless.

photodude · 2016-08-15T02:24:58Z

libraries/joomla/cache/storage/memcache.php

-		{
-			$index = array();
-		}
+		$index = $index ? $index : array();


I'm not sure if changing to a ternary operator is a good design choice here. The ternary operator in PHP does a copy before evaluating the statement, This results in a performance hit as a data set being evaluated grows (i.e. the more data in the $index variable the slower the ternary operator evaluation is)

Personally, I avoid using ternary operators with objects or arrays since you risk the performance hit from the variable copy it does before evaluation.

Thanks for tip.

csthomas · 2016-08-15T22:36:38Z

I have removed ternary and rebased to the latest staging to resolve the conflict.
Summary of changes has been updated.

photodude · 2016-08-16T18:55:08Z

@csthomas Would you change line 190 in the store() method from from $index[] = $tmparr; to array_push($index, $tmparr); ? This change will solve the HHVM error Fatal error: [] operator not supported for strings in /libraries/joomla/cache/storage/memcache.php on line 190 that occurs when $index = static::$_db->get($this->_hash . '-index'); returns a string. (or otherwise deal with the case when Memcache::get returns a string so we don't have this fatal error hanging out there)

I've submitted a PR to your branch for this change.

csthomas · 2016-08-17T13:45:24Z

Is the more issue there?

Can I ask for test.

Test should be simple: check whether cache works or not, test clear cache, etc.
For more advanced testers: code review.

mbabker · 2016-08-17T14:07:00Z

Test should be simple: check whether cache works or not, test clear cache, etc.

Finding people with access to certain PHP configurations is challenging at times. I have MemcacheD set up on my local system but not Memcache and I don't have any servers running the latter to test this against.

csthomas · 2016-08-17T15:15:09Z

If you have access to root account you could run: pecl install memcache-3.0.8 for php5.
After test you can display installed items by pecl list and uninstall by pecl uninstall memcache.

csthomas · 2016-08-17T15:20:08Z

May be someone with HHVM can test, memcache extension is probably built in.

csthomas · 2016-08-17T15:30:02Z

Example which I use for hhvm on kubuntu 16.04 LTS.

Install hhvm
$ sudo apt-get install hhvm
Run www server on port 8000 and set source_root to joomla directory
$ hhvm -m server -d hhvm.server.port=8000 -d hhvm.server.source_root=/home/[...]/public_html/demo362/
Go to http://localhost:8000/administrator/index.php and test:)

photodude · 2016-08-19T03:39:47Z

I have tested this item ✅ successfully on 3aa94f5

Works, with this patch Memcache is now a selectable option for cache. Where before memcache only worked for the session handler. everything functions correctly after enabling memcache in global settings

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/11565.}

yvesh · 2016-08-19T05:09:30Z

libraries/joomla/cache/storage/memcached.php

+
+			if ($servers)
+			{
+				if ($servers[0]['host'] != $host || $servers[0]['host'] != $port)


You can combine that line with the if one above (&&) and avoid one more level of nesting.

Also is this correct $servers[0]['host'] != $port ?

csthomas · 2016-08-19T09:32:12Z

Thanks @yvesh for code review.
Thanks @photodude for your test.

After my commit I probably still need 2 success tests.

gwsdesk · 2016-08-24T15:54:41Z

Maybe a stupid question but we have Memcached running on our servers and on our Joomla 3.6.2 websites without any problems (PHP 5.6.24 with cloudlinux) So what do we need to fix which works for us?

csthomas · 2016-08-24T16:58:25Z

Maybe a stupid question but we have Memcached

This PR only add improvement for memcacheD

But you can not use MemcachE on J3.6.2

csthomas · 2016-08-24T20:38:52Z

If memcache handler does not work on J3.6.2 and nobody complain about it (except me) then could testers test at least memcached handler.

There are the same changes in both, except getConnection(), which has a few more differences.

@photodude @mbabker Can I ask you to to make a test?

gwsdesk · 2016-08-25T02:26:00Z

I have tested Memcached ✅ on PHP 5.6.24/Cloudlinux/Mariadb/cgi-fcgi on Joomla 3.6.2 successfully This runs well on a Joomla 3.6.2 multi lingual testsite gwsdev2.work. I also tested this on the same site with PHP7 enabled and works fine as well

Short remark on Memcache. Memcache is not available yet for PHP7. This is the reason why we have removed it from the PHP-extensions in the PHP-selector of Cloudlinux altogether and for now just provide Memcached. This is not any issue since in the CL environment you get OPCache and that works just so nice and fast in combination with Memcached

csthomas · 2016-08-25T08:33:24Z

Thank you @gwsdesk.

Short remark on Memcache. Memcache is not available yet for PHP7

Official not, but at least one linux distribution have memcache working with PHP7.

If you have Ubuntu 16.04 on some machine then you can install php-memcache (PHP7) from default repository. There is probably unofficial version of memcache compiled in deb.

#11565 (comment)

gwsdesk · 2016-08-25T08:38:35Z

Thanks Thomasz but Cloudlinux does not support Memcache in PHP7 yet. In
Centos we can also install Memcache from default and that works I can
confirm. CL no go at present though they have a Beta version

On 8/25/2016 3:33 PM, Tomasz Narloch wrote:

Thank you @gwsdesk https://github.com/gwsdesk.
Short remark on Memcache. Memcache is not available yet for PHP7
Official not, but at least one linux distribution have memcache
working with PHP7.

If you have Ubuntu 16.04 on some machine then you can install
php-memcache (PHP7) from default repository. There is probably
unofficial version of memcache compiled in deb.

#11565 (comment)
#11565 (comment)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11565 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHzLNkC-weu902hW0b08oA4bdnuGK06zks5qjVNfgaJpZM4JiqOc.

csthomas · 2016-08-25T08:46:59Z

Can your test can be mark as successful at https://issues.joomla.org/tracker/joomla-cms/11565

If you have a few minutes You can test memcache as follow:
#11565 (comment)

gwsdesk · 2016-08-25T09:11:23Z

I have tested this item ✅ successfully on 4e5e96b

I have tested Memcached on PHP 5.6.24/Cloudlinux/Mariadb/cgi-fcgi on Joomla 3.6.2 successfully ✅ .This runs well on a Joomla 3.6.2 multi lingual testsite gwsdev2.work. I also tested this on the same site with PHP7 enabled and works fine as well

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/11565.}

photodude · 2016-08-29T03:45:39Z

I have tested this item ✅ successfully on 4e5e96b

Marking successful, with the following note.

As soon as I applied this revised patch I got the following notice

Warning: Error loading component: com_languages, Could not connect to memcache server

I assume this warning was due to an old cache state; as it went away after turning cache off and then back on in Global settings. Since it was a transient message and resolved after resetting the cache I don't think there is anything to do but note the message.

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/11565.}

gwsdesk · 2016-08-29T05:03:08Z

@photodude this means that it is not successful since it shows the original error (which goes away if you disable caching or change to file. Please remove the 'successful' tag and replace with 'unsuccessful' ? ❌
This is not just a 'warning' is means it does not work!
I retested and cannot replicate that warning

csthomas · 2016-08-29T10:00:43Z

I don not know @photodude steps but I assume he enabled memcachE before applying patch.
Warning was stored in session and displayed at next page view.

Besides above.
There was not any changes added to memcachE handler from the first @photodude test.
The last 2 commits only added changes to memcacheD handler.

photodude · 2016-08-29T14:17:17Z

Here is what I had done. @gwsdesk

tested the patch in its old state using patch tester (see first test)
Removed the patch
Applied the patch in its current state
Warning appeared at this point
Turned off cache in global settings (forced logout)
(Logged in) turned cache back on
No warnings or issues using joomla front end or back end.

As you can see the warning was due to the inconsistent state the cache was in between reapplying this patch after it was updated. I stand by my choice in marking it as a successful test.

gwsdesk · 2016-08-29T14:25:01Z

@photodude I read it correctly now and only noticed now your initial test as well. Thanks for successful testing ;-)

photodude · 2016-08-30T01:46:06Z

@rdeutz can this be marked RTC now?

rdeutz · 2016-08-30T05:17:42Z

@photodude yes it can

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/11565.}

zero-24 · 2016-08-30T06:54:19Z

@joomla-cms-bot can you please help here? Thanks 😄

joomla-cms-bot added the PR-staging label Aug 11, 2016

csthomas changed the title ~~Fix memcache storage, refresh code~~ Fix broken memcache storage on J3.6.2, refresh code Aug 11, 2016

csthomas pushed a commit to csthomas/joomla-cms that referenced this pull request Aug 12, 2016

Sync with Joomla PR joomla#11565

bd3baf9

photodude added a commit to photodude/joomla-cms that referenced this pull request Aug 12, 2016

Copy PR 11565 changes

07c8cc4

Fix broken memcache storage on J3.6.2, refresh code joomla#11565

csthomas force-pushed the memcache_1 branch from 9ce8527 to 4241cbc Compare August 12, 2016 09:26

csthomas pushed a commit to csthomas/joomla-cms that referenced this pull request Aug 12, 2016

Sync with Joomla PR joomla#11565

fca82d0

mbabker reviewed Aug 13, 2016
View reviewed changes

photodude reviewed Aug 15, 2016
View reviewed changes

csthomas force-pushed the memcache_1 branch from 424d036 to d587b4b Compare August 15, 2016 21:49

csthomas mentioned this pull request Aug 16, 2016

[tests] Bug in run test for memcache/memcached in PR #11565 #11634

Closed

csthomas mentioned this pull request Aug 18, 2016

[memcache] Split one array of memcache index to more indexes in order to better performance #10565

Closed

yvesh reviewed Aug 19, 2016
View reviewed changes

Fix typo, avoid netsted if

6253625

Next typo in msg Memcache => Memcached

4e5e96b

csthomas changed the title ~~Fix broken memcache storage on J3.6.2, refresh code~~ [Cache] memcache fix for J3.6.2, memcache(-d) performance improvements, less connections, add support for 'platform specific caching' Aug 22, 2016

photodude mentioned this pull request Aug 23, 2016

HHVM errors from Travis ci #10220

Closed

csthomas mentioned this pull request Aug 24, 2016

[cache] Catch cache storage exception #10815

Closed

joomla-cms-bot added the RTC This Pull Request is Ready To Commit label Aug 30, 2016

rdeutz added this to the Joomla 3.6.3 milestone Aug 30, 2016

rdeutz merged commit 97365ea into joomla:staging Aug 30, 2016

joomla-cms-bot removed the RTC This Pull Request is Ready To Commit label Aug 30, 2016

csthomas deleted the memcache_1 branch August 30, 2016 07:15

[Cache] memcache fix for J3.6.2, memcache(-d) performance improvements, less connections, add support for 'platform specific caching' #11565

[Cache] memcache fix for J3.6.2, memcache(-d) performance improvements, less connections, add support for 'platform specific caching' #11565

Conversation

csthomas commented Aug 11, 2016 • edited by rdeutz

Summary

Summary of Changes

Testing Instructions

Other method to test.

photodude commented Aug 12, 2016 • edited

csthomas commented Aug 12, 2016

photodude commented Aug 12, 2016 • edited

csthomas commented Aug 12, 2016

photodude commented Aug 12, 2016 • edited

csthomas commented Aug 12, 2016

csthomas commented Aug 12, 2016

photodude commented Aug 12, 2016

photodude commented Aug 12, 2016 • edited

csthomas commented Aug 13, 2016

mbabker Aug 13, 2016

Choose a reason for hiding this comment

csthomas Aug 13, 2016

Choose a reason for hiding this comment

mbabker Aug 13, 2016

Choose a reason for hiding this comment

csthomas commented Aug 13, 2016 • edited

photodude Aug 15, 2016 • edited

Choose a reason for hiding this comment

csthomas Aug 15, 2016

Choose a reason for hiding this comment

csthomas commented Aug 15, 2016

photodude commented Aug 16, 2016 • edited

csthomas commented Aug 17, 2016

mbabker commented Aug 17, 2016

csthomas commented Aug 17, 2016

csthomas commented Aug 17, 2016

csthomas commented Aug 17, 2016

photodude commented Aug 19, 2016 • edited

yvesh Aug 19, 2016

Choose a reason for hiding this comment

csthomas commented Aug 19, 2016

gwsdesk commented Aug 24, 2016

csthomas commented Aug 24, 2016

csthomas commented Aug 24, 2016

gwsdesk commented Aug 25, 2016 • edited

csthomas commented Aug 25, 2016

gwsdesk commented Aug 25, 2016

csthomas commented Aug 25, 2016

gwsdesk commented Aug 25, 2016

photodude commented Aug 29, 2016

gwsdesk commented Aug 29, 2016 • edited

csthomas commented Aug 29, 2016

photodude commented Aug 29, 2016

gwsdesk commented Aug 29, 2016

photodude commented Aug 30, 2016

rdeutz commented Aug 30, 2016

zero-24 commented Aug 30, 2016

csthomas commented Aug 11, 2016 •

edited by rdeutz

photodude commented Aug 12, 2016 •

edited

photodude commented Aug 12, 2016 •

edited

photodude commented Aug 12, 2016 •

edited

photodude commented Aug 12, 2016 •

edited

csthomas commented Aug 13, 2016 •

edited

photodude Aug 15, 2016 •

edited

photodude commented Aug 16, 2016 •

edited

photodude commented Aug 19, 2016 •

edited

gwsdesk commented Aug 25, 2016 •

edited

gwsdesk commented Aug 29, 2016 •

edited