Fix/1531 add unix domain socket #1629

jradtilbrook · 2017-07-04T13:28:43Z

This PR fixes #1531 - allowing MPD extension to listen on a unix socket as well as network socket. I've tried to comply with the development guide.
Any and all feedback appreciated.

kingosticks

I also get an unhandled exception in the maximum_connections_exceeded case:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gi/overrides/GLib.py", line 727, in <lambda>
    func_fdtransform = lambda _, cond, *data: callback(channel, cond, *data)
  File "/work/nsteel/projects/mopidy-dev/mopidy/mopidy/internal/network.py", line 122, in handle_connection
    self.reject_connection(sock, addr)
  File "/work/nsteel/projects/mopidy-dev/mopidy/mopidy/internal/network.py", line 144, in reject_connection
    logger.warning('Rejected connection from [%s]:%s', addr[0], addr[1])
IndexError: string index out of range

A test to catch that would be good.

kingosticks · 2017-07-04T14:53:06Z

mopidy/mpd/actor.py

@@ -35,17 +37,17 @@ def __init__(self, config, core):
        super(MpdFrontend, self).__init__()

        self.hostname = network.format_hostname(config['mpd']['hostname'])
-        self.port = config['mpd']['port']
+        self.port = config['mpd']['port'] if 'port' in config['mpd'] else None


Do you need this change?

This was to get around the optional port config value above

Anything defined in the config's schema should always be defined in the dictionary (and set to None if not otherwise assigned a value by any other means). So I don't think you need this.

kingosticks · 2017-07-04T14:57:57Z

mopidy/mpd/__init__.py

@@ -19,7 +19,7 @@ def get_default_config(self):
    def get_config_schema(self):
        schema = super(Extension, self).get_config_schema()
        schema['hostname'] = config.Hostname()
-        schema['port'] = config.Port()
+        schema['port'] = config.Port(optional=True)


If we have this optional then we need some tests to verify that's OK. I'm getting:

ERROR 2017-07-04 15:52:46,569 [20285:MpdFrontend-14] pykka Unhandled exception in MpdFrontend (urn:uuid:e87ce82e-f8b3-41ef-b9a1-5062480a5e92): Traceback (most recent call last): File "/work/nsteel/projects/mopidy-dev/pykka/pykka/actor.py", line 192, in _actor_loop self.on_start() File "/work/nsteel/projects/mopidy-dev/mopidy/mopidy/mpd/actor.py", line 83, in on_start self.zeroconf_service.publish() File "/work/nsteel/projects/mopidy-dev/mopidy/mopidy/zeroconf.py", line 115, in publish self.domain, self.host, dbus.UInt16(self.port), TypeError: int() argument must be a string or a number, not 'NoneType'

In this particular case, does it make sense to advertise on zeroconf when it's a unix socket?

Yeah I suppose not. I'll add tests and remove the Zeroconf related stuff if using a unix socket.
Is there any notion of a config being one of two values? Eg.

schema['hostname'] = config.Hostname() or config.UnixSocket()

If not, should there be or should I just wrap both of these into the Hostname class like I have already?

I don't believe the above snippet is supported. We do want to validate a Unix socket path in the exact same way we do any other path, it'd be nice to reuse that validation code. Ultimately, unix sockets are probably valid everywhere that a hostname would be so I'd say wrapping is ok. Any thoughts @adamcik, @jodal?

So I tried out the following for Hostname.deserialize to try and reuse the existing Path validation code. What do you think?

def deserialize(self, value, display=False): validators.validate_required(value, self._required) if not value.strip(): return None socket_path = path.get_unix_socket_path(value) if socket_path is not None: return Path(not self._required).deserialize(socket_path) try: socket.getaddrinfo(value, None) except socket.error: raise ValueError('must be a resolveable hostname or valid IP') return value

I also moved that regex code to a path.get_unix_socket_path() helper since it occurs in the internal.network code too.

Yeah okay that's much more modular. So just the regex match and group extraction is contained in that helper method? Then delegating path expansion and validation to the Path class?

kingosticks · 2017-07-04T15:27:23Z

mopidy/mpd/session.py

@@ -25,7 +27,13 @@ def __init__(self, connection, config=None, core=None, uri_map=None):
            session=self, config=config, core=core, uri_map=uri_map)

    def on_start(self):
-        logger.info('New MPD connection from [%s]:%s', self.host, self.port)
+        if self.connection.sock.type == socket.AF_UNIX:


As above. Also applies to on_line_received too (which has not been handled).

kingosticks · 2017-07-04T15:32:57Z

mopidy/mpd/actor.py

@@ -60,7 +62,15 @@ def _setup_server(self, config, core):
                'MPD server startup failed: %s' %
                encoding.locale_decode(error))

-        logger.info('MPD server running at [%s]:%s', self.hostname, self.port)
+        if server.server_socket.type == socket.AF_UNIX:
+            logger.info('MPD server running at %s', self.hostname)


Maybe we could have a helper method to format the string for us since it occurs in a few places?

I was thinking that but wanted to get the rest of this out for review as early as possible. I'll add a method for printing this sort of thing in a DRYer fashion

kingosticks · 2017-07-04T15:36:53Z

docs/ext/mpd.rst

@@ -64,6 +64,7 @@ See :ref:`config` for general help on configuring Mopidy.
 .. confval:: mpd/hostname

    Which address the MPD server should bind to.
+    This can be a network address or a file path.


Should be clearer that by file path we are talking about a UNIX domain socket.

Sure, will do! To save another possible commit loop, how does this sound?
This can be a network address or the path to a Unix socket. Paths are expanded as in the Core configuration.

I'd be happy with just the first sentence.

kingosticks · 2017-07-04T15:38:23Z

docs/ext/mpd.rst

@@ -73,6 +74,8 @@ See :ref:`config` for general help on configuring Mopidy.
        Listens on all IPv4 interfaces
    ``::``
        Listens on all interfaces, both IPv4 and IPv6
+    ``~/.mopidy/socket``
+        Listen on the socket in your configuration directory


Maybe something more generic here? For example: /path/to/unix/socket.sock ? And adjust the wording appropriately.

No worries, how about:
Listen on the Unix socket at the specified path

Sounds good.

kingosticks · 2017-07-04T15:52:01Z

tests/internal/network/test_server.py

        sock.setblocking.assert_called_once_with(False)
-        sock.bind.assert_called_once_with((sentinel.host, sentinel.port))
+        sock.bind.assert_called_once_with((str(sentinel.host), sentinel.port))


Could probably do with a version of this for the unix case also since bind() is in both code paths.

kingosticks · 2017-07-04T16:51:37Z

Since we are dealing with paths there might also be some unicode/bytes issues here, I don't recall what we are supposed to be doing w.r.t that these days.

jradtilbrook · 2017-07-04T13:34:00Z

mopidy/config/types.py

@@ -258,6 +258,8 @@ def deserialize(self, value, display=False):
        validators.validate_required(value, self._required)
        if not value.strip():
            return None
+        if re.search('/', value):
+            return path.expand_path(value)


I was tossing up whether to add this here or to create another config type such as Socket or HostnameOrPath to encompass both Hostname config values as well as Paths. Let me know what you think

Also, what do you think about requiring Unix domain sockets to start with 'unix:' like apache and nginx configs? This would remove the hard requirement for a slash when you are specifying foo.sock in the current working directory whilst still being unambiguous.

That sounds good I'll do that

jradtilbrook · 2017-07-05T08:00:21Z

docs/ext/mpd.rst

@@ -64,6 +64,7 @@ See :ref:`config` for general help on configuring Mopidy.
 .. confval:: mpd/hostname

    Which address the MPD server should bind to.
+    This can be a network address or a file path.


Sure, will do! To save another possible commit loop, how does this sound?
This can be a network address or the path to a Unix socket. Paths are expanded as in the Core configuration.

jradtilbrook · 2017-07-05T08:00:45Z

docs/ext/mpd.rst

@@ -73,6 +74,8 @@ See :ref:`config` for general help on configuring Mopidy.
        Listens on all IPv4 interfaces
    ``::``
        Listens on all interfaces, both IPv4 and IPv6
+    ``~/.mopidy/socket``
+        Listen on the socket in your configuration directory


No worries, how about:
Listen on the Unix socket at the specified path

jradtilbrook · 2017-08-19T12:52:10Z

So wow, it took me a long time but I've finally gotten time to get back to this and implement the changes/fixes. Hopefully it looks all good now. Let me know what you think

jradtilbrook · 2017-10-20T02:35:11Z

I believe I have made all the requested changes. I have also been running this for the last few months without any issues.
Any chance of getting this merged?

kingosticks · 2017-10-20T12:11:19Z

Sorry I didn't get back to you on this but as chance would have it I actually started looking at it again just last night. I have a few thoughts on the server code following some debugging I did getting Mopidy (nearly) running on Windows, since socket.AF_UNIX is not defined on that platform. I also experimented with using the Path type within the Hostname validation which I think might be worthwhile. That work is on my home computer, I'll try to share it finish the review later.

kingosticks

Sorry for the delay!

kingosticks · 2017-10-22T15:18:34Z

mopidy/internal/network.py

+    return socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+
+
+def format_socket_connection_string(sock):


Nitpicking here, but you could call this something like format_socket_name so it matches the other similar one below.

kingosticks · 2017-10-22T15:34:28Z

mopidy/config/types.py

@@ -258,6 +258,9 @@ def deserialize(self, value, display=False):
        validators.validate_required(value, self._required)
        if not value.strip():
            return None
+        socket_path = path.get_unix_socket_path(value)
+        if socket_path:


We should to do this if socket_path is not None so that anything starting with 'unix:' gets the Path treatment, even if it's an empty path (that way the user sees a more sensible error message).

Fair enough. In my defence, I was trying to be more pythonic but you argument makes more sense

kingosticks · 2017-10-22T15:50:39Z

mopidy/internal/path.py

@@ -47,6 +48,13 @@ def get_or_create_file(file_path, mkdir=True, content=None):
    return file_path


+def get_unix_socket_path(socket_path):
+    match = re.search('^unix:(.*)', socket_path)
+    if match:


Again, nitpicking but maybe we should rejig this so the None case is a bit more explicit?

if not match: return None: return match.group(1)

kingosticks · 2017-10-22T15:54:26Z

mopidy/config/types.py

@@ -258,6 +258,9 @@ def deserialize(self, value, display=False):
        validators.validate_required(value, self._required)
        if not value.strip():
            return None
+        socket_path = path.get_unix_socket_path(value)
+        if socket_path:
+            return Path(not self._required).deserialize(socket_path)


This loses the 'unix:' prefix so the only way to identify unix socket paths after this point is to check for isinstance(value, ExpandedPath). Alternatively you could just return 'unix:' + Path(...). I think both are reasonable but maybe the latter is more bulletproof?

kingosticks · 2017-10-22T15:55:19Z

mopidy/internal/network.py


    def create_server_socket(self, host, port):
-        sock = create_socket()
+        socket_path = path.get_unix_socket_path(host)
+        if socket_path:  # host is a path so use unix socket


is not None

kingosticks · 2017-10-22T16:42:28Z

mopidy/internal/network.py

@@ -140,7 +179,11 @@ class Connection(object):
    def __init__(self, protocol, protocol_kwargs, sock, addr, timeout):
        sock.setblocking(False)

-        self.host, self.port = addr[:2]  # IPv6 has larger addr
+        if (sock.family == socket.AF_UNIX):


We could avoid bothering Connection with socket types if we handle this in Server.accept_connection(). Something like:

def accept_connection(self): try: sock, addr = self.server_socket.accept() if is_unix_socket(sock): addr = (sock.getsockname(), None) return sock, addr

is_unix_socket is helper where we could also check that socket has the AF_UNIX attribute (i.e. not on Windows).

kingosticks · 2017-10-22T17:39:23Z

mopidy/mpd/session.py

-        logger.info('New MPD connection from [%s]:%s', self.host, self.port)
+        logger.info(
+            'New MPD connection from %s',
+            network.format_socket_connection_string(self.connection.sock))


What about defining a Connection.__str__ function all these are bit prettier and we don't expose the socket?

kingosticks · 2017-10-22T17:51:14Z

mopidy/mpd/actor.py


    def on_start(self):
-        if self.zeroconf_name:
+        if (self.zeroconf_name and
+                self.server.server_socket.family != socket.AF_UNIX):


I'm not keen on exposing low level socket stuff here. What if we just check if self.port is None? We can call something in the constructor to set that based on the hostname e.g.

self.hostname, self.port = network.get_server_address(config['mpd']['hostname']), config['mpd']['port'])

What about using the is_unix_socket helper from above? That makes more sense to me rather than checking that the port is None

Not sure. I was thinking in the general case of if you don't have a port you don't want to advertise. But perhaps that isn't true? I'm. Not sure if zeroconf works outside of normal socket connections.

Github seems to have swallowed this comment up. It's not a big deal either way.

kingosticks · 2017-10-22T18:17:51Z

tests/mpd/test_actor.py

@@ -42,3 +42,38 @@ def test_idle_hooked_up_correctly(event, expected):
        assert not send_mock.call_args
    else:
        send_mock.assert_called_once_with(mock.ANY, expected)
+
+
+@pytest.mark.parametrize("event,expected", [


If we really want this coverage you can just add extra hostname and port parameters to the previous test rather than duplicate all this code. Is this all just to test that making port optional is OK? Perhaps something more targeted would be better. And since zeroconf is None here this doesn't actually test the path that threw the exception in the first review.

Oh yes I forgot about this one. Do you mean adding the hostname to the parameter decorator? So something like:

-@pytest.mark.parametrize("event,expected", [ - (['track_playback_paused', 'tl_track', 'time_position'], None), - (['track_playback_resumed', 'tl_track', 'time_position'], None), - (['track_playback_started', 'tl_track'], None), - (['track_playback_ended', 'tl_track', 'time_position'], None), - (['playback_state_changed', 'old_state', 'new_state'], 'player'), - (['tracklist_changed'], 'playlist'), - (['playlists_loaded'], 'stored_playlist'), - (['playlist_changed', 'playlist'], 'stored_playlist'), - (['playlist_deleted', 'uri'], 'stored_playlist'), - (['options_changed'], 'options'), - (['volume_changed', 'volume'], 'mixer'), - (['mute_changed', 'mute'], 'output'), - (['seeked', 'time_position'], 'player'), - (['stream_title_changed', 'title'], 'playlist'), +@pytest.mark.parametrize("event,expected,hostname", [ + (['track_playback_paused', 'tl_track', 'time_position'], None, 'foobar'), + (['track_playback_resumed', 'tl_track', 'time_position'], None, 'foobar'), + (['track_playback_started', 'tl_track'], None, 'foobar'), + (['track_playback_ended', 'tl_track', 'time_position'], None, 'foobar'), + (['playback_state_changed', 'old_state', 'new_state'], 'player', 'foobar'), + (['tracklist_changed'], 'playlist', 'foobar'), + (['playlists_loaded'], 'stored_playlist', 'foobar'), + (['playlist_changed', 'playlist'], 'stored_playlist', 'foobar'), + (['playlist_deleted', 'uri'], 'stored_playlist', 'foobar'), + (['options_changed'], 'options', 'foobar'), + (['volume_changed', 'volume'], 'mixer', 'foobar'), + (['mute_changed', 'mute'], 'output', 'foobar'), + (['seeked', 'time_position'], 'player', 'foobar'), + (['stream_title_changed', 'title'], 'playlist', 'foobar'), + + (['track_playback_paused', 'tl_track', 'time_position'], None, + 'unix:foobar'), + (['track_playback_resumed', 'tl_track', 'time_position'], None, + 'unix:foobar'), + (['track_playback_started', 'tl_track'], None, 'unix:foobar'), + (['track_playback_ended', 'tl_track', 'time_position'], None, + 'unix:foobar'), + (['playback_state_changed', 'old_state', 'new_state'], 'player', + 'unix:foobar'), + (['tracklist_changed'], 'playlist', 'unix:foobar'), + (['playlists_loaded'], 'stored_playlist', 'unix:foobar'), + (['playlist_changed', 'playlist'], 'stored_playlist', 'unix:foobar'), + (['playlist_deleted', 'uri'], 'stored_playlist', 'unix:foobar'), + (['options_changed'], 'options', 'unix:foobar'), + (['volume_changed', 'volume'], 'mixer', 'unix:foobar'), + (['mute_changed', 'mute'], 'output', 'unix:foobar'), + (['seeked', 'time_position'], 'player', 'unix:foobar'), + (['stream_title_changed', 'title'], 'playlist', 'unix:foobar'), ]) -def test_idle_hooked_up_correctly(event, expected): - config = {'mpd': {'hostname': 'foobar', +def test_idle_hooked_up_correctly(event, expected, hostname): + config = {'mpd': {'hostname': hostname,

kingosticks · 2017-10-22T18:21:36Z

tests/internal/test_path.py

+    def test_correctly_no_match_socket_path(self):
+        self.assertIsNone(
+            path.get_unix_socket_path('127.0.0.1'),
+            None


assertIsNone takes one argument. Doesn't it?!

jradtilbrook · 2017-11-04T11:51:57Z

I think I've covered all your requested changes now. Whenever you have time again, let me know what you think.
In the meantime, the tests are failing because a new version of flake8 with E722 do not use bare except has arisen. I think its out of scope for the PR but I can open another to fix the error, then rebase/merge that in once its resolved

kingosticks

If you can sort my last comment this looks brilliant. Like those tests you've added too. Thanks for all this effort.

kingosticks · 2017-11-04T13:07:36Z

mopidy/internal/network.py

@@ -179,13 +189,13 @@ class Connection(object):
    def __init__(self, protocol, protocol_kwargs, sock, addr, timeout):
        sock.setblocking(False)

-        if (sock.family == socket.AF_UNIX):
+        if is_unix_socket(sock):


With the changes you just made to accept_connection you don't need this condition at all anymore.

Do you mean:

- if is_unix_socket(sock): - self.host = sock.getsockname() - self.port = None - else: - self.host, self.port = addr[:2] # IPv6 has larger addr + self.host, self.port = addr[:2] # IPv6 has larger addr

I think the extra verbosity of the current code makes more sense though. I'll wait to commit and push until you confirm you'd prefer it this way.

I personally don't think it needs to worry about this extra special case since it fits in the general case

Okay cool I'll push those changes

kingosticks · 2017-11-04T13:14:25Z

Oh and I guess that rebase too

Update log lines and clean up the socket file upon exit

The Hostname config type now supports a Unix socket path prefixed with `unix:`

Also add test to for optional port setting

Also add test for removing socket path on shutdown

Refactor duplicate test to better use parameterization. Also remove redundant unix socket check.

kingosticks

Sorry, couple more thoughts today.

kingosticks · 2017-11-05T12:11:23Z

tests/mpd/test_actor.py

-    (['mute_changed', 'mute'], 'output'),
-    (['seeked', 'time_position'], 'player'),
-    (['stream_title_changed', 'title'], 'playlist'),
+@pytest.mark.parametrize("event,expected,hostname", [


I apologise for my earlier comment on this, it should have been clearer and just said this probably that isn't worth having. The idle stuff has nothing to do with what type of socket we have and this test doesn't even create one. If adding tests here we should concentrate on testing the changes i.e. the frontend can still start under the new config condition (port now optional), checking if starting zeroconf is skipped and server.stop() gets called.

No worries. I have removed this test

kingosticks · 2017-11-05T12:29:07Z

mopidy/internal/network.py

+            sock.bind(socket_path)
+        else:
+            sock = create_tcp_socket()
+            sock.bind((host, port))


Now that Port is optional, you get an un-handled exception from this if you run with mopidy -o mpd/hostname=:: -o mpd/port=. This is obviously a mistake in the user's config but we still need to handle it. It's shame it can't be done in the config validation where it belongs without having Port be non-optional again (which you could technically do even though it doesn't always make sense anymore..). What do you think?

Sorry, I'm not following. Do you mean make the port setting non-optional again?

That would be one option, the alternative is to explicitly error if the port is not a number when socket is not Unix.

Yeah I think option two is better? Happy for this to raise an exception here if port is None/empty? I'm looking through all the exceptions and wondering which type to use though. Current thoughts are ValueError, ValidationError, or BackendError?

I opted for the helper method from validation.check_integer which in-turn throws a ValidationError. Let me know if there is a more preferable way

jradtilbrook · 2017-12-18T07:34:48Z

I think the failing tests are due to changes updates to the dependent pip packages. Have you considered adding version constraints to the dev-requirements.txt file to reduce the chance of this happening? I could probably fix them here, but it's a little out of scope

jradtilbrook · 2018-02-11T00:47:58Z

I've pushed changes that should fix the new flake8 violations but I'm still hesitant about fixing the other failing tests because its in code unrelated and untouched by this PR. Could you advise what to do? I'd like to get this PR off my pending list since it's been a long time coming 😉

kingosticks · 2018-02-11T17:00:51Z

Hey Yeh I've been meaning to sort it out. And yes let's get this done and dusted. I can't look at it today but I will do tomorrow evening.

kingosticks · 2018-02-12T22:08:33Z

tests/internal/network/test_server.py

@@ -19,17 +21,18 @@ def setUp(self):  # noqa: N802

    def test_init_calls_create_server_socket(self):
        network.Server.__init__(
-            self.mock, sentinel.host, sentinel.port, sentinel.protocol)
+            self.mock, sentinel.host, 1234, sentinel.protocol)


It's been so long.. was there a reason for replacing all the sentinel port values with 1234 in these tests?

Yeah for me too. I believe it was because I added integer checking here as part of making the port optional but ensuring its an integer when specified

I've pushed changes that use sentinel where possible. I think there are actually only three cases where it must be an integer for testing purposes

That makes sense, thanks. I guess having validation down in this code is a bit unusual for us, I don't think we have anywhere else where the config setting's requirement is conditional on another bit of our config. For another day, I think.

kingosticks · 2018-02-12T22:39:53Z

tests/mpd/protocol/__init__.py

@@ -25,6 +25,9 @@ def queue_send(self, data):
        lines = (line for line in data.split('\n') if line)
        self.response.extend(lines)

+    def getsockname(self):


I can't see where this is used, still needed?

Good point. Removed 😄

kingosticks · 2018-02-13T12:47:02Z

kingosticks requested changes Jul 4, 2017

View reviewed changes

jradtilbrook commented Jul 5, 2017

View reviewed changes

kingosticks reviewed Oct 22, 2017

View reviewed changes

kingosticks approved these changes Nov 4, 2017

View reviewed changes

jradtilbrook added 15 commits November 4, 2017 23:11

mpd: add functionality for unix domain socket

bbcb433

Update log lines and clean up the socket file upon exit

mpd: update tests for unix sockets

1e09828

fix flake violations

9b46a35

docs: Update docs for MPD unix sockets

af9d576

docs: minor changes relating to unix sockets

f92a7e2

mpd: refactor socket creation

fd8bc23

mpd: modify Hostname config type and add test

a491fc3

The Hostname config type now supports a Unix socket path prefixed with `unix:`

mpd: DRY up logging

fa2d245

fix more flake violations

15cda57

mpd: disable zeroconf if using unix socket

57affd4

Also add test to for optional port setting

mpd: DRY up unix socket related code

b9da25e

mpd: More verbose if statements and names

fe4bd7f

Also add test for removing socket path on shutdown

mpd: refactor socket connection string logging

3f33591

mpd: DRY up unix socket type checks

ec3d918

mpd: some cleaning up after review

6992cb8

Refactor duplicate test to better use parameterization. Also remove redundant unix socket check.

jradtilbrook force-pushed the fix/1531-add-unix-domain-socket branch from 17ddb88 to 6992cb8 Compare November 4, 2017 15:33

kingosticks reviewed Nov 6, 2017

View reviewed changes

jradtilbrook added 2 commits November 6, 2017 11:20

mpd: remove redundant test

ee473aa

mpd: explicitly raise exception with no/invalid port

dafb60d

flake8: fix new violations

5f8fef2

kingosticks reviewed Feb 12, 2018

View reviewed changes

tests: Use sentinel where possible and remove unused code

16e83b3

		return socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)


		def format_socket_connection_string(sock):

Fix/1531 add unix domain socket #1629

Fix/1531 add unix domain socket #1629

Conversation

jradtilbrook commented Jul 4, 2017 • edited Loading

kingosticks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jradtilbrook Jul 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kingosticks commented Jul 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jradtilbrook Jul 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jradtilbrook commented Aug 19, 2017

jradtilbrook commented Oct 20, 2017

kingosticks commented Oct 20, 2017

kingosticks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kingosticks Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kingosticks Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

kingosticks Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kingosticks Oct 22, 2017 • edited Loading

Choose a reason for hiding this comment

jradtilbrook commented Nov 4, 2017

kingosticks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kingosticks commented Nov 4, 2017

kingosticks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jradtilbrook commented Dec 18, 2017

jradtilbrook commented Feb 11, 2018

kingosticks commented Feb 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jradtilbrook commented Jul 4, 2017 •

edited

Loading

jradtilbrook Jul 5, 2017 •

edited

Loading

jradtilbrook Jul 5, 2017 •

edited

Loading

kingosticks Oct 22, 2017 •

edited

Loading

kingosticks Oct 22, 2017 •

edited

Loading

kingosticks Oct 22, 2017 •

edited

Loading

kingosticks Oct 22, 2017 •

edited

Loading