Prefer loading blocks near predicted future player position #6447

lhofhansl · 2017-09-20T06:21:34Z

Very simple and stupid player position prediction.

When a player is not (or slowly) moving, the logic is unchanged.
But if a player is moving fast (f.e. flying in fast mode) the server will prefer to send block some distance out in the direction the player is traveling.
This effect is capped at 50% of the visible range - that is farthest shift the player will allow.

This makes for a smoother experience. Human players tend to look into the direction they move, and then part will be rendered sooner. Capping it at 50% of the view range avoids rendering faraway "block-islands".

Note: Part of this effect existed already, but it was set to exactly one MAPBLOCK out in the player's moving direction regardless of the player's speed. This makes it speed dependent.

ghost · 2017-09-20T08:38:58Z

It's funny because I was using fast mode earlier and observing these islands you mention, and considered at the time that this was by design!

Ps. It should be noted that this only occurs when noclip mode is on, because otherwise the player stops at current mapblocks boundary before the next is loaded.

paramat · 2017-09-20T10:56:32Z

Addresses #1442
Will help with falling at high speed too?

paramat · 2017-09-20T11:04:05Z

src/clientiface.cpp

-	// Predict to next block
-	v3f playerpos_predicted = playerpos + playerspeeddir*MAP_BLOCKSIZE*BS;
+	f32 playerspeedlen = playerspeed.getLength();
+	if(playerspeedlen > 1.0*BS)


Spaces around '*'

Space after 'if'

paramat · 2017-09-20T11:04:31Z

src/clientiface.cpp

+
+	// Predict where player will be soon, load blocks around there first
+	v3f playerpos_predicted = playerpos + playerspeeddir *
+		MYMIN(playerspeedlen * BS, d_blocks_in_sight*0.5f);


Spaces around '*'

'd_blocks_in_sight' is in units of nodes or mapblocks? needs to be multiplied by BS or MAP_BLOCKSIZE * BS?

So BS can then be factored out if present in all terms.

d_blocks_in_sight is already scaled correctly (in nodes, i.e. multiplied by MAP_BLOCKSIZE * BS), it is used later in the code (I just moved this code up).

paramat · 2017-09-20T11:05:05Z

src/clientiface.cpp

+	float camera_fov = sao->getFov();
+	// if FOV, wanted_range are not available (old client), fall back to old default
+	if (wanted_range <= 0) wanted_range = 1000;
+	if (camera_fov <= 0) camera_fov = (72.0*M_PI/180) * 4./3.;


Spaces around '*' and '/'

State all numbers as floats
0.0f
72.0f
180.0f
4.0f
3.0f

wanted_range = and camera_fov = belong each into a separate line.
EDIT: Even if this is only a code movement, it should be cleaned up when touching it.

Oh yeah, missed that, agreed.

paramat · 2017-09-20T11:12:39Z

Excellent idea and much wanted.
I feel this should be

+	v3f playerpos_predicted = playerpos + playerspeeddir *
+		rangelim(playerspeedlen * BS, MAP_BLOCKSIZE * BS, d_blocks_in_sight * 0.5f);

otherwise for all speeds < 16 nodes per second (almost all speeds) this PR is reducing the predictive block loading. Also, since block loading acts in blocks there is no point in using a prediction distance smaller than 1 mapblock, because that would often result in no predictive action at all.

nerzhul · 2017-09-20T21:26:21Z

src/clientiface.cpp

+	s16 wanted_range = sao->getWantedRange();
+	float camera_fov = sao->getFov();
+	// if FOV, wanted_range are not available (old client), fall back to old default
+	if (wanted_range <= 0) wanted_range = 1000;


don't inline conditions

nerzhul · 2017-09-20T21:26:44Z

src/clientiface.cpp

+	float camera_fov = sao->getFov();
+	// if FOV, wanted_range are not available (old client), fall back to old default
+	if (wanted_range <= 0) wanted_range = 1000;
+	if (camera_fov <= 0) camera_fov = (72.0*M_PI/180) * 4./3.;


compare float with floats and use the .f marker to prevent using doubles

nerzhul · 2017-09-20T21:26:58Z

src/clientiface.cpp

+	if (wanted_range <= 0) wanted_range = 1000;
+	if (camera_fov <= 0) camera_fov = (72.0*M_PI/180) * 4./3.;
+
+	const s16 full_d_max = MYMIN(g_settings->getS16("max_block_send_distance"), wanted_range);


std::min instead of MYMIN

BluebirdGreycoat · 2017-09-20T22:26:08Z

I hope this makes it into the engine, it will help with the issue mentioned in #1442. It's very timely also, considering that I have just discovered that my players are taking advantage of this when dropping hundreds of blocks from cavern ceilings. (They point the camera up and are able to reach the floor without dying.)

lhofhansl · 2017-09-20T22:59:10Z

Couple of comments:

It will help with falling, as long as the client reports the player's speed correctly.
most of the code nits are on code I copied around. Will fix.

lhofhansl · 2017-09-21T00:06:23Z

Not sure it's going to help with players not dying when falling from great height.

lhofhansl · 2017-09-21T02:26:32Z

Update addressing review comments.

lhofhansl · 2017-09-21T02:33:30Z

I think there's more that can be done. For example at high velocity we could reduce the fov of the retrieval cone. The effect would be that server and client would be able to better keep generating and retrieving blocks along the axis of movements, and not load the sides (which would soon not be visible anyway - we're moving).

In fact, maybe that's a better solution altogether...? Not trying to predict the future position, but instead predict a good fov, so that blocks in the smaller cone are loaded faster. I'll try that too.
The effect would be almost the same.

In any case, if someone could test what I have now, that'd be cool.

lhofhansl · 2017-09-21T03:45:12Z

Arrghhh... Lemme fix the other MYMINs as well. Not sure why I did not see them.

lhofhansl · 2017-09-21T04:07:28Z

Tried the fov adjustment, but it's no good since one does not have to look into the direction of the movement (can move forward, but look down, in which case this is weird).

lhofhansl · 2017-09-21T04:37:44Z

There's one thing I noticed: If we do not predict the player's position, the player will have moved on before some of the close by blocks could be retrieved. That means now with the prediction client and server will likely render more blocks. This is apparent when you "fly" around in some altitude and have a mod that loads blocks somewhat slowly.

There's no free lunch. :(

paramat · 2017-09-21T19:47:05Z

src/clientiface.cpp

+	s16 wanted_range = sao->getWantedRange();
+	float camera_fov = sao->getFov();
+
+	// if FOV, wanted_range are not available (old client), fall back to old default


We're breaking compat with old clients now, so old client support can be dropped.

paramat · 2017-09-21T19:50:29Z

Now you're using std::min(), #include <cmath> is needed.
Will test.
It won't fix falling issues at high fall speed but will help.

lhofhansl · 2017-09-21T21:37:59Z

Compiled and ran it, the compile should have failed with loaded from somewhere. Anyway it's nice to be included where it is needed.
On the backwards compatibility, I was thinking the same, but felt that should be a different PR to separate the concerns. If there's no concern I'll do it here.

lhofhansl · 2017-09-21T21:40:27Z

Tangentially related... How much do we generally want to trust the client?

lhofhansl · 2017-09-21T22:04:20Z

OK. Done.

Also reduce the effective blockrange sent in one roundtrip to 1 (instead of 2).
In steady that makes no difference (that I could tell), with fast movements it allows the server to keep up (when a client moves the center is reset, and send starts from the beginning again).
It also has the nice side-effect that the client tends to receive smaller sets of maplocks each time.

paramat · 2017-09-21T23:59:29Z

On the backwards compatibility, I was thinking the same, but felt that should be a different PR

No need, we're breaking compat in code inside each PR as we go.

Done.

nerzhul · 2017-09-22T08:48:15Z

src/clientiface.cpp

+	s16 wanted_range = sao->getWantedRange();
+	float camera_fov = sao->getFov();
+
+	const s16 full_d_max = std::min(g_settings->getS16("max_block_send_distance"), wanted_range);


g_settings->getS16("max_block_send_distance") and the other variable should be cached somewhere

This is part of the code that loads blocks, and sends them over the network, i.e. not hot. This seems overkill.

Also surely it being 'const' means it is as good as cached?

const means only we cannot modified it, its for dev, not for runtime

How about 'static const'? Since these settings are not changed often.
Maybe i misunderstand 'const' and 'static const' i thought these were only calculated once per game session or program session.

Sorry indeed these can't be static const since there is a min with wanted range.

Not sure if this is possible but i would cache the actual g_settings->get() values as private members of class RemoteClient and initialise them in the constructor?
They don't need to be refreshed because they are settings not often changed by a user, we actually have many settings that only take effect on a program restart, for performance reasons.

However it's not high priority so i might still +1 if you do not optimise, but i would prefer you do if it is possible.

I really do not think that is necessary. There are many more configs that we retrieve this way - not an excuse not to fix it, but in this code it is simply not needed and will not help with code readability.

We could have a general PR to think about caching of configs.

Ok i won't insist on it.

paramat · 2017-09-23T04:05:10Z

Tangentially related... How much do we generally want to trust the client?

As little as possible due to hacked clients, which are a big problem for server owners. Minetest's general approach is that the server is in control of as much as reasonably possible.

paramat · 2017-09-24T13:18:25Z

src/clientiface.cpp

+	// Predict where player will be soon, load blocks around there first
+	// We use the client-reported speed as an indicator, but do not move the
+	// center point further than 1/2 the visible range away
+	v3f playerpos_predicted = playerpos + playerspeeddir *


How about optimising this by moving this calculation into the condition above so that it only occurs when player speed > 1 * BS?

v3f playerpos_predicted = playerpos; if (playerspeedlen > 1.0f * BS) { v3f playerspeeddir = playerspeed / playerspeedlen; playerpos_predicted += playerspeeddir * rangelim(playerspeedlen * BS, MAP_BLOCKSIZE * BS, d_blocks_in_sight * 0.5f); }

? and delete line 104.

Optimizing is not needed here.
You mean for readability...? There I would agree.

I know it's not hot code, but we still optimise code everywhere as these small optimisations do add up, over time this helps to make MT more lightweight. So in my opinion there is no such thing as 'optimising is not needed here'.

Thanks for doing this.

lhofhansl · 2017-09-24T18:43:31Z

Is it possible for somebody else to test this and share the experience?

jmdevy · 2017-09-25T00:10:19Z

@lhofhansl, I tested it and it seems to be fine. I have made a video that includes me using fast move without the pull request and with the pull request. I cloned the newest version of the Minetest repository, so I'm using what will be 5.0: https://streamable.com/cu4uj If anyone sees this within the next few minutes, then streamable is still uploading.

The video timeline is as such: MV forward without PR -> Jump down hole without PR -> MV forward with PR -> Jump down hole with PR. There are some transition titles so you know what I'm doing:

Forgive my video editing skills (for some reason that robot voice won't stay out of my videos).

Moving with the pull request seems to be faster, meaning the terrain generates in front of me faster. In the video I get into terrain I did not generate yet, so it lags a little, just compare it to the first part of the first video. I'm using the default view distance of 100, and also take note that I'm not at the exact same height when moving forward in both parts of that video.

The best part about this though is that when falling straight down I don't hit the ignore nodes with your pull request. In the video, you can see me hit the ignore nodes when I do not have your pull request. When playing Minetest, hitting those invisible ignore nodes is frustrating and it makes the game seem buggier than it is; this will make Minetest seem so much more refined!

I'm very glad this is being solved and implemented. Someone should tell @Wuzzy2 to check this out.

Maybe we can close #1442.

paramat · 2017-09-28T13:48:04Z

Wuzzy2

Saying that you need for std::min is the kind of nonsense which must be called out immediately

I welcome my mistake being pointed ouot, but it's just a minor mistake and no big deal as you imply with 'the kind of nonsense which must be called out immediately'.

The somewhat harsh tone by kilbith was appropriate.

No the response was not at all appropriate, you can't justify it, i'll assume you're angry at me for some reason. It seems that when you make a minor mistake you want people to swear, write 'yet another mistake, you're supposed to be a contributor' :) I couldn't do that.

To be clear i do not want him removed from the organisation, he is/was a good contributor.

paramat · 2017-09-28T14:10:33Z

playerpos_predicted += playerspeed * 2.0f;

All are in Irrlicht units it seems, so flying at 20n/s prediction is 40 nodes ahead (2.5 mapblocks),
wanted range is increased by 20 nodes,
block-loading FOV is divided by 1.4. (* 0.7, roughly 2/3rds).

Falling at 100n/s, the predicted point will be 200 nodes below, wanted range is increased by 100 nodes, and block-loading FOV is divided by 3.

Seems reasonable.

When walking, predicted point 8 nodes ahead, so half the time there is no prediction, not a big issue.

It would be good to test this on an actual multiplayer server, i feel my singleplayer tests are not enough.
I do think this has potential and is welcome, i'll do another test. I'm close to +1 for this.
Players can learn to look exactly down when when falling to use this feature, but they will tend to be doing this anyway.

lhofhansl · 2017-09-28T16:23:00Z

There's some more tweaks I want to try:

Have the fov reduced at the client when the player moves quickly. This might look weird (like a warp effect) but I want to try it.
Increasing the wanted_range only when moving upon reflection seems not that helpful. I think instead it's better to simply always add 1 to what the client requests (so that one extra MAPBLOCK sliver beyond the visible range is retrieved). Since this is for the visible cone only that seems better.
I want to reduce the fov slightly more when moving fast. The amount of work that can be saved is staggering. If the fov is reduce by just 25% the work about 1/2.

I wonder whether I'm doing too many updates on this PR - showing the process and my thinking for discussion... Or whether it'd be better to just post a PR that I think is mostly finished...?

HybridDog · 2017-09-29T15:41:43Z

I wonder whether I'm doing too many updates on this PR - showing the process and my thinking for discussion

It's fine. Please also extend the first comment (PR description).

lhofhansl · 2017-09-29T18:57:01Z

FOV changes at client look weird. You move and suddenly you're zoomed in a bit. It does hide the loading artifact, but net it's worse I think.
I'm still experimenting with what's best. For example if you move backwards and look forward retrieving blocks behind you (as is done now) makes it worse (since the view cone is forward and those blocks would be filtered anyway). So now I'm also using the direction of the camera and direction of movement.

So hold off on testing this. Some more changes coming. :)

HybridDog · 2017-09-29T19:13:11Z

FOV adjustment based on speed was rejected in minetest (#385), it was merged in freeminer and disabled by default if memory serves.

When the player goes by cart or is attached to some other moving object, mapblocks in the direction of the velocity of the player don't need to be loaded earlier because the cart collision is calculated server-side, do they?

BluebirdGreycoat · 2017-09-29T19:35:24Z

Yes, they do. It kinda defeats the point of the PR if they aren't loaded (and sent to client) ahead of the player while riding a cart. Also, player is most frequently moving fast when riding some vehicle (if they aren't falling).

BluebirdGreycoat · 2017-09-29T19:39:04Z

@lhofhansl if I may suggest, the direction of player movement should always override the direction of the view cone, if they are contradictory and the player is moving faster than normal. Without this PR, riding a cart on a server with a slow network is annoying because the cart keeps flying off into the grey.

lhofhansl · 2017-09-29T20:09:43Z

@HybridDog Yeah, client side FOV adjustment looks weird, I tried (see last message) and came to the same conclusion. This PR does it server side fov adjustment, since the faster you move the faster blocks "fall out" on the sides and hence we'd better focus on loading more blocks in the viewing/moving direction.

@BluebirdGreycoat Not quite. The player can move forward and look up or down, or look forward and move backwards. In that case it is quite important to check for the viewing direction.
Or are you saying we should change the logic that in that case the server won't perform the cone check? That'd be an option I hadn't considered, yet. Hard to know how many blocks to load behind, though; i.e. hard to limit the blocks to be loaded at that point. (This PR at least tries to not uselessly move the predicted position to somewhere where no blocks would be loaded anyway)

BluebirdGreycoat · 2017-09-29T20:18:34Z

Yes, I'm thinking that if the player is moving sufficiently fast then the viewing cone should be ignored. Otherwise, a falling player could prevent blocks below them from loading simply by looking upwards, which is the current situation anyway.

I also think that the speed at which the viewing cone should be ignored should be configurable per-server, since no set speed is going to work for all servers. For example, a server could have carts go at a speed just lower than 'fast' and thus might want to lower the limit for ignoring the viewing cone.

Edit: the predicted future position doesn't need to move very far for this to work. It only needs to move ahead the distance of the server's configured block send range, say.

lhofhansl · 2017-09-29T23:27:14Z

Looked at #385. Note that this is the opposite! I am decreasing the FOV (server side only, so no client jump) when moving forward, since blocks on the side will disappear quickly anyway.

lhofhansl · 2017-09-29T23:49:02Z

Not enforcing the view-cone is tricky. Which blocks should I retrieve from behind the player? All? That'd be a lot of blocks potentially. Could try to move shift the view-cone. Lemme think on that.

BluebirdGreycoat · 2017-09-29T23:52:52Z

A simple if {} can fix that. If player is falling but view cone is pointed elsewhere, just retrieve blocks directly below the player (prioritize them above the blocks the player is looking at). If player is moving sideways but looking elsewhere, same thing, just in the X and Z directions. After all if the player isn't looking in the direction they're moving, the reason we'd be retrieving blocks is to allow player movement to continue uninterrupted as much as possible, so there'd be no point in retrieving more blocks than that.

Edit: 90% of the time, player is looking in same direction they move so I'm just proposing doing something different for the 10% of the time they spend falling (or sitting in a cart and looking back).

paramat · 2017-09-30T02:26:06Z

@Wuzzy2 you're the only person who has consistently thumbed-up the abuse in this thread, i'm surprised and disappointed.

paramat · 2017-09-30T05:37:13Z

Have the fov reduced at the client when the player moves quickly.

I really don't like this effect, unfortunately because MC uses it (the opposite infact) many think it's a good idea, it's visually disorientating and nauseating.

Increasing the wanted_range only when moving upon reflection seems not that helpful. I think instead it's better to simply always add 1 to what the client requests

Good idea.

I want to reduce the fov slightly more when moving fast

The block-loading FOV? Yeah sounds good. 20n/s is a fairly universal top speed for players or admin, reducing block FOV to half in this situation may be acceptable.
Top speed in MT is unlimited if falling a long way (like several kilonodes in a space or floatland realm), so i think the block FOV needs a lower limit for this case.

if I may suggest, the direction of player movement should always override the direction of the view cone, if they are contradictory and the player is moving faster than normal.

I have been thinking about this too, currently falling damage can be reduced by simply looking upwards to cause many unloaded block collisions (see the falling issue).
It seems that loading in the direction of motion should have priority above a certain speed (above walking speed perhaps?). If the player needs to load something in that is not in direction of motion they can always stop for a moment, and players usually look in the direction of motion.
I think it's worth trying and considering.

This is interesting and appreciated work, i will test what you come up with.

HybridDog · 2017-09-30T08:02:20Z

BluebirdGreycoat, it doesn't make sense to me, why does the player have to have the mapblock that the cart can drive there although real object movement is done on the server?

BluebirdGreycoat · 2017-09-30T13:23:09Z

@HybridDog there's too distinct cases here. In the case of falling the bottleneck is that the server does not send blocks fast enough to the client (falling is controlled by client). In the case of a (reasonably fast) cart, the server does not load blocks fast enough; this is evident even on fast servers if there are many players. In this case you are right, the client is not the issue, because it is the server that does the movement calculation. But it has to load the blocks in order to do the movement calculation correctly, otherwise the cart hits the grey and keeps going -- in my experience, up to 100 blocks before the server realizes that the rail changed direction long before then. I suspect the reason for this particular problem is that the server is too busy trying to send blocks to the client (which has gone off into the grey) instead of prioritizing the loading of blocks where the rail actually is. Hope this makes sense, I am unsure how to condense it.

I just had a cool idea: I can modify the cart mod on my server to preload the rail ahead of the player. Even if there's no API yet to instruct the server to send specific blocks to the client, in this case it won't matter, as long as the rail is loaded server-side. Fun!

lhofhansl · 2017-09-30T16:24:32Z

OK... What I tried was this:
AS we calculate the cone of blocks to retrieve, if the speed is above some threshold use the direction of the velocity as direction of the cone, other continue to the camera direction. As a test I used 100 as the threshold. That's seems to be doing pretty well. Lemme file another PR for this.
Lemme also file a PR to cache all those configs, since I am going to add a new one.

BUT... It turns out that when a player sits in a cart (or a boat) the speed is not reported correctly from the client. Seems the client says the player's speed is 0, since the cart is moving, not the player.

AND... Estimating the actual speed at the server is unreliable, even when I put some dampening on the calculation.

lhofhansl · 2017-10-01T05:31:32Z

@proller your comment got lost in the discussion. I looked the code you pointed out, but I do not quite get it. Any chance you can explain?

HybridDog · 2017-10-02T10:25:14Z

I guess you can get the velocity of the object the player is attached to.

paramat · 2017-10-02T16:15:57Z

Proller's code:

		// Fast fall/move optimize. speed_in_blocks now limited to 6.4
		if (speed_in_blocks>0.8 && d <= 2) {
			can_skip = false;
			if (d == 0) {
				for(s16 addn = 0; addn < (speed_in_blocks+1)*2; ++addn)
					list.push_back(floatToInt(playerspeeddir*addn, 1));
			} else if (d == 1) {
				for(s16 addn = 0; addn < (speed_in_blocks+1)*1.5; ++addn) {
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( 0,  0,  1)); // back
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( -1, 0,  0)); // left
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( 1,  0,  0)); // right
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( 0,  0, -1)); // front
				}
			} else if (d == 2) {
				for(s16 addn = 0; addn < (speed_in_blocks+1)*1.5; ++addn) {
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( -1, 0,  1)); // back left
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( 1,  0,  1)); // left right
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( -1, 0, -1)); // right left
					list.push_back(floatToInt(playerspeeddir*addn, 1) + v3POS( 1,  0, -1)); // front right
				}
			}

Looks like it does similar to your other PR but in a more hacky way. For large player speed and for d <= 2 it loads in a narrow cylinder of blocks in the speed direction.

lhofhansl · 2017-10-02T18:28:09Z

@HybridDog how reliable is this? Who keeps track of the speed in this case? It looks like the speed reported by the client is more of a "desired speed", i.e. the speed requested by the user, not the actual speed.

What I have here will always be correct since the server calculates the speed based on the actual succession of positions.

HybridDog · 2017-10-03T06:26:22Z

The velocity, position and acceleration of objects other than the player are defined server side. The client only extrapolates the velocity and position to avoid stuttering.

lhofhansl · 2017-10-20T04:51:56Z

I'm going to revive this, now that my other PRs in this area are in.

lhofhansl · 2017-10-22T02:49:43Z

I'm going to close this one. Some of this is in #6483 (merged), some in #6552 .

lhofhansl · 2018-07-11T08:57:28Z

I'd like to revive this. The mistake I made earlier was to not include dtime in the calculation.
The right prediction would be where the player would be the next time we call getNextBlocks. The only wrinkle is that dtime is kinda jumpy, so it'd need to smoothed a bit with a simple running, weighted, average.
I'll do a PR.

SmallJoker · 2018-07-11T09:00:44Z

You could also revive this PR and apply your new changes to the already existing branch. However, if you prefer starting with an empty discussion again, it's also possible to open a 2nd PR (also for the same branch name IIRC).

paramat added @ Server / Client / Env. Performance labels Sep 20, 2017

paramat reviewed Sep 20, 2017

View reviewed changes

paramat mentioned this pull request Sep 20, 2017

Colliding with unloaded blocks at high velocities is problematic #1442

Closed

nerzhul previously requested changes Sep 20, 2017

View reviewed changes

paramat reviewed Sep 21, 2017

View reviewed changes

nerzhul reviewed Sep 22, 2017

View reviewed changes

paramat reviewed Sep 24, 2017

View reviewed changes

lhofhansl mentioned this pull request Oct 1, 2017

Retrieve extra speed-directed, narrow view cone to load blocks the player will encounter soon. #6483

Closed

lhofhansl mentioned this pull request Oct 22, 2017

Reduce server FOV with forward speed. #6552

Closed

lhofhansl closed this Oct 22, 2017

Prefer loading blocks near predicted future player position #6447

Prefer loading blocks near predicted future player position #6447

Conversation

lhofhansl commented Sep 20, 2017

ghost commented Sep 20, 2017 • edited by ghost

paramat commented Sep 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SmallJoker Sep 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paramat commented Sep 20, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BluebirdGreycoat commented Sep 20, 2017

lhofhansl commented Sep 20, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

paramat Sep 21, 2017 • edited

Choose a reason for hiding this comment

paramat commented Sep 21, 2017 • edited

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

lhofhansl commented Sep 21, 2017

paramat commented Sep 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paramat Sep 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paramat Sep 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paramat commented Sep 23, 2017

paramat Sep 24, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paramat Sep 25, 2017 • edited

Choose a reason for hiding this comment

lhofhansl commented Sep 24, 2017

jmdevy commented Sep 25, 2017 • edited

paramat commented Sep 28, 2017 • edited

paramat commented Sep 28, 2017 • edited

lhofhansl commented Sep 28, 2017

HybridDog commented Sep 29, 2017

lhofhansl commented Sep 29, 2017

HybridDog commented Sep 29, 2017

BluebirdGreycoat commented Sep 29, 2017

BluebirdGreycoat commented Sep 29, 2017

lhofhansl commented Sep 29, 2017

BluebirdGreycoat commented Sep 29, 2017 • edited

lhofhansl commented Sep 29, 2017

lhofhansl commented Sep 29, 2017

BluebirdGreycoat commented Sep 29, 2017 • edited

paramat commented Sep 30, 2017 • edited

paramat commented Sep 30, 2017 • edited

HybridDog commented Sep 30, 2017

BluebirdGreycoat commented Sep 30, 2017

lhofhansl commented Sep 30, 2017 • edited

lhofhansl commented Oct 1, 2017

HybridDog commented Oct 2, 2017

paramat commented Oct 2, 2017 • edited

lhofhansl commented Oct 2, 2017

ghost commented Sep 20, 2017 •

edited by ghost

paramat commented Sep 20, 2017 •

edited

SmallJoker Sep 20, 2017 •

edited

paramat commented Sep 20, 2017 •

edited

paramat Sep 21, 2017 •

edited

paramat commented Sep 21, 2017 •

edited

paramat Sep 23, 2017 •

edited

paramat Sep 24, 2017 •

edited

paramat Sep 24, 2017 •

edited

paramat Sep 25, 2017 •

edited

jmdevy commented Sep 25, 2017 •

edited

paramat commented Sep 28, 2017 •

edited

paramat commented Sep 28, 2017 •

edited

BluebirdGreycoat commented Sep 29, 2017 •

edited

BluebirdGreycoat commented Sep 29, 2017 •

edited

paramat commented Sep 30, 2017 •

edited

paramat commented Sep 30, 2017 •

edited

lhofhansl commented Sep 30, 2017 •

edited

paramat commented Oct 2, 2017 •

edited

lhofhansl commented Oct 22, 2017 •

edited