Chanfitness: Track peer uptime #3332

carlaKC · 2019-07-22T15:02:01Z

This PR introduces a new package chanfitness which is used to track scores for a node's channels.
Scores are currently kept in memory, with the intention of persisting them once the set of metrics needed to score nodes is more clearly defined. It is related to #1253, although the issue is quite old.

This change implements tracking for peer uptime, by maintaining a log of channel events for each channel. Peer online/offline events are monitored by a goroutine on a per peer basis, and channel creation/close events are monitored by a single goroutine.

This change also includes adding a channel fitness rpc subserver, but that can be split out into another PR if the change is too big.

PR Checklist

All changes are Go version 1.12 compliant (used go1.12.6)
Go fmt, commented and wrapped at 80 lines
Make check, lint and go vet ok, builds properly!
Code accompanied by tests

halseth

Very impressed by this first iteration of the package! The design can be streamlined a little bit, but all in all I think this has taken the right direction.

First of all, this size of this PR is quite large, so I would suggest breaking it into smaller parts. It seems natural to let the pure event tracking be its own PR, then we can follow up with PRs to get scores, exposing them on the RPC etc.

Secondly with the above comment in mind, I think we should attempt to consolidate monitoring of channel events and peer activity. The first step here would be to add a SubscribePeerEvents API similar to what already exists for channels.

chanfitness/chanfitness.go

halseth · 2019-07-24T07:12:28Z

chanfitness/chanfitness.go

+	"github.com/lightningnetwork/lnd/subscribe"
+)
+
+// ScoreStore maintains a set of scores for the node's channels. It is intended


Instead of the notion of this storing a "score", should we change it to store "events". The score can later be calculated on the basis of the events. We can name this ChannelEventStore?

chanfitness/chanfitness.go

halseth · 2019-07-24T07:14:46Z

chanfitness/chanfitness.go

+		score = &ChannelScore{
+			id:        channelID,
+			peer:      peer,
+			quit:      make(chan struct{}, 1),


take a look at how quit channels are defined and used other places in the codebase :)

chanfitness/chanfitness.go

chanfitness/chanscore.go

carlaKC · 2019-07-25T14:48:57Z

Thanks for the review @halseth 🙏
Agreed on monitoring all peers rather than peers on a per channel basis, makes a lot more sense.

What do you think about adding a peernotifier package that provides a subscribe.Client like channotifier currently does? That way, chanfitness can subscribe to peer events in the same way that it does channels?

With that approach, I'd break this (entirely too big - sorry 😅) PR down into:

Add peernotifier package in new a PR
Reduce this PR to just consume channel events for now (+ remove grpc)
Add peer monitoring to chanfitness
Add grpc subserver for querying chanfitnes
Sound ok?

halseth · 2019-07-29T10:19:25Z

What do you think about adding a peernotifier package that provides a subscribe.Client like channotifier currently does? That way, chanfitness can subscribe to peer events in the same way that it does channels?

👍

With that approach, I'd break this (entirely too big - sorry 😅) PR down into:

Add peernotifier package in new a PR

Reduce this PR to just consume channel events for now (+ remove grpc)

Add peer monitoring to chanfitness

Add grpc subserver for querying chanfitnes
Sound ok?

I would start by doing 1 and 2, then we can take a look at how big 2 becomes and what we could reasonably include there 😄

halseth · 2019-08-07T09:23:08Z

Needs rebase!

halseth

This is coming along nicely!

I think we could also incorporate peer event tracking in this PR, to see how it will interact with the channel event store. Alternatively a new PR that builds on this one that adds peer tracking can be created, to see how the final result will look.

chanfitness/chanevent.go

chanfitness/chanfitness.go

halseth · 2019-08-07T10:42:42Z

chanfitness/chanevent.go

+	// id is the uInt64 of the short channel ID.
+	id uint64
+
+	peer [33]byte


missing godoc

for pubkeys, there is also the route.Vertex type

No route.Vertex?

~~I don't think we need to add a dependency on routing, so would prefer keeping as is.~~ I'm fine with either approach.

~~Can we leave it as is for now and then I'll update PeerNotifier and this code in a separate PR?~~
Discussed, will change in this PR then update PeerNotifier separately.

chanfitness/chanevent.go

chanfitness/chanfitness.go

chanfitness/chanfitness_test.go

joostjager

I only did a high level pass.

With regard to scope: I prefer a pr to have some meaningful impact (personal opinion). Add functionality, fix a bug, speed up something, etc. In general I think it is best if it also makes the changes end to end, so that one can verify (run the code) that the goal of the pr is met.

This pr is mostly a horizontal slice with code that isn't being called into except for starting the main event loop. A technical layer. Imo that bears the risk that code get merged that needs to be revised later when doing the work in the higher level layers. It may take some creativity to define an end to end pr that is not too big, but it seems that in this new domain there must be some good opportunities.

Also, if the goal is to track channel open/closes on chain: it may be that that info is already available and doesn't need to be tracked. The channel point and the closing tx id are persisted in the database. Maybe the peer (tcp) up time metric would have been more suitable to kick off the fitness system.

chanfitness/chanevent.go

joostjager · 2019-08-07T12:42:22Z

log.go

@@ -118,6 +120,7 @@ func init() {
 	chanbackup.UseLogger(chbuLog)
 	monitoring.UseLogger(promLog)
 	wtclient.UseLogger(wtclLog)
+	chanfitness.UseLogger(chftLog)


You could choose to use the addSubLogger construction here and define the log prefix in the package itself.

chanfitness/chanevent.go

chanfitness/chanfitness.go

carlaKC · 2019-08-07T14:44:37Z

Thanks for taking a look @joostjager

In general I think it is best if it also makes the changes end to end, so that one can verify (run the code) that the goal of the pr is met.

Imo that bears the risk that code get merged that needs to be revised later when doing the work in the higher level layers. It may take some creativity to define an end to end pr that is not too big, but it seems that in this new domain there must be some good opportunities.

I definitely agree on having a more horizontal PR in terms of being able to test the changes and figuring out how it will be used by higher levels. My original pass at this PR had a rpcsubserver for getting scores included, but that ended up being a pretty huge PR so I removed it. My thoughts on expanding this PR to be more than just kicking off a logical loop:

Re-add a RPC server so the endpoint can be called, although I don't feel like this properly covers what you mean by figuring out how it will be used on higher levels?
Add some opt-in channel closing behaviour to autopilot. Since we're only going to have uptime metrics available (see below), I'd say that this would have to be done with some minimum channel lifetime param (eg must be at least x days old) and threshold flap rate which is not acceptable so that we don't start maniacally closing channels. I'll give that a go and see how it looks, might end up being too much for one change.

Also, if the goal is to track channel open/closes on chain: it may be that that info is already available and doesn't need to be tracked. The channel point and the closing tx id are persisted in the database. Maybe the peer (tcp) up time metric would have been more suitable to kick off the fitness system.

This PR has been waiting on #3354 (which was originally included in my way too big first attempt) so that uptime can be added to the events store - that is the main goal for this PR. When peer events were split out, I updated this to just have open/close events to get an idea of what the structure would look like. I'm going to add online/offline events now that #3354 is merged - I should have made that clearer, sorry!

In terms of having open/ close events at all, I think it still makes sense to have them while events are being kept in memory? That way the package doesn't have to touch the DB at all for now (and does not need to query it every time a score is requested). If/when we start looking at persisting score data, we can probably just get channel open/close events from chanDB.

joostjager · 2019-08-07T15:09:52Z

I actually meant creating a vertical PR, cutting through several layers. Ok, horizontal, vertical, it isn't very self-describing, but I think we mean the same thing.

I would at least choose either channel open/close or peer online/offline, not both, for this pr. If you choose peer online/offline, maybe you can just expose a single value "average peer uptime %" over the rpc. Or even more minimalistic, just log the average peer uptime value periodically. Then at least, some business value is created and users can see that it works.

For channel open/close: I don't know if it is a requirement to not touch the database for now. If you think you'll query chandb later, it may not be worth to write the code to send channel events now and remove that again later. In general, with storing redundant information things may get out of sync.

halseth · 2019-08-07T15:44:40Z

I think only doing peer uptime for now sounds like a good middle ground. That was what this PR originally set out to do, and would provide information we don't currently have anywhere.

The question is whether we can get away with channel events completely. Since a peer's uptime might not be very interesting if we don't have an open channel with it, it makes sense to somehow relate it to channel events. This information can possibly be fetched from the DB, but I'm not sure if we have it all (we only have block heights, not timestamps, even though we can fetch these from block headers), and I can imagine this could get more complicated than the current approach, as it is quite self contained. Storing open/close events is also not that big of a deal IMO, as it is not in any way critical if it gets out of sync.

For using this information, we could start with something as simple as returning peer uptime % during a listchannels call :)

joostjager · 2019-08-08T11:14:35Z

One other question: how does the channel fitness subsystem relate to lndmon?

From lndmon:

while lnd provides some information, lndmon by far does the heavy lifting with regards to metrics. With lndmon's data, you can track routing fees over time [..]

This seems to overlap with what is started in this pr. For example iterating over time series to extract useful values.

carlaKC · 2019-08-08T22:12:10Z

One other question: how does the channel fitness subsystem relate to lndmon?

I haven't looked at lndmon in detail, but I think that chanfitness is a reduced, inhouse version of lndmon that we need for querying a specific subset of data? There are some things (like peer uptime) that could be exported to lndmon - which actually makes a case for tracking peer online status independent of channels in peernotifier so that we can query uptime over a period (which could be exported to lndmon and chanfitness . I think that may be overkill for now, but moving the logic in chanfitness to peerNotifier at a later time will be pretty trivial.

This seems to overlap with what is started in this pr. For example iterating over time series to extract useful values.

In terms of metrics that may be duplicated on both sides, I don't think we need to specifically track an event log for the payment level data (fees, failed/succeeded amounts) so that would minimise the amount of time series processing we do. Uptime is the only "stateful" series that made more sense to track as time series (I did implement a running counter for uptime but it was very messy). If it comes to a point where there's straight up duplication between the two, then we can look at chanfitness exporting metrics for lndmon?

joostjager · 2019-08-09T08:19:41Z

We had a bit more discussion on the topic. One thing that was brought up, is the case where tracking of data can trigger certain actions like auto-closing of channels. With lndmon, a loop would be created where lnd exports to lndmon, lndmon does time series/analysis/etc and then pushes the result back into lnd to take action. This may not be the architecture that we want.

I think the main thing is for us to be aware that the fitness subsystem and lndmon are related and make a deliberate decision on where to implement new functionality.

I am a bit worried though to which extent we can actually foresee what we need. But yes, you are right, the fitness subsystem could in that case export to lnd to prevent duplication.

With regard to useful things to track as time series in the fitness subsystem (in the future), the metric 'channel profitability' always comes to my mind. To be able to auto-close non-profitable channels. For channel profitability, an analysis over time of local balance and fwding fees is required.

joostjager · 2019-08-09T08:23:38Z

Tagging @valentinewallace as she's been developing lndmon.

carlaKC · 2019-08-12T16:07:01Z

PR updated to add a few changes:

Include uptime in lnrpc/Channel which is returned by ListChannels
Record online/offline events on startup for each channel and on open channel events

When we restart, we need to add a peer online/offline event for existing channels otherwise we will have to wait until they go online/offline to start tracking
A channel open event is not a guarantee that a peer is online, because they can go offline while the funding tx confirms, so we need to know their starting state

There is an edge case for existing channels where peers are recorded as offline because we have not connected to them at restart time, and then online when we establish a connection, which means existing channels will have 99% uptime rather than 100%. This can be addressed by adding a wait before recording initial peer state for channels (eg sleep one minute then check status) if necessary.

valentinewallace · 2019-08-13T00:08:25Z

With lndmon, a loop would be created where lnd exports to lndmon, lndmon does time series/analysis/etc and then pushes the result back into lnd to take action. This may not be the architecture that we want.

I agree that sounds like an architecture to avoid 😄

There are some things (like peer uptime) that could be exported to lndmon

That sounds useful. Prometheus may be able to track peer uptime already since it does know when the set of peers changes 🤔 Let me know if I can contribute more lndmon insight.

joostjager

Definitely much nicer now that something is visible on the rpc interface 👍

I left some comments. My main concern is about the flow of events from subsystems to the fitness tracker.

rpcserver.go

joostjager · 2019-08-14T09:41:12Z

chanfitness/chanevent.go

+	// id is the uInt64 of the short channel ID.
+	id uint64
+
+	peer [33]byte


for pubkeys, there is also the route.Vertex type

chanfitness/chanevent.go

chanfitness/chanevent_test.go

chanfitness/chanevent.go

joostjager · 2019-08-14T10:14:04Z

chanfitness/chanfitness.go

+
+	// PeerEvents provides a subscription client which provides a stream of
+	// peer online/offline events.
+	PeerEvents func() (*subscribe.Client, error)


If these are untyped anyway, and especially if this is extended to a large list in the future, can't the store not just have a single channel over which it receives all events? And then have all event sources drop their typed events into that channel.

So we'd construct a function which pipes all of the subscriptions into a channel which we create in server.go and then pass that to Config? I think that constructing that kind of aggregation from the caller bubbles complexity up rather than down sometimes, because if we have a lot of event sources it's simpler to just provide a list of functions than have some long aggregations of them all in the middle of the newServer call?

Having had my philosophical ramble, I think this may address some of the concerns you have about the unit tests so will give this a try :)

I was actually thinking something simpler. Just make the fitness system a dependency of the other subsystems. It exposes a single function SendEvent to those subsystems. In the implementation of SendEvent, the only thing that happens is that the (untyped) event is dropped into a single buffered channel or queue. The main loop of the fitness system is receiving on that channel and processing the events via a type switch.

But I could be missing something here of course.

My main concern with a SendEvent call is that it feels a bit counterintuitive to be sending events when we have two subscriptions specifically intended for receiving events.

In the case where we don't have subscriptions, a SendEvent makes sense. And I suspect this will be the case for the remainder of the events we add to channel scores (ie htlc level events).

Since it probably (?) doesn't make sense to make subscriptions for htlc level events the options would be:

Use SendEvent everywhere, accepting that it's a bit of an off construction to send events in the cases where you have subscriptions available

Use SendEvent for non-subscription events and still pass the Peer/Chan Subscriptions in, but pipe those into the SendEvent channel so that monitoring happens in one central place.

Isn't it that these subscriptions only exist to serve the event store? In server.go, the call s.peerNotifier.NotifyPeerOnline(pubKey) is made, which could be replaced by a call directly into the event store SendEvent?

I am looking for the simplest thing that works, but may miss something because I didn't follow the design decision around peer notifier.

Although an events chan is looking a lot cleaner for tests. The only ugliness is that it has to be called from the channel open/close notification functions which feels like doing the same thing twice. So it really is dependent on whether subscriptions actually have other use cases. which I'm unsure of.

chanfitness internally manages a queue to not block reporting sub-systems. There only needs to be a single queue for all events.

It would more or less need to re-implement the current queue handling in the subscribe package to do this. Therefore we should re-use what we have rather than attempt to over optimize for this use case which isn't performance critical and still nascent.

I don't think we need to get caught up in this relatively minor difference. The original primary reviewer (johan) was OK with the current approach. We shouldn't undo that prior lineage due to a new reviewer stepping in. For the sake of making progress, I don't think there's a fundamental principle that tends us in one direction or the other. Therefore we can continue down the current path which had already received review by the former primary reviewer.

If this was more critical code such as the channel state machine or revocation handling, then I would be willing to spend more time here, but it's a relatively independent sub-system that is non-critical.

Even though Johan was ok with the current approach, it could still be that he likes the alternative more. I don't think we've discussed that. But yes, both approaches lead to functioning, correct code. This is only about preventing boiler plate. I can put aside my concerns about that and see how it works out.

I think the current approach is more general and would scale better with more subscription types, and more subscribers to the different types.

We had to take a similar decision when we initially added the NotifyPeerOnline API to the fundingmanager. We could have made the server call into the funding manager directly, but rather we made it a general subscription API that any subsystem could call. This has turned out useful to have access to also in other packages.

I also don't see the boiler plate we would avoid, since it is mostly contained in the subscribe package?

chanfitness/chanfitness.go

joostjager · 2019-08-14T10:23:41Z

lnrpc/rpc.proto

+    The percentage of the channel's lifetime that the remote peer has been
+    online.
+    */
+    int64 uptime_percentage = 22 [json_name = "uptime_percentage"];


Maybe we can also add the channel life time here, as we have it available anyway. And I think we should also mark the fields as 'experimental' in the comment, so that we can make breaking changes later if we want to.

Sounds good, it also makes it less likely that the uptime is misinterpreted by users as applying to the whole lifetime of the channel.

How about:
observed_lifetime: total seconds we have been monitoring the channel
observed_uptime: total seconds we have observed the peer online during monitoring

Then users can do their own calculations, and there's no rounding to be dealt with. There's probably a better var name than observed because that makes is sound like we didn't know about the channel before that, but I'll scour synonyms.com later :)

Sounds good. Sounds better. I don't think the observed prefix is needed, is it?

How should I mark fields as experimental? I can't see any examples in the code.
Thinking of something along these lines in the field comments?

[experimental]: this field is experimental and may be subject to change.

We indeed don't have that yet. All our experimental fields are in sub-servers atm. Yes, something along those lines.

IMO we don't have to mark them experimental. We can rather deprecate and add new fields later if we need to.

joostjager · 2019-08-14T10:32:29Z

chanfitness/chanfitness.go

+
+// ChannelEventStore maintains a set of event logs for the node's channels.
+// It is intended to provide insight into the performance of channels.
+type ChannelEventStore struct {


Main type in this file doesn't match file name

+1 on renaming the file

Should I rename the package? I think chanfitness (or channelscores) still works because the package will eventually be revealing scores, but at the moment there won't be a file with the package name which weird.

I would keep the package name, only change the file name :)

joostjager · 2019-10-08T07:25:53Z

chanfitness/chanevent.go

+func (e *chanEventLog) uptime(startTime, endTime time.Time) time.Duration {
+	// Sanity check the period provided.
+	if endTime.Before(startTime) {
+		return 0


Return an error here, to help identify bugs?

joostjager · 2019-10-08T07:34:33Z

chanfitness/chanevent.go

+			// If event is before the period we are calculating uptime for, or we
+			// are currently in an online state, we do not need to increment uptime.
+			if event.timestamp.Before(startTime) {
+				continue


Does this continue do anything?

chanfitness/chanevent.go

chanfitness/chaneventstore_test.go

chanfitness/chaneventstore.go

lnrpc/rpc.proto

joostjager · 2019-10-08T08:04:26Z

chanfitness/chaneventstore_test.go

+			close(events)
+
+			store.wg.Add(1)
+			store.monitorChannelEvents(events)


This looks like it is using a lot if implementation knowledge of the store, making this test expensive to maintain. It needs to change if the internals of the store change. It is preferable to test it more as a black box.

Agree that it's not ideal, but in this case it will be a bit hard to test without accessing this internal method since calling Start will start the subscriptions (which aren't that easy to mock). Also the test is called TestMonitorChannelEvents so maybe okay that it accesses this method directly?

I think this is good enough for now 😄

I think the solution to prevent peeking inside the event lists (implementation specific), would be to create a mock event log. But I am ok leaving it as is.

halseth · 2019-10-14T09:09:04Z

chanfitness/chaneventstore.go

+
+	response := make(chan uptimeResponse)
+
+	c.uptimeRequests <- uptimeRequest{


must select on quit

joostjager · 2019-10-15T06:00:22Z

chanfitness/chanevent.go

+	// id is the uInt64 of the short channel ID.
+	id uint64
+
+	peer [33]byte


No route.Vertex?

chanfitness/chanevent.go

chanfitness/chaneventstore.go

joostjager · 2019-10-15T06:24:23Z

rpcserver.go

+	// If the channel has not been closed yet, and it was found in the channel
+	// store set its endTime to now to calculate lifetime and uptime until the
+	// present.
+	if endTime.IsZero() && err != nil {


Match on a specific error here?

joostjager · 2019-10-15T06:25:25Z

rpcserver.go

+	chanID := dbChannel.ShortChannelID.ToUint64()
+
+	// Get the lifespan observed by the channel event store. It it is unknown,
+	// zero values will be returned (which wil yield a zero lifetime).


If the channel is unknown, shouldn't we then stop making additional calls?

I think it will be handled correctly as it currently is?

channelnotifier/channelnotifier.go

halseth

Also needs a rebase!

chanfitness/chanevent.go

chanfitness/chaneventstore.go

halseth · 2019-10-23T10:38:09Z

chanfitness/chaneventstore_test.go

+			GetChannels: func() (channels []*channeldb.OpenChannel, e error) {
+				return nil, errors.New("intentional test err")
+			},
+			expectStartErr: true,


add a test case for no error?

If we don't error, the start func will try to touch the subscribe clients updates.ChanOut, since it's unexported it can't be mocked. Will remove that bool since and update the comment to note that.

halseth · 2019-10-23T10:39:49Z

chanfitness/chaneventstore_test.go

+			}
+
+			// Stop the store's go routine.
+			store.Stop()


can defer stop this?

channelnotifier/channelnotifier.go

halseth · 2019-10-23T10:43:46Z

rpcserver.go

+	chanID := dbChannel.ShortChannelID.ToUint64()
+
+	// Get the lifespan observed by the channel event store. It it is unknown,
+	// zero values will be returned (which wil yield a zero lifetime).


carlaKC requested review from cfromknecht and Roasbeef as code owners July 22, 2019 15:02

carlaKC force-pushed the chanfitness-trackchanneluptime branch from e04f971 to 1683f2a Compare July 22, 2019 18:23

halseth suggested changes Jul 24, 2019

View reviewed changes

carlaKC force-pushed the chanfitness-trackchanneluptime branch from 1683f2a to 7551588 Compare July 25, 2019 19:12

carlaKC mentioned this pull request Jul 29, 2019

Peernotifier: Add Peer Notifier package #3354

Merged

4 tasks

Roasbeef added channel management The management of the nodes channels enhancement Improvements to existing features / behaviour P3 might get fixed, nice to have v0.8 labels Jul 29, 2019

Roasbeef removed their request for review July 29, 2019 23:48

Roasbeef requested review from joostjager and removed request for cfromknecht August 7, 2019 06:51

halseth suggested changes Aug 7, 2019

View reviewed changes

joostjager reviewed Aug 7, 2019

View reviewed changes

carlaKC force-pushed the chanfitness-trackchanneluptime branch from 7551588 to 9238cf0 Compare August 12, 2019 15:57

joostjager reviewed Aug 14, 2019

View reviewed changes

carlaKC force-pushed the chanfitness-trackchanneluptime branch from 9238cf0 to 8309a24 Compare August 28, 2019 18:15

joostjager added the v0.9.0 label Oct 7, 2019

joostjager added this to the 0.9 milestone Oct 7, 2019

carlaKC requested review from halseth and joostjager October 7, 2019 11:59

joostjager reviewed Oct 8, 2019

View reviewed changes

carlaKC force-pushed the chanfitness-trackchanneluptime branch 3 times, most recently from ea5d140 to 8b50c8c Compare October 14, 2019 07:59

carlaKC requested a review from joostjager October 14, 2019 08:00

halseth reviewed Oct 14, 2019

View reviewed changes

carlaKC force-pushed the chanfitness-trackchanneluptime branch from 8b50c8c to f74db87 Compare October 14, 2019 15:50

joostjager reviewed Oct 15, 2019

View reviewed changes

halseth suggested changes Oct 23, 2019

View reviewed changes


		response := make(chan uptimeResponse)

		c.uptimeRequests <- uptimeRequest{

Chanfitness: Track peer uptime #3332

Chanfitness: Track peer uptime #3332

Conversation

carlaKC commented Jul 22, 2019

PR Checklist

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlaKC commented Jul 25, 2019

halseth commented Jul 29, 2019

halseth commented Aug 7, 2019

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth Oct 23, 2019 • edited Loading

Choose a reason for hiding this comment

carlaKC Oct 24, 2019 • edited Loading

Choose a reason for hiding this comment

joostjager left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlaKC commented Aug 7, 2019

joostjager commented Aug 7, 2019

halseth commented Aug 7, 2019

joostjager commented Aug 8, 2019

carlaKC commented Aug 8, 2019

joostjager commented Aug 9, 2019

joostjager commented Aug 9, 2019

carlaKC commented Aug 12, 2019

valentinewallace commented Aug 13, 2019

joostjager left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager Aug 14, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth Oct 23, 2019 •

edited

Loading

carlaKC Oct 24, 2019 •

edited

Loading

joostjager Aug 14, 2019 •

edited

Loading