funding: decouple funding wait from fundingLocked and chanAnn #319

Crypt-iQ · 2017-09-07T08:46:51Z

This PR tackles the first step of issue #305. waitForFundingConfirmation is decoupled from the channel announcements and sendFundingLockedAndAnnounceChannel is now decoupled into sendFundingLocked and sendChannelAnnouncement.

A future PR will handle:

Waiting for 6 confirmations before channel announcement in sendChannelAnnouncement.
Adding the channel to the channel graph immediately after sending the FundingLocked message.

mention-bot · 2017-09-07T08:46:54Z

@Crypt-iQ, thanks for your PR! By analyzing the history of the files in this pull request, we identified @Roasbeef, @bryanvu and @AndrewSamokhvalov to be potential reviewers.

halseth

Thanks for working on this! The funding flow code definitely needs some refactoring 😆 I'll try to answer the questions you have 😄

See comments inline, I think this is close to what we want, but it should also be decoupled from waitForFundingWithTimeout.
The spec says that

If sent, announcement_signatures messages MUST NOT be sent until [...] the funding transaction is has at least 6 confirmations.

So I believe in this case we should actually wait for at least six confirmations.

Looking at *fundingConfig.NumRequiredConfs I think this is meant as a method to determine our preferred minimum_depth for the accept_channel message:
https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#the-accept_channel-message.
This basically determine how many blocks we want the funding confirmation to have before we send funding_locked. This is different from the announcement message minimum confirmations:
https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md#the-announcement_signatures-message, but as you can see it also requires the funding_locked to have been sent, meaning it will indirectly require max(numConfs, 6) confirmations.

halseth · 2017-09-07T09:25:49Z

fundingmanager.go

+				// Successfully received confirmation details
+				// from waitForFundingConfirmation.  Now we
+				// we wait to receive from doneChan.
+			case <-doneChan:


You don't need two channels here, the confirmChan should be enough.

halseth · 2017-09-07T09:28:13Z

fundingmanager.go

+			// The funding tx has the correct number of confirmations,
+			// so we can now broadcast the channel announcement messages
+			// to the greater network.
+			f.announceChannelAfterConfirmations(completeChan, confDetails)


this should be done outside waitForFundingWithTimeout

halseth · 2017-09-07T09:31:56Z

fundingmanager.go

+// Finally, the fundingLocked message is sent to the peer and the channel is announced
+// to the greater network.  This method should be called only after waitForFundingConfirmation
+// completes successfully.
+func (f *fundingManager) announceChannelAfterConfirmations(


Before announcing the channel, this method marks the channel opened in the db, and sends the fundingLocked message to the peer, so I think the name is a bit misleading. Maybe something like processConfirmedFunding could be more telling?

I agree, I actually had trouble coming up with a name for this...

Crypt-iQ · 2017-09-07T11:53:05Z

Thanks for the comments, especially re: confirmations stuff. I was a bit confused by the mention of confirmations for different messages, but I get it now. I'm going to rework it and update the PR today most likely.

halseth

Nice, getting very close I think! 💯

The 6 confirmations check should be done here: https://github.com/lightningnetwork/lnd/pull/319/files#diff-691ca64693e704fd2e5080998f3e640eR1457

I think it should also be fairly easy to add a test for this to fundingmanager_test.go

halseth · 2017-09-07T15:32:33Z

fundingmanager.go

-				// Success, funding transaction was confirmed.
+			case err = <-errChan:
+				// Encountered an error
+				fndgLog.Errorf("Encountered an error")


log the actual error

halseth · 2017-09-07T15:36:43Z

fundingmanager.go

-		case <-doneChan:
+		case err := <-errChan:
+			// Encountered an error
+			fndgLog.Errorf("Encountered an error: %v", err)


Better text here is "error waiting for funding transaction to confirm: %v"

halseth · 2017-09-07T15:38:38Z

fundingmanager.go

 			// Success, funding transaction was confirmed.
 			f.deleteReservationCtx(peerKey,
 				fmsg.msg.PendingChannelID)
+			// Now we can call processConfirmedFunding
+			f.processConfirmedFunding(completeChan, confDetails)


this must be done before the reservation context is deleted, to keep the original behavior.

halseth · 2017-09-07T15:42:58Z

fundingmanager.go

 		}()

 		select {
 		case <-f.quit:
 			return
-		case <-doneChan:
+		case err := <-errChan:
+			fndgLog.Errorf("Encountered an error: %v", err)


halseth · 2017-09-07T15:44:26Z

fundingmanager.go

@@ -1200,32 +1222,35 @@ func (f *fundingManager) handleFundingSigned(fmsg *fundingSignedMsg) {
 // passed from bestHeight. In the case of timeout, the timeoutChan will be
 // closed. In case of confirmation or error, doneChan will be closed.
 func (f *fundingManager) waitForFundingWithTimeout(completeChan *channeldb.OpenChannel,
-	doneChan chan<- struct{}, timeoutChan chan<- struct{}) {
+	errChan chan<- error, timeoutChan chan<- struct{},


maybe have the channels in this order: confirmChan, timeoutChan, errChan? :)

halseth · 2017-09-07T15:47:15Z

fundingmanager.go

-	cancelChan <-chan struct{}, doneChan chan<- struct{}) {
-
-	defer close(doneChan)
+	cancelChan <-chan struct{}, errChan chan<- error,


order: cancelChan, confirmChan, errChan

halseth · 2017-09-07T15:49:35Z

fundingmanager.go


 	// Register with the ChainNotifier for a notification once the funding
 	// transaction reaches `numConfs` confirmations.
 	txid := completeChan.FundingOutpoint.Hash
 	numConfs := uint32(completeChan.NumConfsRequired)
+	if (numConfs < 6) {
+		numConfs = 6


This will be wrong. This will prevent the fundingLocked being sent with < 6 confirmations, which is not what we want. This check should be done after fundingLocked is sent.

halseth · 2017-09-07T15:52:53Z

fundingmanager.go

@@ -1310,6 +1344,20 @@ func (f *fundingManager) waitForFundingConfirmation(completeChan *channeldb.Open
 		return
 	}

+	// Send confirmation details to confirmChan for use in processConfirmedFunding
+	confirmChan <- confDetails


move this together with the !ok check inside the select

Crypt-iQ · 2017-09-08T20:18:52Z

I believe that I have successfully decoupled waitForFundingConfirmation from the channel announcement. Adding the channel to the ChannelRouter immediately after the fundingLocked message has been sent is still TODO.

I still have some concerns:

Do I need to wait for 6 confirmations here before sendChannelAnnouncement is called?
As I was testing my new code with the TestFundingManagerNormalWorkflow test in fundingmanager_test.go, it did not pass successfully. It was not until I duplicated lines 531 & 532, and adjusted mockNotifier.confChannel to have a capacity of 2 that the test passed. Looking here, I noticed that the tuple (txid, numConfs) had to be identical for it to be dispatched to ALL clients. However, I tried setting numConfs in waitForFundingConfirmation to 6, but this still resulted in failure.

Crypt-iQ · 2017-09-10T00:29:27Z

My recent changes also include the first steps to a fix for issue #306. I added ChannelFlags to the OpenChannel struct and have specific methods for putting, fetching, & deleting ChannelFlags in channel.go.

Crypt-iQ · 2017-09-10T12:57:42Z

I have several TODOs left such as a command-line option for channelFlags / "stealth channel", and handling confirmations here.

Additionally in this latest commit 5d6498a, issue #306 should be completed.

Crypt-iQ · 2017-09-12T13:53:38Z

I added a --stealth option which sets the LSB of channel_flags to 0 if the option is included.
I also handled confirmations here.

My main obstacle that I must tackle before this is completed is that to have a functional channel graph where users can send payments to one another, both edges in a channel must be defined. Channels are bidirectional. It is trivial to send OUR data (ChannelEdgeInfo & ChannelEdgePolicy) to the ChannelRouter before any announcements are made. However, it is not trivial to send our peer's ChannelEdgePolicy to the ChannelRouter before announcements happen. Is some sort of exchange of ChannelEdgePolicys made at the TODO here?

Additionally, what channel info is added in the case of ChannelFlags LSB being set to 0 and when it's set to 1? Do we need to add our peer's ChannelEdgePolicy when the LSB is 1 (since it will get added anyways during the announcement phase... I'm thinking yes)? In the case of the LSB being 0, I'm thinking that ChannelEdgeInfo, ChannelEdgePolicy, our peer's ChannelEdgePolicy must all be added at that TODO.

Crypt-iQ · 2017-09-12T17:44:55Z

@halseth suggested splitting this PR into multiple PRs. The current PR now only contains the decoupling of sending the fundingLocked message and the sending of the channel announcement from waitForFundingConfirmation.

halseth

Looks good! Only stylistic nits from me at this point.

Please also update the PR description to what parts of the issue this currently is handling, and what will be done in a follow-up PR.

Also, if you take a look at the contribution guidelines and earlier commits, you can see how commit messages should be formatted. Thanks!

halseth · 2017-09-13T13:51:58Z

fundingmanager.go

+				}
+				defer lnChannel.Stop()
+
+				if err = f.sendFundingLocked(ch, lnChannel,


might fit on one line if you do

err = f.sendFundingLocked( ... if err != nil { ...

halseth · 2017-09-13T14:18:43Z

fundingmanager.go

+						"fundingLocked: %v", err)
+					return
+				}
+				if err = f.sendChannelAnnouncement(ch, lnChannel,


halseth · 2017-09-13T14:21:09Z

fundingmanager.go

+				}
+				defer lnChannel.Stop()
+
+				if err = f.sendFundingLocked(channel, lnChannel,


halseth · 2017-09-13T14:21:12Z

fundingmanager.go

+						"fundingLocked: %v", err)
+					return
+				}
+				if err = f.sendChannelAnnouncement(channel,


halseth · 2017-09-13T14:22:03Z

fundingmanager.go

-				f.sendFundingLockedAndAnnounceChannel(channel,
-					shortChanID)
+
+				// With the channel marked open, we'll create


probably redundant comment

halseth · 2017-09-13T14:26:26Z

fundingmanager.go

+				fndgLog.Errorf("Failed to send fundingLocked: %v", err)
+				return
+			}
+			if err = f.sendChannelAnnouncement(completeChan,


halseth · 2017-09-13T14:26:27Z

fundingmanager.go

+			}
+			defer lnChannel.Stop()
+
+			if err = f.sendFundingLocked(completeChan, lnChannel,


halseth · 2017-09-13T14:27:12Z

fundingmanager.go

@@ -1230,21 +1342,21 @@ func (f *fundingManager) handleFundingSigned(fmsg *fundingSignedMsg) {
 // waitForFundingWithTimeout is a wrapper around waitForFundingConfirmation that
 // will cancel the wait for confirmation if maxWaitNumBlocksFundingConf has
 // passed from bestHeight. In the case of timeout, the timeoutChan will be
-// closed. In case of confirmation or error, doneChan will be closed.
+// closed. In case of confirmation or error, confChan will be closed.


update comment to reflect the returned value on success

halseth · 2017-09-13T14:31:19Z

fundingmanager.go

@@ -1297,9 +1414,9 @@ func (f *fundingManager) waitForFundingWithTimeout(completeChan *channeldb.OpenC
 // when a channel has become active for lightning transactions.
 // The wait can be canceled by closing the cancelChan.


add confChan to comment

halseth · 2017-09-13T14:33:26Z

fundingmanager.go

@@ -2105,4 +2211,4 @@ func (f *fundingManager) deleteChannelOpeningState(chanPoint *wire.OutPoint) err
 		}
 		return nil
 	})
-}
+}


add back newline

halseth

LGTM now 👍

halseth · 2017-09-14T11:45:51Z

fundingmanager.go

-			case <-doneChan:
+			case shortChanID, ok := <-confChan:
+				if !ok {
+					fndgLog.Errorf("confChan was closed")


better error message here

halseth

Noticed that the build is failing. Looks like you must run gofmt -s on the changed files.

jimpo · 2017-09-16T23:14:05Z

fundingmanager.go

-			close(doneChan)
+		case shortChanID, ok := <-waitingConfChan:
+			if !ok {
+				fndgLog.Errorf("waiting for funding confirmation" +


This is getting logged twice, both here and in Start. I would drop this log line and just close & return here.

jimpo · 2017-09-16T23:21:53Z

fundingmanager.go

+				// With the channel marked open, we'll create the
+				// state-machine object which wraps the database
+				// state.
+				lnChannel, err := lnwallet.NewLightningChannel(


I know this is only part 1 of a two-part change, but this chunk of code is now repeated 4 times. I think it makes more sense to keep it in its own method for now, and the method name sendFundingLockedAndAnnounceChannel is still appropriate.

I agree with all of the suggested changes besides sendFundingLockedAndAnnounceChannel because my later PR will have complicated logic that is better off from having them decoupled. There is a branch of code which ONLY sends the announcement, but not the funding locked message (this code branch needs to add the channel to the channel graph before announcement). The issue says to include adding to the channel router graph immediately after
sending the funding locked message. I think the code is cleaner having sendFundingLocked, sendChannelAnnouncement, & addToRouterGraph as separate functions.

I'm not suggesting you remove the sendFundingLocked, sendChannelAnnouncement, and addToRouterGraph methods, I'm just suggesting you add another method sendFundingLockedAndAnnounceChannel (or handleFundingConfirmation), that calls those methods in the right order. I read over #332, and it appears the duplicated code exists there as well with no differences between the 4 occurrences.

Ah gotcha. #332 is still dependent on this, so it very well could change - it's more of a placeholder PR at this point.

I like handleFundingConfirmation :)

jimpo · 2017-09-16T23:24:43Z

fundingmanager.go

@@ -1463,7 +1579,7 @@ func (f *fundingManager) sendChannelAnnouncement(completeChan *channeldb.OpenCha
 		*shortChanID, chanID)
 	if err != nil {
 		fndgLog.Errorf("channel announcement failed: %v", err)


This error gets logged twice, both here and where the returned error value is handled. I'll let someone more familiar with this project's style weigh in, but if the error gets returned, I don't think it's appropriate to log here. There are more instances in this method and in sendFundingLocked.

I think the best way to handle this is to return a new error that combines the two:
return fmt.Errorf("channel announcement failed: %v", err)

halseth

A few comments on line length and returned errors, otherwise very good :)

halseth · 2017-09-17T17:16:44Z

fundingmanager.go

+			// wait for funding transaction on startup.
+		case shortChanID, ok := <-confChan:
+			if !ok {
+				fndgLog.Errorf("waiting for funding confirmation failed")


This line looks a bit long. We try to keep a 80 character line length width 8 width tabs.

halseth · 2017-09-17T17:16:58Z

fundingmanager.go

+
+			// Call the function that handles funding
+			// confirmation.
+			err := f.handleFundingConfirmation(completeChan, shortChanID)


probably a bit long

halseth · 2017-09-17T17:17:18Z

fundingmanager.go

-		case <-doneChan:
+		case shortChanID, ok := <-confChan:
+			if !ok {
+				fndgLog.Errorf("waiting for funding confirmation failed")


halseth · 2017-09-17T17:17:33Z

fundingmanager.go

+
+			// Call the function that handles funding
+			// confirmation.
+			err := f.handleFundingConfirmation(completeChan, shortChanID)


halseth · 2017-09-17T17:20:45Z

fundingmanager.go

@@ -455,11 +476,17 @@ func (f *fundingManager) Start() error {
 				if err != nil {
 					fndgLog.Errorf("error creating "+
 						"lightning channel: %v", err)
+					return


Good catch!

halseth · 2017-09-17T17:25:05Z

fundingmanager.go

@@ -1188,8 +1228,10 @@ func (f *fundingManager) handleFundingSigned(fmsg *fundingSignedMsg) {
 		},
 	}

+	f.wg.Add(1)


Nice, this had to be added to make sure the goroutine was properly shutdown between tests.

halseth · 2017-09-17T17:27:06Z

fundingmanager.go

-func (f *fundingManager) sendFundingLockedAndAnnounceChannel(
-	completeChan *channeldb.OpenChannel, shortChanID *lnwire.ShortChannelID) {
+// handleFundingConfirmation is a wrapper method for creating a new lightning
+// channel, calling sendFundingLocked, and for calling sendChannelAnnouncement.


lightning channel ->LightningChannel

halseth · 2017-09-17T17:27:50Z

fundingmanager.go

-	completeChan *channeldb.OpenChannel, shortChanID *lnwire.ShortChannelID) {
+// handleFundingConfirmation is a wrapper method for creating a new lightning
+// channel, calling sendFundingLocked, and for calling sendChannelAnnouncement.
+// This is called when the funding transaction is first confirmed and the channel


... called after the funding...

halseth · 2017-09-17T17:30:31Z

fundingmanager.go

@@ -1416,14 +1493,14 @@ func (f *fundingManager) sendFundingLockedAndAnnounceChannel(
 	nextRevocation, err := channel.NextRevocationKey()
 	if err != nil {
 		fndgLog.Errorf("unable to create next revocation: %v", err)
-		return
+		return err


do return fmt.Errorf("unable to create next revocation: %v", err) here and other places in this method

halseth · 2017-09-17T17:31:15Z

fundingmanager.go

@@ -1474,7 +1550,7 @@ func (f *fundingManager) sendChannelAnnouncement(completeChan *channeldb.OpenCha
 	err = f.deleteChannelOpeningState(&completeChan.FundingOutpoint)
 	if err != nil {
 		fndgLog.Errorf("error deleting channel state: %v", err)
-		return
+		return err


same in this method

jimpo · 2017-09-17T17:38:16Z

fundingmanager.go

 				// Success, funding transaction was confirmed.
+
+				// Call the function that handles funding


This comment is not necessary IMO. The next line is pretty clear.

halseth

Fix the last comments, rebase and squash and it LGTM! 🌔

halseth · 2017-09-17T19:05:43Z

fundingmanager.go

-			f.deleteReservationCtx(peerKey,
-				fmsg.msg.PendingChannelID)
+
+			// Call the function that handles funding


comment can also be removed, as discussed earlier.

halseth · 2017-09-17T19:07:27Z

fundingmanager.go

+			}
+
+			// Call the function that handles funding
+			// confirmation.


Use
// Success, funding transaction was confirmed.

halseth · 2017-09-17T19:08:17Z

fundingmanager.go

-			close(doneChan)
+		case shortChanID, ok := <-waitingConfChan:
+			if !ok {
+				close(confChan)


Add comment
// Failed waiting for confirmation, close confChan to indicate failure.

halseth · 2017-09-17T19:09:26Z

fundingmanager.go

 	if err != nil {
-		fndgLog.Errorf("error creating new lightning channel: %v", err)
-		return
+		return err


fmt.Errorf("failed sending fundingLocked: %v", err)

halseth · 2017-09-17T19:09:41Z

fundingmanager.go

-	defer channel.Stop()
+	err = f.sendChannelAnnouncement(completeChan, lnChannel, shortChanID)
+	if err != nil {
+		return err


fmt.Errorf("failed sending channel announcement: %v", err)

Roasbeef · 2017-09-20T00:05:49Z

fundingmanager.go


-	chanID := lnwire.NewChanIDFromOutPoint(&completeChan.FundingOutpoint)
+	// We create the state-machine object which wraps the database state


Nit: missing a period at the end of the sentence.

Roasbeef · 2017-09-20T00:11:17Z

fundingmanager.go

-	// Now that the funding transaction has the required number of
-	// confirmations, we send the fundingLocked message to the peer.
-	f.sendFundingLockedAndAnnounceChannel(completeChan, &shortChanID)
+	confChan <- &shortChanID


In order to avoid possible future deadlocks on shutdown, this should be converted to instead be a conditional send:

select { case confChan <- &shortChanID: case <-f.quit: return }

Roasbeef · 2017-09-20T00:12:28Z

fundingmanager.go

+				close(confChan)
+				return
+			}
+			confChan <- shortChanID


In order to avoid possible future deadlocks on shutdown, this should be converted to instead be a conditional send:

select { case confChan <- shortChanID: case <-f.quit: return }

jimpo · 2017-09-21T18:16:25Z

fad92ce is a merge. I think it might be preferred to rebase onto master instead (https://github.com/lightningnetwork/lnd/blob/master/docs/code_contribution_guidelines.md#53-acceptance).

This commit decouples the wait for funding transaction confirmations in the waitForFundingConfirmation function from the announcement of the channel in the sendFundingLockedAndAnnounceChannel function. Additionally, the sendFundingLockedAndAnnounceChannel function is now decoupled into the sendFundingLocked and sendChannelAnnouncement functions. There is also now a helper function that houses creation of a lnwire.LightningChannel object, calls to both sendFundingLocked and sendChannelAnnouncement.

Roasbeef

LGTM 💫

halseth suggested changes Sep 7, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch from 36f3cc3 to e377d0c Compare September 8, 2017 04:28

Crypt-iQ force-pushed the blockBasedGracePeriod branch from a1f8fb5 to 14e9aed Compare September 13, 2017 11:08

halseth suggested changes Sep 13, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch 2 times, most recently from 38331d4 to e004e10 Compare September 13, 2017 15:29

Crypt-iQ changed the title ~~Decoupled waitForFundingConfirmation with channel announcement~~ funding: decouple funding wait from fundingLocked and chanAnn Sep 13, 2017

Crypt-iQ force-pushed the blockBasedGracePeriod branch from dd77c4f to 3791a7d Compare September 13, 2017 17:42

halseth approved these changes Sep 14, 2017

View reviewed changes

halseth suggested changes Sep 14, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch 2 times, most recently from 73fdb6f to 3b724d5 Compare September 14, 2017 12:53

Crypt-iQ mentioned this pull request Sep 14, 2017

funding: adding to topology after fundinglocked, private channels, chanAnn after 6 confs #332

Closed

Crypt-iQ force-pushed the blockBasedGracePeriod branch 2 times, most recently from b4b6d10 to 65d4936 Compare September 14, 2017 17:32

jimpo reviewed Sep 16, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch from aa9855b to adda49b Compare September 17, 2017 15:12

halseth suggested changes Sep 17, 2017

View reviewed changes

jimpo reviewed Sep 17, 2017

View reviewed changes

halseth approved these changes Sep 17, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch 2 times, most recently from 58d0532 to 9f5d54f Compare September 17, 2017 19:37

Roasbeef requested changes Sep 20, 2017

View reviewed changes

Crypt-iQ force-pushed the blockBasedGracePeriod branch 2 times, most recently from 6fb2446 to bb2cbfe Compare September 23, 2017 22:07

Crypt-iQ force-pushed the blockBasedGracePeriod branch from bb2cbfe to 9e5813a Compare October 4, 2017 10:59

Crypt-iQ force-pushed the blockBasedGracePeriod branch from 9e5813a to 426a803 Compare October 4, 2017 11:20

Roasbeef approved these changes Oct 27, 2017

View reviewed changes

Roasbeef merged commit 9c0c889 into lightningnetwork:master Oct 27, 2017

		@@ -1297,9 +1414,9 @@ func (f fundingManager) waitForFundingWithTimeout(completeChan channeldb.OpenC
		// when a channel has become active for lightning transactions.
		// The wait can be canceled by closing the cancelChan.

		// Success, funding transaction was confirmed.

		// Call the function that handles funding


		chanID := lnwire.NewChanIDFromOutPoint(&completeChan.FundingOutpoint)
		// We create the state-machine object which wraps the database state

funding: decouple funding wait from fundingLocked and chanAnn #319

funding: decouple funding wait from fundingLocked and chanAnn #319

Conversation

Crypt-iQ commented Sep 7, 2017 • edited

mention-bot commented Sep 7, 2017

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crypt-iQ commented Sep 7, 2017

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crypt-iQ commented Sep 8, 2017 • edited

Crypt-iQ commented Sep 10, 2017

Crypt-iQ commented Sep 10, 2017 • edited

Crypt-iQ commented Sep 12, 2017 • edited

Crypt-iQ commented Sep 12, 2017

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crypt-iQ Sep 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Crypt-iQ Sep 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimpo commented Sep 21, 2017

Roasbeef left a comment • edited

Crypt-iQ commented Sep 7, 2017 •

edited

Crypt-iQ commented Sep 8, 2017 •

edited

Crypt-iQ commented Sep 10, 2017 •

edited

Crypt-iQ commented Sep 12, 2017 •

edited

halseth left a comment •

edited

Crypt-iQ Sep 17, 2017 •

edited

Crypt-iQ Sep 17, 2017 •

edited

Roasbeef left a comment •

edited