Allow extraction of logs from running lnd instances in Travis integration tests

[Roasbeef]

Issue

We currently use Travis (like many other projects) for automated Continuous Integration within the project. We run a few batches of tests across the versions of golang that we officially support, toggling on the [race condition detector] to run the project wide unit tests, and also using our integration testing framework to exercise various real-world node scenarios.

Occasionally, we run into flakes on travis either due to timeouts we have in the integration tests being too long or, due to a newly uncovered bug. At times it's difficult for developers to reproduce the issues locally as we can't exactly reproduce the conditions on the instances that Travis executes the tests on. If we can create a pipeline to export the logs of all running lnd instances within the integration tests, then that would greatly aide our efforts in tracking down certain classes of flakes triggered within the integration tests.

Steps to Completion

• Modify our Travis configuration to extract the logs (either directly from the stdout of each of the running lnd instances or reading directly from disk) to an existing storage system. This system might be s3, glcoud buckets, or simply just a pastebin. This should be done in an automated fashion, and it should be easy to locate the logs after test execution. The medium we use to upload the logs should likely be ephemeral so we don't accumulate a ton of space on whichever system is chosen.

[samvrlewis]

Keen to look into this if it hasn't already been tackled, looks like a good way to get my feet wet!

[Roasbeef]

@samvrlewis this issue hasn't been picked up yet. Solving it would greatly increase our ability to stamp down the remaining flakes we occasionally run into on travis!

[samvrlewis]

Have looked about this and have a few options that I'd be interesting in hearing others thoughts on:

Option 1. Append logs to the travis build log
+ Easy solution
+ Don't need a 3rd party service
+ Logs are easily associated with the build
- Travis limits logs to 4MB. At the moment the lnd build log is ~90KB and the logs of all the integration tests are ~800KB. Some room to grow, but not heaps.
- Some log pollution. Can make use of folding to make logs a bit more readable.

Option 2. AWS/GCloud
+ Flexible in terms of log size etc
- Need credit card
- Need mechanism (script/cronjob) for clearing out old logs
- Difficult to administrate if tied to a particular person's AWS/GCloud account?

Option 3. Bintray
+ Free for open source projects (10GB storage)
- Need mechanism (script/cronjob) for clearing out old logs so storage limits not exceeded
- Reliance on (somewhat unknown) third party service

Option 4. Github repo/gists
+ Easy
+ Don't need a 3rd party service
- Not really the intended use of Github
- Github fair use/limitations??
- Probably need some way of deleting old logs

Do you have any thoughts @Roasbeef or others?

I suppose the AWS/GCloud option would probably be the neatest to pursue, however I'm not sure how to resolve the issue of account ownership for the AWS/GCloud account - particularly as both services require a credit card to be registered to the account. How do open source projects typically register accounts for services like AWS?

Also, I'm new to building Golang applications but I'm having a bit of trouble successfully building my fork of the lnd repo because the packages all refer to github.com/lightningnetwork/lnd. The only way I've managed to get my fork to build is to find and replace github.com/lightningnetwork/lnd with github.com/samvrlewis/lnd. Is there a more eloquent way of doing this? If not, would it be possible for me to get write access to a development branch in this repo to work on this?

[Roasbeef]

Hi @samvrlewis,

Thanks for looking into this!

I appreciate the breakdown of the distinct directions we can pursue to implement this much needed testing/debugging feature. I like the gist option as it doesn't require an individual to manage an account hooked up to a credit or that requires an API key for authentication. Taking it a bit further, perhaps we can use zerobin type paste service that innately has a "burn after reading" or time based paste expiry. This seems like a rather seamless path, as the travis script can pipe the logs to another program that itself uses an HTTP API to create the paste, printing out the URL of the created paste.

Any thoughts?

Also, I'm new to building Golang applications but I'm having a bit of trouble successfully building my fork of the lnd repo because the packages all refer to github.com/lightningnetwork/lnd

The way I usually do this is first to fork the repo itself on github. Once you have a new remote URI for your fork of lnd, you'll then go to the location of lnd within your GOPATH and add your fork as a new remote. With this set up, you'll then manually specify to push to your fork for any commits or updates to a branch. This allows you to keep intact all the imports and glide files, avoiding the need to modify them at all.

[samvrlewis]

Thanks for the thoughts @Roasbeef. I actually came across transfer.sh the other day which looks perfect for this, so will push on using that.

The way I usually do this is first to fork the repo itself on github. Once you have a new remote URI for your fork of lnd, you'll then go to the location of lnd within your GOPATH and add your fork as a new remote. With this set up, you'll then manually specify to push to your fork for any commits or updates to a branch. This allows you to keep intact all the imports and glide files, avoiding the need to modify them at all.

This is the same method I tried I think, but I wasn't able to get TravisCI to build my fork properly as the package names in lnd refer to github.com/lightningnetwork/lnd but Travis had my github.com/samvrlewis/lnd fork. Building on my local machine works fine but for this bug it'd be nice if I could build my fork on Travis. I'll play around a bit more and see if it's something stupid that I'm doing.

As an example of a Travis build from my fork (nothing modified): https://travis-ci.org/samvrlewis/lnd/builds/285520121 . It fails with:

lnd.go:31:2: cannot find package "github.com/lightningnetwork/lnd/autopilot" in any of:
	/home/travis/gopath/src/github.com/samvrlewis/lnd/vendor/github.com/lightningnetwork/lnd/autopilot (vendor tree)
        ....

[samvrlewis]

Aha, figured out an easy way to get builds of the forks to work (from cloudfoundry/cli#286 ):

Add the following to the .travis.yml:

before_install:
  - RepoName=`basename $PWD`; SrcDir=`dirname $PWD`; DestDir="`dirname $SrcDir`/lightningnetwork"
  - if [[ "$SrcDir" != "$DestDir" ]]; then mv "$SrcDir" "$DestDir"; cd ../../lightningnetwork/$RepoName; export TRAVIS_BUILD_DIR=`dirname $TRAVIS_BUILD_DIR`/$RepoName; fi

If this is useful feature for anyone else, I'm happy to open it as a separate PR otherwise will leave it here for posterity.