Fold stellar/horizon into stellar/go/services/horizon #93

nikhilsaraf · 2017-10-04T22:32:40Z

Notes for the current commit:

stellar/go/services/horizon/scripts/run_tests.bash passes, and the horizon server builds and serves on localhost:8000
I've updated stellar/go/services/horizon/README.md to link to the stellar/go/releases page so these download links are currently broken. We should backfill these links by creating new releases, or update these links to point to the old files in stellar/horizon/releases until we decide to push out a new horizon release.
The horizon docs now point to filing all issues to the “go” monorepo (stellar/go/issues) so we should expect that change in behavior from the community down the road.
The only bash script I've updated is stellar/go/services/horizon/scripts/run_tests.bash; all other scripts in stellar/go/services/horizon/scripts are currently broken because they depend on gb. We should work on fixing these scripts as a separate commit if we can tolerate having them broken for a short window — I can work on fixing them after this commit.
I will also need to port any changes made to horizon that were made after I started my changes, there's 2 so far I think — I can include these changes after this commit is approved.

Changes needed in other repos to support this move:

Need to update https://github.com/stellar/developers/blob/master/repos.json to point to the new url — I will work on this after this commit. (UPDATE: see updated developers repo to reflect the change in horizon's location stellar-deprecated/developers#88)
I need to add a deprecated message to the old HORIZON repo’s README.md with a pointer to the new location — I can work on this after this commit. (UPDATE: see add repo moved message to stellar/horizon repo stellar-deprecated/horizon#407)
UPDATE: stellar/docs needs to be updated with the new url (see make "Software" tab in docs point to the new horizon location stellar-deprecated/docs#233)

…path (go/services/horizon)

nikhilsaraf · 2017-10-04T23:44:07Z

had not included the /tree/master portion for web-based github urls so fixed that in commit no. 4

nullstyle · 2017-10-05T15:54:13Z

Looks like we're getting failures in Travis when running the test suite.

…x test failures in goji.io/mux ("context" import)

nikhilsaraf · 2017-10-05T17:13:43Z

fixed a dependency issue (in commit 5) by picking the older option between stellar/go and stellar/horizon for goji.io, will address the remaining test failures to resolve the build.

…o prevent test failures caused by db parallelism

did not add the following lines (build_artifacts.bash is broken right now, let's see if it works without that): before_deploy: - bash scripts/build_artifacts.bash cache: directories: - vendor/src - pkg

nullstyle · 2017-10-05T15:54:57Z

services/horizon/.envrc

@@ -0,0 +1,2 @@
+export PATH=$PWD/bin:$PATH


This file is no longer needed since we aren't using gb anymore

nullstyle · 2017-10-05T18:42:24Z

.travis.yml

 go:
 - '1.6'
 - '1.7'
 - '1.8'
 - tip
 env:
- GO15VENDOREXPERIMENT="1"
+  global:
+  - GO15VENDOREXPERIMENT="1"


We don't need this anymore... go's vendor functionality is standard now on all of our supported versions (1.7, 1.8, 1.9)

nullstyle · 2017-10-05T18:43:47Z

.travis.yml

- GO15VENDOREXPERIMENT="1"
+  global:
+  - GO15VENDOREXPERIMENT="1"
+  - DATABASE_URL=postgres://localhost:5432/horizon_test?sslmode=disable&user=postgres


These environment variables (DATABASE_URL and STELLAR_CORE_DATABASE_URL) should not be needed to run our tests in the monorepo. Before we merge I think we should make the necessary changes to excise the need for their existence from the test suite.

I've removed it in the latest commit, hopefully the build passes with that change.

nullstyle · 2017-10-05T18:45:49Z

.travis.yml

 install:
 - wget https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz
 - tar -xzvf glide-v0.12.3-linux-amd64.tar.gz
 - cp linux-amd64/glide $GOPATH/bin
 before_script: 
+- psql -c 'create database "horizon_test";' -U postgres


likewise to the environment variables I think we should remove the static test databases from horizon's test suite before we merge. If possible, switch to the dbtest.Postgres() system we talked about.

as we discussed, I'll include this change in a separate commit once this is merged, will be easier to track the changes for the db-parallelization.

nullstyle · 2017-10-05T18:47:52Z

support/scripts/run_tests

@@ -1,7 +1,7 @@
 #! /bin/bash
 set -e

-TESTS=$(glide novendor | sed 's/^\./github.com\/stellar\/go/g')
+TESTS=$(glide novendor | grep "./services/" | sed 's/^\./github.com\/stellar\/go/g')


I don't believe this is correct: This script should be testing everything in the monorepo but horizon. The command you've added here encodes "only run the services tests", right?

yes, this is incorrect, it should have the -v flag. I added that as a fix in the commit after that.

…e folding

nikhilsaraf · 2017-10-05T23:04:11Z

Looks like the build is passing. This leaves the following changes which I plan to make once this PR is merged to fully complete the changes needed in this repo:

clean up the bash scripts in services/horizon/scripts -- this will unblock deployments for horizon since build_artifacts.bash is one of those scripts
port any changes that were made since I started my port (1 change)
convert tests to use the Postgres() call instead of the Start() call to parallelize the horizon tests and use the existing test script in .travis.yml

These are a lot of commits. Do you want me to clean them up into a fewer number of commits (or one commit) and re-submit with this finalized version?
@nullstyle

nikhilsaraf · 2017-10-05T18:52:09Z

services/horizon/db2/history/ledger_cache_test.go

@@ -11,18 +11,15 @@ func TestLedgerCache(t *testing.T) {
 	defer tt.Finish()
 	q := &Q{tt.HorizonSession()}

-	t.Run("queue and load", func(t *testing.T) {


this was failing in the recent run because it could not recognize t.Run (it should have).
I did not look into fixing it because this is the only case where we use t.Run and it seemed redundant for this test case
please correct me if that's wrong @nullstyle

nikhilsaraf · 2017-10-05T22:47:42Z

support/scripts/run_tests

@@ -1,7 +1,7 @@
 #! /bin/bash
 set -e

-TESTS=$(glide novendor | sed 's/^\./github.com\/stellar\/go/g')
+TESTS=$(glide novendor | grep "./services/" | sed 's/^\./github.com\/stellar\/go/g')


yes, this is incorrect, it should have the -v flag. I added that as a fix in the commit after that.

nikhilsaraf · 2017-10-05T22:51:01Z

.travis.yml

- GO15VENDOREXPERIMENT="1"
+  global:
+  - GO15VENDOREXPERIMENT="1"
+  - DATABASE_URL=postgres://localhost:5432/horizon_test?sslmode=disable&user=postgres


I've removed it in the latest commit, hopefully the build passes with that change.

nikhilsaraf · 2017-10-05T22:54:23Z

.travis.yml

 install:
 - wget https://github.com/Masterminds/glide/releases/download/v0.12.3/glide-v0.12.3-linux-amd64.tar.gz
 - tar -xzvf glide-v0.12.3-linux-amd64.tar.gz
 - cp linux-amd64/glide $GOPATH/bin
 before_script: 
+- psql -c 'create database "horizon_test";' -U postgres


as we discussed, I'll include this change in a separate commit once this is merged, will be easier to track the changes for the db-parallelization.

nikhilsaraf

finished the last bit of changes for this repo by fixing the bash scripts

nikhilsaraf · 2017-10-06T16:46:00Z

services/horizon/scripts/build_artifacts.bash

@@ -1,7 +1,7 @@
 #! /usr/bin/env bash
 set -e

-DIST="dist"
+DIST="$GOPATH/dist"


tested these locally and they work (except dev.bash)

all these follow the same pattern:

add the $GOPATH

convert gb build [-flags] to go install [-flags] github.com/stellar/go/services/horizon/cmd/horizon

convert gb test X to go test X

convert gb generate X to go generate X

nullstyle · 2017-10-06T17:22:09Z

services/horizon/scripts/load_scenario.bash

-CORE_SQL=./src/github.com/stellar/go/services/horizon/test/scenarios/$SCENARIO-core.sql
-HORIZON_SQL=./src/github.com/stellar/go/services/horizon/test/scenarios/$SCENARIO-horizon.sql
+CORE_SQL=$GOPATH/src/github.com/stellar/go/services/horizon/test/scenarios/$SCENARIO-core.sql
+HORIZON_SQL=$GOPATH/src/github.com/stellar/go/services/horizon/test/scenarios/$SCENARIO-horizon.sql


Unfortunately, GOPATH isn't guaranteed to be single-valued, so we can't use it directly. You might look into using a script-relative path, the equivalent of ../../test/scenarios/$SCENARIO-core.sql.

ok, converted all scripts to use a script-relative path, and created a new variable $GOTOP in the scripts where it was needed

nullstyle · 2017-10-06T17:23:32Z

services/horizon/test/db/main.go

@@ -5,8 +5,6 @@ package db

 import (
 	"log"
-	"os"


You might be interested in leveraging https://godoc.org/golang.org/x/tools/cmd/goimports to help avoid this in the future

thanks, will look into integrating this into my IDE.

nikhilsaraf · 2017-10-06T18:03:16Z

updated the scripts. let me know if there's anything else. I'm looking into any changes needed in the stellar/docs repo.

nikhilsaraf added 3 commits October 4, 2017 10:16

1 - copy horizon files into go/services/horizon

158b1f8

2 - update glide.yaml after resolving dependency conflicts + glide.lock

a8f1412

3 - updated import references + non-go files to point to new horizon …

4b05e14

…path (go/services/horizon)

nikhilsaraf added the in progress label Oct 4, 2017

nikhilsaraf requested review from nullstyle and bartekn October 4, 2017 22:34

4 - fix url for web-view links in markdown files

7f29e77

bartekn mentioned this pull request Oct 5, 2017

updated developers repo to reflect the change in horizon's location stellar-deprecated/developers#88

Merged

5 - use older version of goji.io taken from original stellar/go to fi…

a10b449

…x test failures in goji.io/mux ("context" import)

nikhilsaraf added 5 commits October 5, 2017 10:51

6 - run horizon tests via the horizon run_tests.bash script for now t…

d18fd50

…o prevent test failures caused by db parallelism

fix grep command in support/scripts/run_tests

24e4d3e

fold .travis.yml from stellar/horizon into stellar/go

f00af4c

did not add the following lines (build_artifacts.bash is broken right now, let's see if it works without that): before_deploy: - bash scripts/build_artifacts.bash cache: directories: - vendor/src - pkg

fix travis.yml to make it valid

32b009d

update ledger_cache_test.go to remove unused t.Run call

660cc41

nullstyle reviewed Oct 5, 2017

View reviewed changes

nikhilsaraf added 4 commits October 5, 2017 13:13

remove GO15VENDOREXPERIMENT flag and remove 2 redundant files from th…

1841dda

…e folding

added before_deploy section from horizon's .travis.yaml file

cd8beeb

force usage of default test database

b74b78e

fix import unused issue

ca495cc

nikhilsaraf commented Oct 5, 2017

View reviewed changes

fix all bash scripts from horizon merge

c42aa6c

nikhilsaraf commented Oct 6, 2017

View reviewed changes

nullstyle reviewed Oct 6, 2017

View reviewed changes

nikhilsaraf added 2 commits October 6, 2017 10:47

convert GOPATH in scripts to GOTOP

b7b4653

more hardening of bash scripts to make them runnable from any directory

07addb0

nikhilsaraf mentioned this pull request Oct 6, 2017

make "Software" tab in docs point to the new horizon location stellar-deprecated/docs#233

Merged

nullstyle approved these changes Oct 6, 2017

View reviewed changes

nikhilsaraf merged commit 53155ad into stellar:master Oct 6, 2017

nikhilsaraf removed the in progress label Oct 6, 2017

This was referenced Oct 12, 2017

Integer overflow in page_query.go #102

Closed

Build broken stellar-deprecated/developers#89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fold stellar/horizon into stellar/go/services/horizon #93

Fold stellar/horizon into stellar/go/services/horizon #93

nikhilsaraf commented Oct 4, 2017 •

edited

Loading

nikhilsaraf commented Oct 4, 2017

nullstyle commented Oct 5, 2017

nikhilsaraf commented Oct 5, 2017

nullstyle Oct 5, 2017

nullstyle Oct 5, 2017

nullstyle Oct 5, 2017

nikhilsaraf Oct 5, 2017

nullstyle Oct 5, 2017

nikhilsaraf Oct 5, 2017

nullstyle Oct 5, 2017

nikhilsaraf Oct 5, 2017

nikhilsaraf commented Oct 5, 2017

nikhilsaraf Oct 5, 2017

nikhilsaraf Oct 5, 2017

nikhilsaraf Oct 5, 2017

nikhilsaraf Oct 5, 2017

nikhilsaraf left a comment

nikhilsaraf Oct 6, 2017

nullstyle Oct 6, 2017

nikhilsaraf Oct 6, 2017

nullstyle Oct 6, 2017

nikhilsaraf Oct 6, 2017

nikhilsaraf commented Oct 6, 2017

Fold stellar/horizon into stellar/go/services/horizon #93

Fold stellar/horizon into stellar/go/services/horizon #93

Conversation

nikhilsaraf commented Oct 4, 2017 • edited Loading

nikhilsaraf commented Oct 4, 2017

nullstyle commented Oct 5, 2017

nikhilsaraf commented Oct 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhilsaraf commented Oct 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhilsaraf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhilsaraf commented Oct 6, 2017

nikhilsaraf commented Oct 4, 2017 •

edited

Loading