Cannot build from source

[pataquets]

I want to create a build-from-source Docker image to make easier to start hacking.
My Dockerfile is:

FROM golang

COPY . /go/src/github.com/dataux/dataux/
WORKDIR /go/src/github.com/dataux/dataux/

RUN \
  go build

When I run docker build -t dataux ., it always fails this way:

backends/bigquery/resultreader.go:7:2: cannot find package "cloud.google.com/go/bigquery" in any of:
	/usr/local/go/src/cloud.google.com/go/bigquery (from $GOROOT)
	/go/src/cloud.google.com/go/bigquery (from $GOPATH)
backends/bigtable/resultreader.go:10:2: cannot find package "cloud.google.com/go/bigtable" in any of:
	/usr/local/go/src/cloud.google.com/go/bigtable (from $GOROOT)
	/go/src/cloud.google.com/go/bigtable (from $GOPATH)
backends/bigquery/resultreader.go:8:2: cannot find package "cloud.google.com/go/civil" in any of:
	/usr/local/go/src/cloud.google.com/go/civil (from $GOROOT)
	/go/src/cloud.google.com/go/civil (from $GOPATH)
backends/datastore/datasource.go:13:2: cannot find package "cloud.google.com/go/datastore" in any of:
	/usr/local/go/src/cloud.google.com/go/datastore (from $GOROOT)
	/go/src/cloud.google.com/go/datastore (from $GOPATH)
backends/bigtable/resultreader.go:11:2: cannot find package "github.com/araddon/dateparse" in any of:
	/usr/local/go/src/github.com/araddon/dateparse (from $GOROOT)
	/go/src/github.com/araddon/dateparse (from $GOPATH)
main.go:27:2: cannot find package "github.com/araddon/gou" in any of:
	/usr/local/go/src/github.com/araddon/gou (from $GOROOT)
	/go/src/github.com/araddon/gou (from $GOPATH)
backends/bigquery/resultreader.go:13:2: cannot find package "github.com/araddon/qlbridge/datasource" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/datasource (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/datasource (from $GOPATH)
main.go:14:2: cannot find package "github.com/araddon/qlbridge/datasource/files" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/datasource/files (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/datasource/files (from $GOPATH)
backends/bigquery/resultreader.go:14:2: cannot find package "github.com/araddon/qlbridge/exec" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/exec (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/exec (from $GOPATH)
backends/bigquery/sql_to_bq.go:19:2: cannot find package "github.com/araddon/qlbridge/expr" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/expr (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/expr (from $GOPATH)
vendored/mixer/proxy/handler_sharded.go:17:2: cannot find package "github.com/araddon/qlbridge/expr/builtins" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/expr/builtins (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/expr/builtins (from $GOPATH)
backends/bigtable/sql_to_bt.go:18:2: cannot find package "github.com/araddon/qlbridge/lex" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/lex (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/lex (from $GOPATH)
backends/bigquery/sql_to_bq.go:20:2: cannot find package "github.com/araddon/qlbridge/plan" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/plan (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/plan (from $GOPATH)
backends/bigquery/resultreader.go:15:2: cannot find package "github.com/araddon/qlbridge/rel" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/rel (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/rel (from $GOPATH)
backends/bigquery/source.go:19:2: cannot find package "github.com/araddon/qlbridge/schema" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/schema (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/schema (from $GOPATH)
backends/bigquery/resultreader.go:16:2: cannot find package "github.com/araddon/qlbridge/value" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/value (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/value (from $GOPATH)
backends/bigtable/sql_to_bt.go:23:2: cannot find package "github.com/araddon/qlbridge/vm" in any of:
	/usr/local/go/src/github.com/araddon/qlbridge/vm (from $GOROOT)
	/go/src/github.com/araddon/qlbridge/vm (from $GOPATH)
planner/plan_server.go:14:2: cannot find package "github.com/coreos/etcd/clientv3" in any of:
	/usr/local/go/src/github.com/coreos/etcd/clientv3 (from $GOROOT)
	/go/src/github.com/coreos/etcd/clientv3 (from $GOPATH)
backends/cassandra/source.go:11:2: cannot find package "github.com/gocql/gocql" in any of:
	/usr/local/go/src/github.com/gocql/gocql (from $GOROOT)
	/go/src/github.com/gocql/gocql (from $GOPATH)
planner/msgs.pb.go:18:8: cannot find package "github.com/golang/protobuf/proto" in any of:
	/usr/local/go/src/github.com/golang/protobuf/proto (from $GOROOT)
	/go/src/github.com/golang/protobuf/proto (from $GOPATH)
backends/cassandra/source.go:12:2: cannot find package "github.com/hailocab/go-hostpool" in any of:
	/usr/local/go/src/github.com/hailocab/go-hostpool (from $GOROOT)
	/go/src/github.com/hailocab/go-hostpool (from $GOPATH)
vendored/mixer/proxy/handler_sharded.go:15:2: cannot find package "github.com/kr/pretty" in any of:
	/usr/local/go/src/github.com/kr/pretty (from $GOROOT)
	/go/src/github.com/kr/pretty (from $GOPATH)
models/config.go:8:2: cannot find package "github.com/lytics/confl" in any of:
	/usr/local/go/src/github.com/lytics/confl (from $GOROOT)
	/go/src/github.com/lytics/confl (from $GOPATH)
planner/sqlactor.go:11:2: cannot find package "github.com/lytics/dfa" in any of:
	/usr/local/go/src/github.com/lytics/dfa (from $GOROOT)
	/go/src/github.com/lytics/dfa (from $GOPATH)
backends/lytics/resultreader.go:9:2: cannot find package "github.com/lytics/go-lytics" in any of:
	/usr/local/go/src/github.com/lytics/go-lytics (from $GOROOT)
	/go/src/github.com/lytics/go-lytics (from $GOPATH)
planner/leader.go:10:2: cannot find package "github.com/lytics/grid/grid.v3" in any of:
	/usr/local/go/src/github.com/lytics/grid/grid.v3 (from $GOROOT)
	/go/src/github.com/lytics/grid/grid.v3 (from $GOPATH)
planner/plan_server.go:16:2: cannot find package "github.com/sony/sonyflake" in any of:
	/usr/local/go/src/github.com/sony/sonyflake (from $GOROOT)
	/go/src/github.com/sony/sonyflake (from $GOPATH)
backends/bigquery/resultreader.go:10:2: cannot find package "golang.org/x/net/context" in any of:
	/usr/local/go/src/golang.org/x/net/context (from $GOROOT)
	/go/src/golang.org/x/net/context (from $GOPATH)
backends/datastore/datasource.go:16:2: cannot find package "golang.org/x/oauth2/google" in any of:
	/usr/local/go/src/golang.org/x/oauth2/google (from $GOROOT)
	/go/src/golang.org/x/oauth2/google (from $GOPATH)
backends/datastore/datasource.go:17:2: cannot find package "golang.org/x/oauth2/jwt" in any of:
	/usr/local/go/src/golang.org/x/oauth2/jwt (from $GOROOT)
	/go/src/golang.org/x/oauth2/jwt (from $GOPATH)
backends/bigquery/resultreader.go:11:2: cannot find package "google.golang.org/api/iterator" in any of:
	/usr/local/go/src/google.golang.org/api/iterator (from $GOROOT)
	/go/src/google.golang.org/api/iterator (from $GOPATH)
backends/datastore/datasource.go:19:2: cannot find package "google.golang.org/api/option" in any of:
	/usr/local/go/src/google.golang.org/api/option (from $GOROOT)
	/go/src/google.golang.org/api/option (from $GOPATH)
backends/mongo/mgo_results.go:9:2: cannot find package "gopkg.in/mgo.v2" in any of:
	/usr/local/go/src/gopkg.in/mgo.v2 (from $GOROOT)
	/go/src/gopkg.in/mgo.v2 (from $GOPATH)
backends/mongo/mgo_results.go:10:2: cannot find package "gopkg.in/mgo.v2/bson" in any of:
	/usr/local/go/src/gopkg.in/mgo.v2/bson (from $GOROOT)
	/go/src/gopkg.in/mgo.v2/bson (from $GOPATH)
backends/kubernetes/client.go:14:2: cannot find package "k8s.io/apimachinery/pkg/apis/meta/v1" in any of:
	/usr/local/go/src/k8s.io/apimachinery/pkg/apis/meta/v1 (from $GOROOT)
	/go/src/k8s.io/apimachinery/pkg/apis/meta/v1 (from $GOPATH)
backends/kubernetes/source.go:15:2: cannot find package "k8s.io/client-go/kubernetes" in any of:
	/usr/local/go/src/k8s.io/client-go/kubernetes (from $GOROOT)
	/go/src/k8s.io/client-go/kubernetes (from $GOPATH)
backends/kubernetes/client.go:15:2: cannot find package "k8s.io/client-go/pkg/api/v1" in any of:
	/usr/local/go/src/k8s.io/client-go/pkg/api/v1 (from $GOROOT)
	/go/src/k8s.io/client-go/pkg/api/v1 (from $GOPATH)
backends/kubernetes/source.go:16:2: cannot find package "k8s.io/client-go/rest" in any of:
	/usr/local/go/src/k8s.io/client-go/rest (from $GOROOT)
	/go/src/k8s.io/client-go/rest (from $GOPATH)
backends/kubernetes/source.go:17:2: cannot find package "k8s.io/client-go/tools/clientcmd" in any of:
	/usr/local/go/src/k8s.io/client-go/tools/clientcmd (from $GOROOT)
	/go/src/k8s.io/client-go/tools/clientcmd (from $GOPATH)

I haven't found any build specific info elsewhere in the repo.

Also, a build-from-source Dockerfile would allow setting up an Automatic Build and would be a very convenient way of providing both tagged and daily builds automatically on each push, with the added benefit of better audit and traceability of the source used to build the Docker image. For some people, images pushed in a 'black box' way are a deal breaker for security reasons. I would happily send a PR if that is deemed a worthwhile improvement on the current Dockerfile.

Thanks in advance.

[araddon]

Sorry about that, let me get it 1) vendored and 2) published as docker-images. (which kinda exists, but doc sucks) https://github.com/dataux/dataux/blob/master/backends/kubernetes/conf/dataux.yaml

I have been converting all the schema-discovery to allow for this approach which supports using dynamic schema registry using CREATE SOURCE statements.

https://github.com/dataux/dataux/tree/master/backends/lytics

[pataquets]

Nothing to be sorry of :) Sounds perfectly fine to me.
I just want to have a built-from-source Docker Image to start testing Dataux. Not sure if hacking will ever come, since I'm still a Go newbie, but if it can also help others get on board, that would be great.
Are you considering creating a Docker Hub 'official' image? Better yet, one built straight from source (Tip: the username 'dataux' is available on Docker Hub).

Thanks for making this software available and for the superfast reply!

[araddon]

Ok, i have been having a horrible time getting all the dependencies to work. The kubernetes client/ etcd client just use different versions of grpc. So, i disabled kubernetes source for now, pushed up some changes and added dep files. So, it is working for me using dep (and go 1.9.1), also pushed up a newer docker container. I updated readme home page to hopefully help as well.

# I am not currently checking in /vendor until i figure out a way to 
# possibly move the main.go to separate repo so this can still be used as library
dep ensure -v
go build

docker run -e "GOOGLE_APPLICATION_CREDENTIALS=/.config/gcloud/application_default_credentials.json" \
  -e "LOGGING=debug" \
  --rm -it \
  -p 4000:4000 \
  -v ~/.config/gcloud:/.config/gcloud \
  gcr.io/dataux-io/dataux:latest


mysql -h127.0.0.1 -P4000

show databases;

use baseball;

select * from appearances limit 10;

[pataquets]

After adding the corresponding go get for dep, now it works. Thank you!
Now I've managed to create a Docker image built straight from source.