Consul doesn't build on SmartOS #159

Closed
yruss972 opened this Issue May 18, 2014 · 41 comments

Projects

None yet
@yruss972

I tried building Consul on a SmartMachine in Joyent Public Cloud. This is my first time working with go software so please excuse me if I'm missing something that should be obvious.

SmartOS build joyent_20131120T074720Z
Image base64 14.1.0
go version devel +f8b50ad4cac4 Mon Apr 21 17:00:27 2014 -0700 + solaris/amd64
go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="solaris"
GOOS="solaris"
GOPATH="/root/go"
GORACE=""
GOROOT="/opt/local/go"
GOTOOLDIR="/opt/local/go/pkg/tool/solaris_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0"
CXX="g++"
CGO_ENABLED="0"

go get github.com/hashicorp/consul

github.com/armon/gomdb

github.com/armon/gomdb/mdb.c:39 6c: No such file or directory: sys/types.h

cd /root/go/src/github/hashicorp/consul

make

--> Installing build dependencies
--> Running go fmt
command/force_leave_test.go
command/agent/agent_endpoint_test.go
command/agent/catalog_endpoint_test.go
command/agent/dns_test.go
command/agent/health_endpoint_test.go
command/agent/kvs_endpoint_test.go
command/agent/local_test.go
command/agent/rpc_client_test.go
command/agent/status_endpoint_test.go
command/agent/ui_endpoint_test.go
consul/catalog_endpoint_test.go
consul/client_test.go
consul/health_endpoint_test.go
consul/internal_endpoint_test.go
consul/kvs_endpoint_test.go
consul/leader_test.go
consul/server_test.go
testutil/wait.go
--> Installing dependencies to speed up builds...

github.com/armon/gomdb

../../armon/gomdb/mdb.c:39 6c: No such file or directory: sys/types.h
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 2

The sys/types.h file exists in /usr/include

ls -l /usr/include/sys/types.h

-rw-r--r-- 1 root bin 18137 Nov 20 10:08 /usr/include/sys/types.h

I tried modifying GOGCCFLAGS to include "-I/usr/include" but "go env" wouldn't recognize the environment variable or else it reset it :?

I change CGO_ENABLED to 1 and tried make again but got an error which baffles me:

export CGO_ENABLED=1

make

--> Installing build dependencies
--> Running go fmt
--> Installing dependencies to speed up builds...

runtime/cgo

Undefined first referenced
symbol in file
_cgo_sys_thread_start $WORK/runtime/cgo/_obj/gcc_util.o
ld: fatal: symbol referencing errors. No output written to $WORK/runtime/cgo/_obj/cgo.o
collect2: error: ld returned 1 exit status
Makefile:5: recipe for target 'all' failed
make: *** [all] Error 2

@mitchellh
Member

This looks like MDB itself doesn't compile on smartos. Please report a bug upstream to see if they're interested in fixing this. Otherwise, we're looking into alternatives to MDB as well that we may merge in. Until then though, there is no simple way for Consul to fix this. Sorry!

(And note: by MDB I mean the actual MDB source, not gomdb)

@mitchellh mitchellh closed this May 18, 2014
@yruss972

I downloaded mdb from here: https://git.gitorious.org/mdb/mdb.git
It compiled without any issues as far as I can tell.

git clone https://git.gitorious.org/mdb/mdb.git

Cloning into 'mdb'...
remote: Counting objects: 3962, done
remote: Finding sources: 100% (3962/3962)
remote: Compressing objects: 100% (2155/2155)
remote: Compressing objects: 100% (2155/2155)
Receiving objects: 100% (3962/3962), 794.42 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1712/1712), done.
Checking connectivity... done.

cd mdb/libraries/liblmdb

make

gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -fPIC -c mdb.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -fPIC -c midl.c
ar rs liblmdb.a mdb.o midl.o
ar: creating liblmdb.a
gcc -pthread -shared -o liblmdb.so mdb.o midl.o
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mdb_stat.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mdb_stat.o liblmdb.a -o mdb_stat
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mdb_copy.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mdb_copy.o liblmdb.a -o mdb_copy
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest.o liblmdb.a -o mtest
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest2.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest2.o liblmdb.a -o mtest2
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest3.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest3.o liblmdb.a -o mtest3
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest4.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest4.o liblmdb.a -o mtest4
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest5.c
gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest5.o liblmdb.a -o mtest5

make test

mkdir testdb
./mtest && ./mdb_stat testdb
Adding 218 values
20 duplicates skipped
key: fffffd7ffe002fdc 000 , data: fffffd7ffe002fe0 000 0 foo bar
...
Status of Main DB
Tree depth: 2
Branch pages: 1
Leaf pages: 2
Overflow pages: 0
Entries: 62

@mitchellh
Member

Well that is interesting. /cc @armon

Reopening.

@mitchellh mitchellh reopened this May 18, 2014
@yruss972

I don't know if this is helpful but I managed to compile github.com/armon/gomdb without errors using the following command after some digging:

go build -x -compiler gccgo

WORK=/tmp/go-build593558379
mkdir -p $WORK/github.com/armon/gomdb/_obj/
mkdir -p $WORK/github.com/armon/
cd /root/go/src/github.com/armon/gomdb
gcc -Wall -g -I $WORK/github.com/armon/gomdb/_obj/ -I /opt/local/go/pkg/solaris_amd64 -o $WORK/github.com/armon/gomdb/_obj/mdb.o -D GOOS_solaris -D GOARCH_amd64 -m64 -D "GOPKGPATH="github_com_armon_gomdb"" -c ./mdb.c
gcc -Wall -g -I $WORK/github.com/armon/gomdb/_obj/ -I /opt/local/go/pkg/solaris_amd64 -o $WORK/github.com/armon/gomdb/_obj/midl.o -D GOOS_solaris -D GOARCH_amd64 -m64 -D "GOPKGPATH="github_com_armon_gomdb"" -c ./midl.c
ar cru $WORK/github.com/armon/libgomdb.a $WORK/github.com/armon/gomdb/_obj/mdb.o $WORK/github.com/armon/gomdb/_obj/midl.o

It seems that by default it was using a compiler called '6c' so I tried building with -ccflags '-I /usr/include'
That gave me some strange errors:

/usr/include/sys/ccompile.h:42 /usr/include/sys/feature_tests.h:30 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/ccompile.h:49 /usr/include/sys/feature_tests.h:30 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/ccompile.h:62 /usr/include/sys/feature_tests.h:30 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/ccompile.h:86 /usr/include/sys/feature_tests.h:30 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/ccompile.h:96 /usr/include/sys/feature_tests.h:30 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:225 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:227 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:231 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:268 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:271 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: if
/usr/include/sys/isa_defs.h:290 /usr/include/sys/feature_tests.h:31 /usr/include/sys/types.h:33 ../../armon/gomdb/mdb.c:38 unknown #: elif

It seems to be upset about this:
39 /*
40 * Allow for version tests for compiler bugs and features.
41 */
42 #if defined(GNUC)
43 #define GNUC_VERSION
44 (__GNUC
* 10000 + GNUC_MINOR * 100 + GNUC_PATCHLEVEL)
45 #else
46 #define __GNUC_VERSION 0
47 #endif

I tried building everything with gccgo but that didn't work either. It seems some dependencies won't build with gccgo and some won't build with 6c (at least not with the parameters I've tried).

I see that some libraries have been created by gccgo and some by 6c in $GOPATH/pkg but the build process doesn't seem to pay attention to any of these and tries to build everything every time I run it.
Maybe this is standard for go? I really don't know.

@jim80net

I don't know if this helps, but when I iterate through each individual dependency (if go build; else go build -compiler gccgo -x), it appears that:

gccgo successfully compiles:
github.com/armon/gomdb

Neither gccgo nor 6c successfully compile:
github.com/hashicorp/consul/command/agent
github.com/hashicorp/consul/consul # github.com/hashicorp/consul/consul/structs OK

6c compiles:
the rest

@ryanuber
Member

@jim80net not sure if it is related or not, but #168 fixes a build issue with command/agent.

@armon
Member
armon commented May 22, 2014

@jim80net There was a missing file that @ryanuber fixed. Could you try again? I think that the cgo support for SmartOS is limited, which may prevent Consul from compiling.

@jim80net

I've tried ryanuber's branch, but i was unable to determine a difference between it and hashicorp. That said, my knowledge is quite rudimentary, and I apologize that I'm a poor witness to the issue. I think the issue stems from the inability of 6c to compile the github.com/armon/gomdb dependency. In retrospect, I believe the failures for consul/* were an artifact of my incomplete test methodology.

If you'll note how I went wrong:

function myinstall() {
  if go install $1 1>/dev/null 2>&1
  then
     echo "SUCCESS-6c $1"
  elif  go install -compiler gccgo -x $1 1> /dev/null 2>&1
  then
    echo "SUCCESS-gccgo $1"
  else
    echo "FAIL neither $1"
  fi
}

function iterate() {
  echo "Iterating from $(pwd)"
  for i in $(go list -f '{{range .TestImports}}{{.}} {{end}}' ./... )
  do
      myinstall $i
  done
}

[root@consuls1 ~/go/src/github.com/ryanuber/consul]# iterate
Iterating from /root/go/src/github.com/ryanuber/consul
SUCCESS-6c encoding/base64
SUCCESS-6c errors
SUCCESS-6c fmt
FAIL neither github.com/hashicorp/consul/command/agent
FAIL neither github.com/hashicorp/consul/consul
SUCCESS-6c github.com/hashicorp/consul/testutil
SUCCESS-6c github.com/hashicorp/serf/serf
SUCCESS-6c github.com/mitchellh/cli
SUCCESS-6c io
SUCCESS-6c io/ioutil
SUCCESS-6c math/rand
SUCCESS-6c net
SUCCESS-6c os
SUCCESS-6c strings
SUCCESS-6c sync/atomic
SUCCESS-6c testing
SUCCESS-6c time
SUCCESS-6c bytes
SUCCESS-6c encoding/base64
SUCCESS-6c encoding/json
SUCCESS-6c errors
SUCCESS-6c flag
SUCCESS-6c fmt
FAIL neither github.com/hashicorp/consul/consul
SUCCESS-6c github.com/hashicorp/consul/consul/structs
SUCCESS-6c github.com/hashicorp/consul/testutil
SUCCESS-6c github.com/hashicorp/logutils
SUCCESS-6c github.com/hashicorp/serf/serf
SUCCESS-6c github.com/miekg/dns
SUCCESS-6c github.com/mitchellh/cli
SUCCESS-6c io
SUCCESS-6c io/ioutil
SUCCESS-6c log
SUCCESS-6c net
SUCCESS-6c net/http
SUCCESS-6c net/http/httptest
SUCCESS-6c os
SUCCESS-6c path/filepath
SUCCESS-6c reflect
SUCCESS-6c strconv
SUCCESS-6c strings
SUCCESS-6c sync/atomic
SUCCESS-6c testing
SUCCESS-6c time
SUCCESS-6c bytes
SUCCESS-6c crypto/tls
SUCCESS-6c errors
SUCCESS-6c fmt
SUCCESS-gccgo github.com/armon/gomdb
SUCCESS-6c github.com/hashicorp/consul/consul/structs
SUCCESS-6c github.com/hashicorp/consul/testutil
SUCCESS-6c github.com/hashicorp/raft
SUCCESS-6c github.com/hashicorp/serf/serf
SUCCESS-6c github.com/ugorji/go/codec
SUCCESS-6c io/ioutil
SUCCESS-6c net
SUCCESS-6c net/rpc
SUCCESS-6c os
SUCCESS-6c reflect
SUCCESS-6c regexp
SUCCESS-6c sort
SUCCESS-6c strings
SUCCESS-6c testing
SUCCESS-6c time

Because "go install/build" will use the same compiler for each dependency, the hashicorp fails are simply because the whole set will not compile with just 6c nor with just gccgo.

@armon
Member
armon commented Jun 11, 2014

I'm closing this issue since we cannot fix this until Go supports CGO on SmartOS upstream.

@armon armon closed this Jun 11, 2014
@MerlinDMC
Contributor

As a sidenote:

For a golang toolchain on Illumos/SmartOS with cgo wesolows has been doing some great work:
https://github.com/wesolows/golang-old (1.4)
https://github.com/wesolows/golang (1.5)
(You'll need a SmartOS host that is from December 2014 or newer because of some fixes in the system includes)

Those repositories have cgo support patched in and work well so far. (I'm running consul on SmartOS)

@jim80net
jim80net commented Jan 6, 2015

Awesome. Thanks for the update MerlinDMC!

@klynton
klynton commented Mar 12, 2015

With the release of 2014Q4 Consul will now build on the base* boxes.

@cmacrae
cmacrae commented Nov 28, 2015

Has anyone had any luck building Consul on recent SmartOS base images?
CGO is now supported on SmartOS and Go 1.5.1 is in Joyent's official repo's:

$ go version
go version go1.5.1 solaris/amd64
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="solaris"
GOOS="solaris"
GOPATH=""
GORACE=""
GOROOT="/opt/local/go"
GOTOOLDIR="/opt/local/go/pkg/tool/solaris_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

I'm writing an Ansible role to deploy/orchestrate Consul, and my primary development platform is SmartOS. When trying to build, as instructed in the README, the following errors are thrown:

# github.com/fsouza/go-dockerclient/external/github.com/Sirupsen/logrus
../../fsouza/go-dockerclient/external/github.com/Sirupsen/logrus/text_formatter.go:28: undefined: IsTerminal
# github.com/fsouza/go-dockerclient/external/github.com/docker/docker/pkg/system
../../fsouza/go-dockerclient/external/github.com/docker/docker/pkg/system/stat_unsupported.go:16: s.Mtimespec undefined (type *syscall.Stat_t has no field or method Mtimespec)
# github.com/mitchellh/cli
../../mitchellh/cli/ui.go:79: undefined: terminal.IsTerminal
../../mitchellh/cli/ui.go:81: undefined: terminal.ReadPassword
@slackpad
Contributor

Hi @cmacrae if you are building the latest master then there's no longer a CGO dependency, but it looks like there's no Solaris support for those terminal functions (see golang/go#13085, which links to several issues). I think at the current time you'd have to do some surgery on those dependencies in order to build on this platform. For the CLI dependency you could probably remove the IsTerminal branch of that if in ui.go since Consul doesn't use that. For Docker I'm less sure, though you could take out Docker check support from Consul's command/agent/check.go (and unit test) since that's the only place it is used. Sorry this is ugly at the moment, but it looks like things are still being sorted out on the Go side.

@cmacrae
cmacrae commented Nov 29, 2015

Thanks @slackpad! Managed to get it to build with your tips ๐Ÿ‘
I'll break this out into an optional step in my Ansible role using a patch from the changes I made to get it to build.

I'll likely be doing automated builds, and host a zipped archive for the amd64 binary for Solaris (more specifically, illumos/SmartOS) systems on my personal domain: http://cmacr.ae/consul/solaris_amd64.zip
Mostly for my own convenience, but, others can grab it too, if they so wish :)

EDIT: Chucked a quick page together for builds: http://cmacr.ae/consul/

@doublerebel

@cmacrae thanks for this, very helpful.

I would like to get Consul into pkgsrc. Is there anyone here who can help fasttrack the process?

@cmacrae
cmacrae commented Dec 6, 2015

That'd be great! @jperkin would be the best person to talk to regarding
this. But I'm not sure how "ethical" the patches are that I'm using.

On Sat, 5 Dec 2015 at 17:15, Charles Phillips notifications@github.com
wrote:

@cmacrae https://github.com/cmacrae thanks for this, very helpful.

I would like to get Consul into pkgsrc
https://pkgsrc.joyent.com/docs/creating/, I see hashicorp/vault
https://github.com/hashicorp/vault is already present
http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/hs-vault/README.html.
Is there anyone here who can help fasttrack the process?

โ€”
Reply to this email directly or view it on GitHub
#159 (comment).

@benjamin-bergia

I would also be interested by a package for smartos.
By the way @cmacrae I gave a try to your binary and it works seamlessly on 15.3.0.
Thank you for sharing.

@cmacrae
cmacrae commented Dec 7, 2015

No problem @benjamin-bergia :)
I could make a pkgsrc port for it, and submit it for review. But, like I say, I'm not sure how reasonable the patches are.

@dekobon
dekobon commented Dec 10, 2015

Is it possible to get an official downloadable binary available on the Consul site for SmartOS before it gets added to pkgsrc?

@cmacrae
cmacrae commented Dec 10, 2015

@dekobon I doubt it, SmartOS is not an officially supported platform for Consul

@doublerebel

@cmacrae I've spoken to my contact at Joyent, we may be able to get Consul in sooner than later.

@cmacrae
cmacrae commented Dec 10, 2015

@doublerebel Awesome news!

@dekobon
dekobon commented Dec 10, 2015

@cmacrae Who's your contact at Joyent? I'm also with Joyent.

@dekobon
dekobon commented Dec 10, 2015

@doublerebel - sorry the previous comment was for you.

@doublerebel

@dekobon Oh excellent! Robby Andrews. Mentioned that some of you were interested in Consul inclusion, so he was right :)

@Chewyrobbo

@doublerebel have you hung out on #consul on IRC yet?

@doublerebel

@Chewyrobbo no, just #smartos and #joyent. Just joined #consul now. Thanks!

@mamash
mamash commented Dec 11, 2015

@cmacrae Can you share the patches, any way you're comfortable with? I'm with Joyent and working on pkgsrc.

@cmacrae
cmacrae commented Dec 11, 2015

@mamash Sure :) Just on my way into the office at the moment, I'll share them once I'm in

@cmacrae
cmacrae commented Dec 11, 2015

@mamash So, patches are necessary to both mitchellh/cli & hashicorp/consul.

Here are the patch files, pretty much just stripping stuff out:

@MerlinDMC
Contributor

@cmacrae looks like your patch to cli will not ask for input via stdin anymore if secret=true. You removed the bit that does read the input without replacing it.

@cmacrae
cmacrae commented Dec 11, 2015

@MerlinDMC Thanks for pointing this out - these patches were chucked together quickly one evening after work, and I haven't had time to review them properly. Given the fact that this now seems to be moving toward to an official pkgsrc port at Joyent, I trust that those involved will address any issues posed by my hasty patching ;)

@MerlinDMC
Contributor

https://gist.github.com/MerlinDMC/57d6043067e0bad8e2af

I'm not happy with the ReadPassword() ... and I hate the fact that the IOCTL syscalls are used everywhere and not implemented on Illumos but this patches $GOPATH/src tree does provide a consul binary that can run as an agent at least - I didn't test much more.

@doublerebel

@MerlinDMC thanks for this, I'm able to get hashicorp/vault almost built as well. Looking now at patching both vault and consul to use bgentry/speakeasy for cross-platform getpasswd support.

@mamash
mamash commented Dec 15, 2015

I was able to get a crude pkgsrc package done with the above patches, but it still requires one of the patches applied manually.

In order to land in pkgsrc though, I will need to break out all the 30+ Go dependencies into their separate packages, because pkgsrc doesn't like build-time source fetching and the embedded deps files would clash over PLIST entries with other Go packages as support for such grows in pkgsrc.

@doublerebel

@mamash that's a great start, thanks for the help. I'm also working on vault and consul-template, which have similar and related dependencies. Is there anything we can do about the projects' structure to make the builds simpler? When compiled, the end result is just one binary that needs one config file.

@slackpad
Contributor
slackpad commented Jan 5, 2016

Hey all - it looks like @jen20 got a CLI change merged yesterday that fixes the syscall issue:

mitchellh/cli#25

And the Docker client must have gotten fixed upstream:

https://twitter.com/jen20/status/684450871606939648

I'll take this and add solaris to the list of built flavors.

@slackpad slackpad reopened this Jan 5, 2016
@slackpad slackpad self-assigned this Jan 5, 2016
@slackpad slackpad added the enhancement label Jan 5, 2016
@slackpad slackpad closed this in #1568 Jan 6, 2016
@cmacrae
cmacrae commented Jan 6, 2016

Awesome! Thanks @slackpad & @jen20 ๐Ÿ‘

@mamash
mamash commented Jan 7, 2016

Thanks guys! I updated the pkgsrc definition for Consul, it will show up in the standard Joyent 2015Q4 build (soon).

It lives here (WIP for now):

https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=tree;f=consul;hb=HEAD

@0-wiz-0 0-wiz-0 pushed a commit to NetBSD/pkgsrc-wip that referenced this issue Jan 7, 2016
@mamash mamash Update Consul deps, this lets us get rid of the patches, as changes have
been implemented with the respective upstreams, some mentioned in the
original SmartOS support ticket:

  hashicorp/consul#159
e96163e
@cmacrae
cmacrae commented Jan 7, 2016

Excellent, thanks @mamash!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment