New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crossbuilding with xgo #708

Closed
endorama opened this Issue Jul 2, 2018 · 18 comments

Comments

Projects
None yet
5 participants
@endorama

endorama commented Jul 2, 2018

Hello, first of all a big thank you for this project. Is awesome and works like a charm.

Is your feature request related to a problem? Please describe.
I'm having some issues cross-compiling a go package ( github.com/99designs/keyring )

Trying to cross-build from linux to osx I needed to use xgo. This is due to the use of the keychain osx feature ( no idea why it doesn't compile with CGO, I'm still a golang newbie ).

I tried looking at the docs and the repo ( issues and code ) but wasn't able to find a solution.

Building with CGO does not seems to work ( I'm waiting some more informations on how to cross build that library 99designs/keyring#23 )

Describe the solution you'd like
I'd like to use xgo to build the package.

The command I'm using is:

$ xgo \
    --targets="darwin/amd64,linux/amd64" \
    --dest=dist \
    github.com/99designs/keyring

Describe alternatives you've considered
Doing it manually, reproducing the goreleaser steps ( build, sign, release ).

Thank you!

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 2, 2018

Member

you don't need xgo, you just need to disable cgo.

Take a look at goreleaser's goreleaser.yml file: https://github.com/goreleaser/goreleaser/blob/master/.goreleaser.yml#L5-L6

Let me know if it doesn't work!

Cheers

GitHub
goreleaser - Deliver Go binaries as fast and easily as possible
Member

caarlos0 commented Jul 2, 2018

you don't need xgo, you just need to disable cgo.

Take a look at goreleaser's goreleaser.yml file: https://github.com/goreleaser/goreleaser/blob/master/.goreleaser.yml#L5-L6

Let me know if it doesn't work!

Cheers

GitHub
goreleaser - Deliver Go binaries as fast and easily as possible

@caarlos0 caarlos0 closed this Jul 2, 2018

@caarlos0 caarlos0 added the invalid label Jul 2, 2018

@endorama

This comment has been minimized.

Show comment
Hide comment
@endorama

endorama Jul 2, 2018

Hi @caarlos0 thanks for the quick reply. Unfortunately does not seem to work.

.goreleaser.yml

# .goreleaser.yml
# Build customization
builds:
- binary: two-factor-authenticator
  env:
    - CGO_ENABLED=0
  goos:
    - darwin
    - linux
  goarch:
    - amd64

output

$ goreleaser release --skip-publish --rm-dist --snapshot

   • releasing using goreleaser 0.79.0...
   • loading config file       file=.goreleaser.yml
   • SETTING DEFAULTS FOR:
      • loading environment variables
      • snapshoting              
      • releasing to GitHub      
      • project name             
      • creating archives        
      • building binaries        
      • creating Linux packages with fpm
      • creating Linux packages with nfpm
      • creating Linux packages with snapcraft
      • calculating checksums    
      • signing artifacts        
      • creating Docker images   
      • releasing to Artifactory 
      • releasing to s3          
      • creating homebrew formula
      • creating Scoop Manifest  
   • RUNNING BEFORE HOOKS
   • CHECKING ./DIST  
      • --rm-dist is set, cleaning it up
   • GETTING AND VALIDATING GIT STATE
      • releasing v0.1.0-beta.2, commit 1a149cbc10e15a2db1de167576265f5b78bf1ec4
      • skipped                   reason=disabled during snapshot mode
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • skipped                   reason=not available for snapshots
   • LOADING ENVIRONMENT VARIABLES
      • skipped                   reason=publishing is disabled
   • BUILDING BINARIES
      • building                  binary=dist/linux_amd64/two-factor-authenticator
      • building                  binary=dist/darwin_amd64/two-factor-authenticator
      • added new artifact        name=two-factor-authenticator path=dist/linux_amd64/two-factor-authenticator type=Binary
   ⨯ release failed after 0.63s error=failed to build for darwin_amd64: # github.com/endorama/two-factor-authenticator/vendor/github.com/99designs/keyring
vendor/github.com/99designs/keyring/keychain.go:39:11: undefined: keychain.NewItem
vendor/github.com/99designs/keyring/keychain.go:40:20: undefined: keychain.SecClassGenericPassword
vendor/github.com/99designs/keyring/keychain.go:43:22: undefined: keychain.MatchLimitOne
vendor/github.com/99designs/keyring/keychain.go:49:28: undefined: keychain.NewWithPath
vendor/github.com/99designs/keyring/keychain.go:53:18: undefined: keychain.QueryItem
vendor/github.com/99designs/keyring/keychain.go:54:12: undefined: keychain.ErrorItemNotFound
vendor/github.com/99designs/keyring/keychain.go:76:9: undefined: keychain.Keychain
vendor/github.com/99designs/keyring/keychain.go:87:12: undefined: keychain.NewItem
vendor/github.com/99designs/keyring/keychain.go:88:21: undefined: keychain.SecClassGenericPassword
vendor/github.com/99designs/keyring/keychain.go:199:36: undefined: keychain.Keychain
vendor/github.com/99designs/keyring/keychain.go:88:21: too many errors

Issue seems in the 99designs/keyring package importing keybase/go-keychain.

I go getted keybase/go-keychain ( both in GOPATH and in vendor/ ) but the build still fails.

I suspect this happen because keybase/go-keychain effectively requires CGO?

endorama commented Jul 2, 2018

Hi @caarlos0 thanks for the quick reply. Unfortunately does not seem to work.

.goreleaser.yml

# .goreleaser.yml
# Build customization
builds:
- binary: two-factor-authenticator
  env:
    - CGO_ENABLED=0
  goos:
    - darwin
    - linux
  goarch:
    - amd64

output

$ goreleaser release --skip-publish --rm-dist --snapshot

   • releasing using goreleaser 0.79.0...
   • loading config file       file=.goreleaser.yml
   • SETTING DEFAULTS FOR:
      • loading environment variables
      • snapshoting              
      • releasing to GitHub      
      • project name             
      • creating archives        
      • building binaries        
      • creating Linux packages with fpm
      • creating Linux packages with nfpm
      • creating Linux packages with snapcraft
      • calculating checksums    
      • signing artifacts        
      • creating Docker images   
      • releasing to Artifactory 
      • releasing to s3          
      • creating homebrew formula
      • creating Scoop Manifest  
   • RUNNING BEFORE HOOKS
   • CHECKING ./DIST  
      • --rm-dist is set, cleaning it up
   • GETTING AND VALIDATING GIT STATE
      • releasing v0.1.0-beta.2, commit 1a149cbc10e15a2db1de167576265f5b78bf1ec4
      • skipped                   reason=disabled during snapshot mode
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • skipped                   reason=not available for snapshots
   • LOADING ENVIRONMENT VARIABLES
      • skipped                   reason=publishing is disabled
   • BUILDING BINARIES
      • building                  binary=dist/linux_amd64/two-factor-authenticator
      • building                  binary=dist/darwin_amd64/two-factor-authenticator
      • added new artifact        name=two-factor-authenticator path=dist/linux_amd64/two-factor-authenticator type=Binary
   ⨯ release failed after 0.63s error=failed to build for darwin_amd64: # github.com/endorama/two-factor-authenticator/vendor/github.com/99designs/keyring
vendor/github.com/99designs/keyring/keychain.go:39:11: undefined: keychain.NewItem
vendor/github.com/99designs/keyring/keychain.go:40:20: undefined: keychain.SecClassGenericPassword
vendor/github.com/99designs/keyring/keychain.go:43:22: undefined: keychain.MatchLimitOne
vendor/github.com/99designs/keyring/keychain.go:49:28: undefined: keychain.NewWithPath
vendor/github.com/99designs/keyring/keychain.go:53:18: undefined: keychain.QueryItem
vendor/github.com/99designs/keyring/keychain.go:54:12: undefined: keychain.ErrorItemNotFound
vendor/github.com/99designs/keyring/keychain.go:76:9: undefined: keychain.Keychain
vendor/github.com/99designs/keyring/keychain.go:87:12: undefined: keychain.NewItem
vendor/github.com/99designs/keyring/keychain.go:88:21: undefined: keychain.SecClassGenericPassword
vendor/github.com/99designs/keyring/keychain.go:199:36: undefined: keychain.Keychain
vendor/github.com/99designs/keyring/keychain.go:88:21: too many errors

Issue seems in the 99designs/keyring package importing keybase/go-keychain.

I go getted keybase/go-keychain ( both in GOPATH and in vendor/ ) but the build still fails.

I suspect this happen because keybase/go-keychain effectively requires CGO?

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 2, 2018

Member

if it requires CGO, I think you'll need to explicitly enable it (change 0 to 1)... can you try that as well?

Member

caarlos0 commented Jul 2, 2018

if it requires CGO, I think you'll need to explicitly enable it (change 0 to 1)... can you try that as well?

@endorama

This comment has been minimized.

Show comment
Hide comment
@endorama

endorama Jul 3, 2018

Still no luck unfortunately.

.goreleaser.yml

# .goreleaser.yml
# Build customization
builds:
- binary: two-factor-authenticator
  env:
    - CGO_ENABLED=1
  goos:
    - darwin
    - linux
  goarch:
    - amd64

output

$ goreleaser release --skip-publish --rm-dist --snapshot

   • releasing using goreleaser 0.79.0...
   • loading config file       file=.goreleaser.yml
   • SETTING DEFAULTS FOR:
      • loading environment variables
      • snapshoting              
      • releasing to GitHub      
      • project name             
      • creating archives        
      • building binaries        
      • creating Linux packages with fpm
      • creating Linux packages with nfpm
      • creating Linux packages with snapcraft
      • calculating checksums    
      • signing artifacts        
      • creating Docker images   
      • releasing to Artifactory 
      • releasing to s3          
      • creating homebrew formula
      • creating Scoop Manifest  
   • RUNNING BEFORE HOOKS
   • CHECKING ./DIST  
      • --rm-dist is set, cleaning it up
   • GETTING AND VALIDATING GIT STATE
      • releasing v0.1.0-beta.2, commit 1a149cbc10e15a2db1de167576265f5b78bf1ec4
      • skipped                   reason=disabled during snapshot mode
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • skipped                   reason=not available for snapshots
   • LOADING ENVIRONMENT VARIABLES
      • skipped                   reason=publishing is disabled
   • BUILDING BINARIES
      • building                  binary=dist/linux_amd64/two-factor-authenticator
      • building                  binary=dist/darwin_amd64/two-factor-authenticator
      • added new artifact        name=two-factor-authenticator path=dist/linux_amd64/two-factor-authenticator type=Binary
   ⨯ release failed after 1.16s error=failed to build for darwin_amd64: # os/user
/.../goenv/versions/1.10/src/os/user/getgrouplist_darwin.go: In function ‘mygetgrouplist’:
/.../goenv/versions/1.10/src/os/user/getgrouplist_darwin.go:14:11: warning: implicit declaration of function ‘getgrouplist’ [-Wimplicit-function-declaration]
  int rv = getgrouplist(user, (int) group, buf, ngroups);
           ^
# .../vendor/github.com/keybase/go-keychain
vendor/github.com/keybase/go-keychain/corefoundation_1.10.go:9:43: fatal error: CoreFoundation/CoreFoundation.h: No such file or directory
compilation terminated.
# net
/.../goenv/versions/1.10/src/net/cgo_bsd.go:15:72: could not determine kind of name for C.AI_MASK

Any idea? Thank you for the support so far!

endorama commented Jul 3, 2018

Still no luck unfortunately.

.goreleaser.yml

# .goreleaser.yml
# Build customization
builds:
- binary: two-factor-authenticator
  env:
    - CGO_ENABLED=1
  goos:
    - darwin
    - linux
  goarch:
    - amd64

output

$ goreleaser release --skip-publish --rm-dist --snapshot

   • releasing using goreleaser 0.79.0...
   • loading config file       file=.goreleaser.yml
   • SETTING DEFAULTS FOR:
      • loading environment variables
      • snapshoting              
      • releasing to GitHub      
      • project name             
      • creating archives        
      • building binaries        
      • creating Linux packages with fpm
      • creating Linux packages with nfpm
      • creating Linux packages with snapcraft
      • calculating checksums    
      • signing artifacts        
      • creating Docker images   
      • releasing to Artifactory 
      • releasing to s3          
      • creating homebrew formula
      • creating Scoop Manifest  
   • RUNNING BEFORE HOOKS
   • CHECKING ./DIST  
      • --rm-dist is set, cleaning it up
   • GETTING AND VALIDATING GIT STATE
      • releasing v0.1.0-beta.2, commit 1a149cbc10e15a2db1de167576265f5b78bf1ec4
      • skipped                   reason=disabled during snapshot mode
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • skipped                   reason=not available for snapshots
   • LOADING ENVIRONMENT VARIABLES
      • skipped                   reason=publishing is disabled
   • BUILDING BINARIES
      • building                  binary=dist/linux_amd64/two-factor-authenticator
      • building                  binary=dist/darwin_amd64/two-factor-authenticator
      • added new artifact        name=two-factor-authenticator path=dist/linux_amd64/two-factor-authenticator type=Binary
   ⨯ release failed after 1.16s error=failed to build for darwin_amd64: # os/user
/.../goenv/versions/1.10/src/os/user/getgrouplist_darwin.go: In function ‘mygetgrouplist’:
/.../goenv/versions/1.10/src/os/user/getgrouplist_darwin.go:14:11: warning: implicit declaration of function ‘getgrouplist’ [-Wimplicit-function-declaration]
  int rv = getgrouplist(user, (int) group, buf, ngroups);
           ^
# .../vendor/github.com/keybase/go-keychain
vendor/github.com/keybase/go-keychain/corefoundation_1.10.go:9:43: fatal error: CoreFoundation/CoreFoundation.h: No such file or directory
compilation terminated.
# net
/.../goenv/versions/1.10/src/net/cgo_bsd.go:15:72: could not determine kind of name for C.AI_MASK

Any idea? Thank you for the support so far!

@thapakazi

This comment has been minimized.

Show comment
Hide comment
@thapakazi

thapakazi Jul 4, 2018

👍 @endorama there there 😅
weird, I have the same requirements like you.

I need to do cross platform builds, and sadly my dependencies are dependent on C headers, where I can't afford to make any change. and enabling CGO_ENABLED=1 also didn't work. and I switched to xgo, which happily does the build magics

Now I need to transport those local build to my remote release. I am writing bunch of glue scripts on Makefile using https://github.com/itchio/gothub
But its too lame and is not fun as goreleaser does :)

GitHub
gothub - :octocat: command-line app to create and edit releases on Github (and upload artifacts)

thapakazi commented Jul 4, 2018

👍 @endorama there there 😅
weird, I have the same requirements like you.

I need to do cross platform builds, and sadly my dependencies are dependent on C headers, where I can't afford to make any change. and enabling CGO_ENABLED=1 also didn't work. and I switched to xgo, which happily does the build magics

Now I need to transport those local build to my remote release. I am writing bunch of glue scripts on Makefile using https://github.com/itchio/gothub
But its too lame and is not fun as goreleaser does :)

GitHub
gothub - :octocat: command-line app to create and edit releases on Github (and upload artifacts)
@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 5, 2018

Member

hmm, ok, so the problem is different from what I thought it was... will reopen it!

Member

caarlos0 commented Jul 5, 2018

hmm, ok, so the problem is different from what I thought it was... will reopen it!

@caarlos0 caarlos0 reopened this Jul 5, 2018

@caarlos0 caarlos0 added enhancement and removed invalid labels Jul 5, 2018

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 8, 2018

Contributor

I have done some similar work in Hugo, and I would strongly recommend keeping this out of GoReleaser.

Contributor

bep commented Jul 8, 2018

I have done some similar work in Hugo, and I would strongly recommend keeping this out of GoReleaser.

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 9, 2018

Member

I have done some similar work in Hugo, and I would strongly recommend keeping this out of GoReleaser.

Hi @bep, what are the reasons for that?

Member

caarlos0 commented Jul 9, 2018

I have done some similar work in Hugo, and I would strongly recommend keeping this out of GoReleaser.

Hi @bep, what are the reasons for that?

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 9, 2018

Contributor

Hi @bep, what are the reasons for that?

I don't think it's doable. The xgo project is massive in size and complexity to support CGO crossbuilding to lots of platforms. You will need to run in Docker, for one.

Contributor

bep commented Jul 9, 2018

Hi @bep, what are the reasons for that?

I don't think it's doable. The xgo project is massive in size and complexity to support CGO crossbuilding to lots of platforms. You will need to run in Docker, for one.

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 9, 2018

Member

hmm, that makes sense, I didn't really dug into that yet... but if it is that complicated, I think I'll just not do it...

maybe... maybe, a xgo builder (so people could add language: xgo and goreleaser would just delegate), but not sure about that either...

Member

caarlos0 commented Jul 9, 2018

hmm, that makes sense, I didn't really dug into that yet... but if it is that complicated, I think I'll just not do it...

maybe... maybe, a xgo builder (so people could add language: xgo and goreleaser would just delegate), but not sure about that either...

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 9, 2018

Member

Anyway, thanks for the input @bep 🎉

Member

caarlos0 commented Jul 9, 2018

Anyway, thanks for the input @bep 🎉

@endorama

This comment has been minimized.

Show comment
Hide comment
@endorama

endorama Jul 9, 2018

Hello @bep thanks for the feedback. Do you mind sharing how you implemented this in hugo? I'm not getting how you are using goreleaser there.

I'm currently compiling using this command:

xgo \
    --targets="darwin/amd64,linux/amd64" \
    --dest=dist \
    --ldflags "-X main.version=$version" \
    -v -x \
    github.com/endorama/two-factor-authenticator

I suspect that at some point goreleaser is doing something similar with go build.
What I would propose here as a new feature goes along the line of the language: xgo @caarlos0 proposed.
Could also be further simplified having a way to pass the compile command line command to goreleaser with interpolation for provided variables.

I think that the added value for goreleaser is managing what is around building a package: package sign, checksum, release creation more that the build itself.

Would be awesome if there could be support for this while maintaining a KISS approach. I'm willing to help on this if seems feasible :)

endorama commented Jul 9, 2018

Hello @bep thanks for the feedback. Do you mind sharing how you implemented this in hugo? I'm not getting how you are using goreleaser there.

I'm currently compiling using this command:

xgo \
    --targets="darwin/amd64,linux/amd64" \
    --dest=dist \
    --ldflags "-X main.version=$version" \
    -v -x \
    github.com/endorama/two-factor-authenticator

I suspect that at some point goreleaser is doing something similar with go build.
What I would propose here as a new feature goes along the line of the language: xgo @caarlos0 proposed.
Could also be further simplified having a way to pass the compile command line command to goreleaser with interpolation for provided variables.

I think that the added value for goreleaser is managing what is around building a package: package sign, checksum, release creation more that the build itself.

Would be awesome if there could be support for this while maintaining a KISS approach. I'm willing to help on this if seems feasible :)

@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 9, 2018

Contributor

@endorama have a look at https://github.com/gohugoio/hugo/blob/master/goreleaser-extended.yml

I use my own Docker container to do this CGO cross builds. Think of it as a mini-xgo. So, Goreleaser works great in this context -- the real work is to get the actual builds going, which I think would be a bad fit for this project (too much work). And you pretty fast get into some hard license issues that you also probably want to avoid as a maintainer (the OSX SDK, mainly, which is needed for Darwin builds).

GitHub
hugo - The world’s fastest framework for building websites.
Contributor

bep commented Jul 9, 2018

@endorama have a look at https://github.com/gohugoio/hugo/blob/master/goreleaser-extended.yml

I use my own Docker container to do this CGO cross builds. Think of it as a mini-xgo. So, Goreleaser works great in this context -- the real work is to get the actual builds going, which I think would be a bad fit for this project (too much work). And you pretty fast get into some hard license issues that you also probably want to avoid as a maintainer (the OSX SDK, mainly, which is needed for Darwin builds).

GitHub
hugo - The world’s fastest framework for building websites.

@caarlos0 caarlos0 added wontfix and removed enhancement labels Jul 9, 2018

@stale stale bot removed the wontfix label Jul 9, 2018

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Jul 19, 2018

Member

I'll mark this as wont fix for now.

Member

caarlos0 commented Jul 19, 2018

I'll mark this as wont fix for now.

@caarlos0 caarlos0 closed this Jul 19, 2018

@caarlos0 caarlos0 added the wontfix label Jul 19, 2018

@endorama

This comment has been minimized.

Show comment
Hide comment
@endorama

endorama Jul 20, 2018

@caarlos0 thank you anyway for your time looking into this.

@bep thank your for your feedbacks, I'll have a look into it.

endorama commented Jul 20, 2018

@caarlos0 thank you anyway for your time looking into this.

@bep thank your for your feedbacks, I'll have a look into it.

@foolin

This comment has been minimized.

Show comment
Hide comment
@foolin

foolin Jul 26, 2018

@bep
My project is use the sqlite: https://github.com/mattn/go-sqlite3 and need cross compile with cgo,I see that config:
https://github.com/gohugoio/hugo/blob/master/goreleaser-extended.yml

- CC=x86_64-w64-mingw32-gcc
- CXX=x86_64-w64-mingw32-g++

- CC=o64-clang
- CXX=o64-clang++

How to install the compilers? Can you share it?

Thanks!

GitHub
go-sqlite3 - sqlite3 driver for go using database/sql
GitHub
hugo - The world’s fastest framework for building websites.

foolin commented Jul 26, 2018

@bep
My project is use the sqlite: https://github.com/mattn/go-sqlite3 and need cross compile with cgo,I see that config:
https://github.com/gohugoio/hugo/blob/master/goreleaser-extended.yml

- CC=x86_64-w64-mingw32-gcc
- CXX=x86_64-w64-mingw32-g++

- CC=o64-clang
- CXX=o64-clang++

How to install the compilers? Can you share it?

Thanks!

GitHub
go-sqlite3 - sqlite3 driver for go using database/sql
GitHub
hugo - The world’s fastest framework for building websites.
@bep

This comment has been minimized.

Show comment
Hide comment
@bep

bep Jul 26, 2018

Contributor

@foolin https://github.com/bep/dockerfiles/tree/master/ci-goreleaser

GitHub
Contribute to dockerfiles development by creating an account on GitHub.
Contributor

bep commented Jul 26, 2018

@foolin https://github.com/bep/dockerfiles/tree/master/ci-goreleaser

GitHub
Contribute to dockerfiles development by creating an account on GitHub.
@foolin

This comment has been minimized.

Show comment
Hide comment
@foolin

foolin commented Jul 26, 2018

@bep Thanks!

caarlos0 added a commit that referenced this issue Sep 10, 2018

docs: cgo
refs #708
refs #792
refs #795
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment