Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go plugin: cross-compilation support #1253

Closed
wants to merge 44 commits into from

Conversation

Projects
None yet
8 participants
@kalikiana
Copy link
Contributor

commented Apr 13, 2017

To be used with the --target-arch= argument.

Fixes bug 1664857.

@sergiusens sergiusens requested a review from chipaca Apr 13, 2017

@sergiusens

This comment has been minimized.

Copy link
Collaborator

commented Apr 13, 2017

I've added someone as a reviewer who will give you some magic lines 😉

@chipaca

This comment has been minimized.

Copy link
Member

commented Apr 14, 2017

At a glance, this is missing CGO_ENABLED=1 (otherwise Go runs as if CGO_ENABLED=0 were set, when cross-compiling). Without cgo enabled simple things like current user lookup, and full nssswitch support, won't work. You also probably need a few more flags to get the right ld and etc; I recommend having a simple go program that uses a shared C library for testing.

If you want to suffer, make that a simple go program that uses a shared C++ library. Mix both of these if you want to lose your mind.

The best quick read for this that I know of is still xnox's blog post from a few years ago.

Also note the intricacies of this differ between go versions, so test it on all of them.

@chipaca

This comment has been minimized.

Copy link
Member

commented Apr 14, 2017

As a start, try this:

package main

/*
#cgo pkg-config: glib-2.0
#cgo LDFLAGS: -lm
#include <math.h>
#include <glib.h>

*/
import "C"

import (
	"fmt"
	"os/user"
)

func main() {
	u, err := user.Current()
	if err != nil {
		panic(err)
	}
	fmt.Println("user according to go:", u.Username)
	fmt.Println("user according to glib:", C.GoString((*C.char)(C.g_get_user_name())))
	fmt.Println("more or less 1:", C.sin(3.14/2))
}

@kalikiana kalikiana force-pushed the kalikiana:go-cross branch from 4be66f0 to bfebc84 Apr 19, 2017

@sergiusens

This comment has been minimized.

Copy link
Collaborator

commented Apr 19, 2017

you might need to commit --reauthor your commit with christian@twotoasts.de or sign the CLA ;-)

@sergiusens

This comment has been minimized.

Copy link
Collaborator

commented Apr 19, 2017

@mwhudson mind taking a look at this?

@mwhudson

This comment has been minimized.

Copy link
Contributor

commented Apr 19, 2017

It all looks OK to me.

@elopio
Copy link
Contributor

left a comment

@kalikiana, please add a manual test for this, in manual_tests.md.

}
env['GOARCH'] = go_archs.get(self.project.deb_arch,
self.project.deb_arch)
env['GOARM'] = '7'

This comment has been minimized.

Copy link
@sergiusens

sergiusens Apr 20, 2017

Collaborator

do you want to set GOARM even if GOARCH is not arm?

This comment has been minimized.

Copy link
@mwhudson

mwhudson Apr 20, 2017

Contributor

It's harmless, if a little strange.

This comment has been minimized.

Copy link
@sergiusens

sergiusens May 18, 2017

Collaborator

I would add an if self.project.debarch == 'armhf' here, GOARM is probably 8 for arm64 anyways

If you want to add other functionalities to this snap, please don't.
Make a new one.
icon: icon.png

This comment has been minimized.

Copy link
@sergiusens

sergiusens Apr 24, 2017

Collaborator

let's remove this (and the actual icon file)

@@ -31,6 +33,7 @@
from testtools.matchers import MatchesRegex

from snapcraft.tests import fixture_setup
from snapcraft.internal.repo import _deb

This comment has been minimized.

Copy link
@sergiusens

sergiusens Apr 24, 2017

Collaborator

this gives me a bad vibe, not sure how to solve it yet. Maybe reimplement here, @elopio ?

@@ -129,6 +132,32 @@ def run_snapcraft_parser(self, arguments):
raise
return snapcraft_output

@contextmanager
def setup_multi_arch_sources(self, arch):

This comment has been minimized.

Copy link
@elopio

elopio Apr 24, 2017

Contributor

This is nice, but in tests we usually have fixtures instead of context managers. I personally would prefer to follow that practice, but I don't have a very strong opinion against this context manager.


arch = 'armhf'
with self.setup_multi_arch_sources(arch):
self.run_snapcraft('stage', 'go-cgo', '--target={}'.format(arch))

This comment has been minimized.

Copy link
@elopio

elopio Apr 24, 2017

Contributor

looks good! Thanks :)

('i386', dict(deb_arch='i386', go_arch='386')),
('x86_64', dict(deb_arch='amd64', go_arch='amd64')),
('ppc64le', dict(deb_arch='ppc64el', go_arch='ppc64le')),
]

This comment has been minimized.

Copy link
@elopio

elopio Apr 24, 2017

Contributor

Please move the scenarios before the test.

@elopio
Copy link
Contributor

left a comment

looks very good to me. I'm +1, with the request to move the scenarios before the test, and a comment about using a fixture instead of a context manager.

@sergiusens sergiusens changed the title Cross-compilation support for the Go plugin go plugin: cross-compilation support Apr 26, 2017

@kalikiana kalikiana force-pushed the kalikiana:go-cross branch from a28bdc0 to 374f6e0 May 19, 2017

@sergiusens

This comment has been minimized.

Copy link
Collaborator

commented May 19, 2017

Seeing some issues

First try, go is not installed

sergiusens@mirkwood:~/source/snapcraft/integration_tests/snaps/go-cgo$ ../../../bin/snapcraft snap --target-arch=armhf
Setting target machine to 'armhf'
"grade" property not specified: defaulting to "stable"
Preparing to pull go-cgo 
Pulling go-cgo 
Please consider setting `go-importpath` for the 'go-cgo' part
go get -t -d ./go-cgo/...
/tmp/tmpirt49ylk: 10: exec: go: not found
....
subprocess.CalledProcessError: Command '['/bin/sh', '/tmp/tmpirt49ylk', 'go', 'get', '-t', '-d', './go-cgo/...']' returned non-zero exit status 127

Second try, first without x-compile

sergiusens@mirkwood:~/source/snapcraft/integration_tests/snaps/go-cgo$ ../../../bin/snapcraft pull
"grade" property not specified: defaulting to "stable"
Installing build dependencies: golang-1.7-go golang-1.7-src golang-go golang-src libglib2.0-dev libglib2.0-dev-bin libpcre3-dev libpcre32-3 libpcrecpp0v5
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  libglib2.0-doc
Recommended packages:
  golang-1.7-race-detector-runtime golang-race-detector-runtime
The following NEW packages will be installed:
  golang-1.7-go golang-1.7-src golang-go golang-src libglib2.0-dev
  libglib2.0-dev-bin libpcre3-dev libpcre32-3 libpcrecpp0v5
0 upgraded, 9 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/28,0 MB of archives.
After this operation, 154 MB of additional disk space will be used.
...
Preparing to pull go-cgo 
Pulling go-cgo 
Please consider setting `go-importpath` for the 'go-cgo' part
go get -t -d ./go-cgo/...

Now I can x-compile, but wait...

sergiusens@mirkwood:~/source/snapcraft/integration_tests/snaps/go-cgo$ ../../../bin/snapcraft snap --target-arch=armhf
Setting target machine to 'armhf'
"grade" property not specified: defaulting to "stable"
Preparing to pull go-cgo 
Pulling go-cgo 
Please consider setting `go-importpath` for the 'go-cgo' part
go get -t -d ./go-cgo/...
Preparing to build go-cgo 
Building go-cgo 
Please consider setting `go-importpath` for the 'go-cgo' part
go build -o /home/sergiusens/source/snapcraft/integration_tests/snaps/go-cgo/parts/go-cgo/go/bin/go-cgo ./go-cgo/...
# runtime/cgo
exec: "arm-linux-gnueabihf-gcc": executable file not found in $PATH
...

@classmethod
def install_build_packages(cls, package_names, arch):
unique_packages = set(cls._get_build_deps(package_names, arch))

This comment has been minimized.

Copy link
@elopio

elopio May 19, 2017

Contributor

This is crashing for me:

/home/ubuntu/workspace/canonical/snapcraft/snapcraft/internal/repo/_deb.py(286)install_build_packages()
-> new_packages = []
(Pdb) unique_packages
{'for', 'Keep', 'privileges', 'so', 'package', 'get', 'build', 'in', 'relevance', "don't", "'/tmp/tmpvnu0mvxc.dsc'", 'tree...', 'file', 'the', 'is', 'dependencies', 'depend', 'real', 'on', 'also', 'deactivated,', 'locking', 'execution.', 'Reading', 'information...', 'current', 'needs', 'Building', 'mind', 'using', 'root', 'state', 'to', 'situation!', 'lists...', 'that', 'Note,', 'dependency'}

$ apt-get build-dep -q -s -aarm64 golang-go:native
NOTE: This is only a simulation!
apt-get needs root privileges for real execution.
Keep also in mind that locking is deactivated,
so don't depend on the relevance to the real current situation!
E: No architecture information available for arm64. See apt.conf(5) APT::Architectures for setup

@sergiusens sergiusens modified the milestones: 2.31, 2.30 May 19, 2017

kalikiana added some commits May 23, 2017

Call dpkg --add-architecture before apt-get build-dep
Otherwise there won't be any build dependency output.
env['CC'] = '{}-gcc'.format(self.project.arch_triplet)
env['CXX'] = '{}-g++'.format(self.project.arch_triplet)
env['CGO_ENABLED'] = '1'
env['PKG_CONFIG_PATH'] = '/usr/lib/{}/pkgconfig'.format(

This comment has been minimized.

Copy link
@kyrofa

kyrofa May 29, 2017

Member

This should not be needed. Check the environment bits in project loader for more details.

if not packages:
packages = ['./{}/...'.format(self._get_local_go_package())]
for package in packages:
binary = self._gopath_bin + '/' + self._binary_name(package)

This comment has been minimized.

Copy link
@kyrofa

kyrofa May 29, 2017

Member

In order to maintain consistency with the rest of the codebase, I suggest os.path.join() here.

@kyrofa
Copy link
Member

left a comment

Can we split this PR in two, please?

  1. Go-specific changes that will only work if changing dpkg arch by hand
  2. The repo changes that add dpkg bits automatically (protected by a prompt)

@sergiusens sergiusens modified the milestones: 2.31, 2.30.1 Jun 1, 2017

@kalikiana

This comment has been minimized.

Copy link
Contributor Author

commented Jun 1, 2017

@kalikiana kalikiana closed this Jun 1, 2017

@kyrofa kyrofa removed this from the 2.31 milestone Jun 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.