New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: provide a way to distribute library without source #12186

Closed
aaronbee opened this Issue Aug 18, 2015 · 14 comments

Comments

Projects
None yet
9 participants
@aaronbee

aaronbee commented Aug 18, 2015

What version of Go are you using (go version)?
I have tried go1.5rc1 and building from source:
go version devel +6f0c7df Tue Aug 18 17:00:59 2015 +0000 linux/amd64

What operating system and processor architecture are you using?
Linux 3.4, amd64

What did you do?
Build a distribution of our library as a tarball of the desired contents of GOPATH/src and GOPATH/pkg. Our distribution script implements the "mtime hack" discussed here, packaging dummy source files that only contain the package declaration line and .a files that have an mtime later than the corresponding source files.

Then to test our distribution we try to build an executable that imports the libraries in our distribution.

What did you expect to see?
A built executable.

What did you see instead?
The go tool complains about missing structs and funcs in our dummy go files.

This issue is also referenced in #2775

I also tried to use the shared library feature of go1.5 to distribute our library, but that appears to also require the source code.

@ianlancetaylor ianlancetaylor changed the title from mtime hack to distribute library without source stopped working in go1.5 to cmd/go: mtime hack to distribute library without source stopped working in go1.5 Aug 18, 2015

@ianlancetaylor ianlancetaylor added this to the Go1.5Maybe milestone Aug 18, 2015

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 19, 2015

Contributor

What's tripping you up is the check that the set of source files in the directory matches the ones that were used to compute the .a file. The "mtime hack" is fundamentally at odds with #3895. It's impossible to tell the difference between one person's "mtime hack" and another person's removing source files from the directory and expecting a rebuild.

You may be able to keep on with this subterfuge provided you create a set of dummy source files with exactly the same names as the original source files you compiled to produce the .a file, no more, no less, no changed names, and you have to put the real import blocks into the source files, so that the computed set of dependency packages is also the same.

Or you could distribute a copy of the go command that disables the p.buildID != targetBuildID check in pkg.go, but I think that will cause more problems than you might like.

Contributor

rsc commented Aug 19, 2015

What's tripping you up is the check that the set of source files in the directory matches the ones that were used to compute the .a file. The "mtime hack" is fundamentally at odds with #3895. It's impossible to tell the difference between one person's "mtime hack" and another person's removing source files from the directory and expecting a rebuild.

You may be able to keep on with this subterfuge provided you create a set of dummy source files with exactly the same names as the original source files you compiled to produce the .a file, no more, no less, no changed names, and you have to put the real import blocks into the source files, so that the computed set of dependency packages is also the same.

Or you could distribute a copy of the go command that disables the p.buildID != targetBuildID check in pkg.go, but I think that will cause more problems than you might like.

@rsc rsc changed the title from cmd/go: mtime hack to distribute library without source stopped working in go1.5 to cmd/go: provide a way to distribute library without source Aug 19, 2015

@rsc rsc modified the milestones: Go1.6Early, Go1.5Maybe Aug 19, 2015

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 19, 2015

Contributor

Changed title. Instead of an mtime hack we should, if we're going to support this, establish a clear signal for the go command.

Contributor

rsc commented Aug 19, 2015

Changed title. Instead of an mtime hack we should, if we're going to support this, establish a clear signal for the go command.

@tsuna

This comment has been minimized.

Show comment
Hide comment
@tsuna

tsuna Aug 19, 2015

Contributor

I can confirm that on a small example, removing the contents of the .go files after the import statements while preserving the mtime on the files allows this trick to continue to work.

I haven't yet managed to get it to work on our real code base, but I probably just need to hack our script some more to rub the .go files in the right way.

Contributor

tsuna commented Aug 19, 2015

I can confirm that on a small example, removing the contents of the .go files after the import statements while preserving the mtime on the files allows this trick to continue to work.

I haven't yet managed to get it to work on our real code base, but I probably just need to hack our script some more to rub the .go files in the right way.

@stemar94

This comment has been minimized.

Show comment
Hide comment
@stemar94

stemar94 Sep 30, 2015

What do you think of doing something like

package iRelyOnBinaries

import "foo/bar.a"

func main(){
    bar.FooBar() //same behavior as if sources would be available
}

Since . is not allowed in a package name identifier, the tool chain can infer that it has to use the precompiled binary ${GOPATH}/${GOOS}_${GOARCH}/pkg/foo/bar.a instead of any source files.

Comments? Update: see below

stemar94 commented Sep 30, 2015

What do you think of doing something like

package iRelyOnBinaries

import "foo/bar.a"

func main(){
    bar.FooBar() //same behavior as if sources would be available
}

Since . is not allowed in a package name identifier, the tool chain can infer that it has to use the precompiled binary ${GOPATH}/${GOOS}_${GOARCH}/pkg/foo/bar.a instead of any source files.

Comments? Update: see below

@stemar94

This comment has been minimized.

Show comment
Hide comment
@stemar94

stemar94 Sep 30, 2015

I guess it all falls down to this:

Go Execution Modes

So never mind my little proposal :D

stemar94 commented Sep 30, 2015

I guess it all falls down to this:

Go Execution Modes

So never mind my little proposal :D

@tsuna

This comment has been minimized.

Show comment
Hide comment
@tsuna

tsuna Sep 30, 2015

Contributor

In your proposal, how would you deal with the fact that during development you have access to the sources of "foo/bar" and you need the compiler to build from source when needed? Or are you suggesting to rewrite the imports to add the .a at the end of the import path at the point you distribute the code without the source of "foo/bar"? Yuck.

Contributor

tsuna commented Sep 30, 2015

In your proposal, how would you deal with the fact that during development you have access to the sources of "foo/bar" and you need the compiler to build from source when needed? Or are you suggesting to rewrite the imports to add the .a at the end of the import path at the point you distribute the code without the source of "foo/bar"? Yuck.

@stemar94

This comment has been minimized.

Show comment
Hide comment
@stemar94

stemar94 Sep 30, 2015

Yeah, the idea was instead of allowing to hack the toolchain with creating fake go files plus valid timestamps, this would be more user friendly. So if you got a library foo/bar.a binary just put it in your pkg folder and import it with the extra ".a" in the import path.

stemar94 commented Sep 30, 2015

Yeah, the idea was instead of allowing to hack the toolchain with creating fake go files plus valid timestamps, this would be more user friendly. So if you got a library foo/bar.a binary just put it in your pkg folder and import it with the extra ".a" in the import path.

@tsuna

This comment has been minimized.

Show comment
Hide comment
@tsuna

tsuna Oct 3, 2015

Contributor

I'm not happy with the hack that requires creating fake Go files, it's just a stopgap measure. But if we're gonna solve this problem, I'd rather not have a solution that involves rewriting imports.

Contributor

tsuna commented Oct 3, 2015

I'm not happy with the hack that requires creating fake Go files, it's just a stopgap measure. But if we're gonna solve this problem, I'd rather not have a solution that involves rewriting imports.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 23, 2015

Contributor

We didn't get to this for Go 1.6.

Contributor

rsc commented Nov 23, 2015

We didn't get to this for Go 1.6.

@akostadinov

This comment has been minimized.

Show comment
Hide comment
@akostadinov

akostadinov Jan 18, 2016

I think it would be better if import is the same between binary and source version. Just put in source the .a file (in this case your source is the binary). Or if that would break too many other assumptions, perhaps have the convention that if source path contains a file .binary, then user wants to compile with a binary only version of the package.

akostadinov commented Jan 18, 2016

I think it would be better if import is the same between binary and source version. Just put in source the .a file (in this case your source is the binary). Or if that would break too many other assumptions, perhaps have the convention that if source path contains a file .binary, then user wants to compile with a binary only version of the package.

@pborman

This comment has been minimized.

Show comment
Hide comment
@pborman

pborman Mar 22, 2016

Contributor

A real simple solution would be to simply have a simple .go file for binary export. Some suggestions:

binary package foo

//binary
package foo

or pick any other simple mechanism to indicate there are no sources for this package so the compiler should never try to rebuild it.

The real world does not/cannot always provide sources. This is not a hard problem, but it is a very important issue to commercial success in a segment of the software business. Those companies will not release source code so they can support/use Go. They will simply not use Go.

Contributor

pborman commented Mar 22, 2016

A real simple solution would be to simply have a simple .go file for binary export. Some suggestions:

binary package foo

//binary
package foo

or pick any other simple mechanism to indicate there are no sources for this package so the compiler should never try to rebuild it.

The real world does not/cannot always provide sources. This is not a hard problem, but it is a very important issue to commercial success in a segment of the software business. Those companies will not release source code so they can support/use Go. They will simply not use Go.

@tsuna

This comment has been minimized.

Show comment
Hide comment
@tsuna

tsuna Mar 22, 2016

Contributor

When //binary is found on a package, what do you envision the expected behavior of the Go compiler to be? If the .a needs to be rebuilt and the sources are available, it builds them and puts a special annotation in the .a, such that if it thinks it needs to rebuild the .a but the existing .a has that annotation it doesn't actually try to rebuild it? Is that along the lines of what you had in mind?

Contributor

tsuna commented Mar 22, 2016

When //binary is found on a package, what do you envision the expected behavior of the Go compiler to be? If the .a needs to be rebuilt and the sources are available, it builds them and puts a special annotation in the .a, such that if it thinks it needs to rebuild the .a but the existing .a has that annotation it doesn't actually try to rebuild it? Is that along the lines of what you had in mind?

@pborman

This comment has been minimized.

Show comment
Hide comment
@pborman

pborman Mar 23, 2016

Contributor

If //binary (maybe //go:nobuild or something) is seen then the .a file is
never rebuilt. Period. You as the vendor would have source code that does
not have //binary in it. Presumably you would have a release mechanism
that built the release with the .go files with //go:nobuild.

You could also include documentation so godoc would continue to provide
something useful.

Perhaps your source is

// Package mine does my stuff
package main

// Data contains something.
type Data struct {
Stuff string
}

// Result contains a result.
type Result struct {
OtherStuff string
}

// MySecretSauce returns an incredible transformation of x, or an error
func MySecretSauce(x _Data) (_Result, error) {
code we dont want to release
...
}

// internalRoutine is something you don't need to know about
func internalRoutine(x *Data, r *Result) error {
code we dont want to release
...
}

You might release a source file of:

package main
//go:norbuild

Or perhaps something with documentation:

// Package mine does my stuff
package main

// go:nobuild

// Data contains something.
type Data struct {
Stuff string
}

// Result contains a result.
type Result struct {
OtherStuff string
}

// MySecretSauce returns an incredible transformation of x, or an error
func MySecretSauce(x _Data) (_Result, error) {
return nil, nil
}

In any event, the .go file is purely informational and the Go command will
never try to build a package with //go:nobuild in the source (even if the
.a file is missing). The .go file would still need to be a legitimate Go
file.

This is only one of many possible ways to indicate to the compiler that
this is a binary release and should not be built.

On Tue, Mar 22, 2016 at 8:38 AM, Benoit Sigoure notifications@github.com
wrote:

When //binary is found on a package, what do you envision the expected
behavior of the Go compiler to be? If the .a needs to be rebuilt and the
sources are available, it builds them and puts a special annotation in the
.a, such that if it thinks it needs to rebuild the .a but the existing .a
has that annotation it doesn't actually try to rebuild it? Is that along
the lines of what you had in mind?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#12186 (comment)

Contributor

pborman commented Mar 23, 2016

If //binary (maybe //go:nobuild or something) is seen then the .a file is
never rebuilt. Period. You as the vendor would have source code that does
not have //binary in it. Presumably you would have a release mechanism
that built the release with the .go files with //go:nobuild.

You could also include documentation so godoc would continue to provide
something useful.

Perhaps your source is

// Package mine does my stuff
package main

// Data contains something.
type Data struct {
Stuff string
}

// Result contains a result.
type Result struct {
OtherStuff string
}

// MySecretSauce returns an incredible transformation of x, or an error
func MySecretSauce(x _Data) (_Result, error) {
code we dont want to release
...
}

// internalRoutine is something you don't need to know about
func internalRoutine(x *Data, r *Result) error {
code we dont want to release
...
}

You might release a source file of:

package main
//go:norbuild

Or perhaps something with documentation:

// Package mine does my stuff
package main

// go:nobuild

// Data contains something.
type Data struct {
Stuff string
}

// Result contains a result.
type Result struct {
OtherStuff string
}

// MySecretSauce returns an incredible transformation of x, or an error
func MySecretSauce(x _Data) (_Result, error) {
return nil, nil
}

In any event, the .go file is purely informational and the Go command will
never try to build a package with //go:nobuild in the source (even if the
.a file is missing). The .go file would still need to be a legitimate Go
file.

This is only one of many possible ways to indicate to the compiler that
this is a binary release and should not be built.

On Tue, Mar 22, 2016 at 8:38 AM, Benoit Sigoure notifications@github.com
wrote:

When //binary is found on a package, what do you envision the expected
behavior of the Go compiler to be? If the .a needs to be rebuilt and the
sources are available, it builds them and puts a special annotation in the
.a, such that if it thinks it needs to rebuild the .a but the existing .a
has that annotation it doesn't actually try to rebuild it? Is that along
the lines of what you had in mind?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#12186 (comment)

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Mar 29, 2016

Contributor

Duplicate of #2775

Contributor

robpike commented Mar 29, 2016

Duplicate of #2775

@robpike robpike closed this Mar 29, 2016

@golang golang locked and limited conversation to collaborators Mar 29, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.