New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: add *_unix.go recognition (and presumably unix build tag) #20322

Open
robpike opened this Issue May 11, 2017 · 18 comments

Comments

Projects
None yet
10 participants
@robpike
Contributor

robpike commented May 11, 2017

It does understand *_windows.go, and I was surprised to (re)learn that it does not support *_unix.go. Would it do so, things would be clearer and all the *_unix.go files that already exist wouldn't need to maintain their

// +build darwin dragonfly freebsd linux netbsd openbsd solaris

tags as the list grows.

@gopherbot gopherbot added this to the Proposal milestone May 11, 2017

@gopherbot gopherbot added the Proposal label May 11, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz May 11, 2017

Member

More concretely, does this mean "unix" would mean anything in a pre-defined list of GOOS values? Would we proactively add unsupported GOOS values that might be supported later, like we did with GOARCH values, or would "unix" mean anything that's not windows or plan9 or nacl? Wait, is nacl unix?

Also, should we define "posix"? Currently we have some _posix.go files which include windows.

I don't object to this, but I'm just curious which definition we'd use.

Member

bradfitz commented May 11, 2017

More concretely, does this mean "unix" would mean anything in a pre-defined list of GOOS values? Would we proactively add unsupported GOOS values that might be supported later, like we did with GOARCH values, or would "unix" mean anything that's not windows or plan9 or nacl? Wait, is nacl unix?

Also, should we define "posix"? Currently we have some _posix.go files which include windows.

I don't object to this, but I'm just curious which definition we'd use.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike May 11, 2017

Contributor

All good questions for which I don't know the answer.

My main concern is maintaining the long list of os names in the build tags. Making things easier would be an advantage too.

Discussion needed.

Contributor

robpike commented May 11, 2017

All good questions for which I don't know the answer.

My main concern is maintaining the long list of os names in the build tags. Making things easier would be an advantage too.

Discussion needed.

@j7b

This comment has been minimized.

Show comment
Hide comment
@j7b

j7b May 12, 2017

I'd suggest the focus should be on user code and the standard library, and a new build tag is required (perhaps "unix"). The standard library has clear manifestations of a notion of "unix" in packages intended for general use (most obviously in net), and I'd suggest any GOOS that implements these should satisfy the build constraint word "unix." For example, if nacl's net package supports network types "unix", "unixgram" and "unixpacket" it would satisfy some of the predicates the "unix" build constraint should require.

The *_unix.go convention could implicitly define +build unix.

j7b commented May 12, 2017

I'd suggest the focus should be on user code and the standard library, and a new build tag is required (perhaps "unix"). The standard library has clear manifestations of a notion of "unix" in packages intended for general use (most obviously in net), and I'd suggest any GOOS that implements these should satisfy the build constraint word "unix." For example, if nacl's net package supports network types "unix", "unixgram" and "unixpacket" it would satisfy some of the predicates the "unix" build constraint should require.

The *_unix.go convention could implicitly define +build unix.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike May 14, 2017

Contributor

That's pretty much what I mean.

Contributor

robpike commented May 14, 2017

That's pretty much what I mean.

@dominikh

This comment has been minimized.

Show comment
Hide comment
@dominikh

dominikh May 14, 2017

Member

Related: #6325

Member

dominikh commented May 14, 2017

Related: #6325

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche May 14, 2017

Member

Looking at the list of OSes, the BSD's may be the most homogeneous and largest sublist. But do you count Solaris, Darwin? Even if not, just defining bsd as dragonfly freebsd netbsd openbsd seems handy regardless of any other tags that might be included. If Darwin and Solaris are included, the build tag in the OP shrinks down to // +build linux bsd

Looking at the feature sets, posix may be the simplest target, but which subset of posix? Is it just the shared subset of all the unix-like OSes currently supported? When a new posix-y OS is added does that mean it can't be added to posix if it's missing one thing the other posix OSes support or do the set of guaranteed features lose a member? Since posix is so vast and as far as I know in practice little agreed upon, that seems the most problematic.

Looking at the stdlib, there seems to be very little agreement in practice among which OSes to include, even in *_unix.go and *_posix.go files. Of course, that doesn't preclude having build tags like // +build unix,!freebsd when needed.

Looking to the future, if the list is not proactive, as Brad put it, that means a new OS could suddenly match with a new version of Go when it should not. That may be worrisome but then again currently we have the situation where a new GOOS doesn't match anything without manual intervention. Of course, even a future proofed list is only as good as targets currently known and can't contain as yet uninvented OSes, and then the same issues arise as with posix.

Member

jimmyfrasche commented May 14, 2017

Looking at the list of OSes, the BSD's may be the most homogeneous and largest sublist. But do you count Solaris, Darwin? Even if not, just defining bsd as dragonfly freebsd netbsd openbsd seems handy regardless of any other tags that might be included. If Darwin and Solaris are included, the build tag in the OP shrinks down to // +build linux bsd

Looking at the feature sets, posix may be the simplest target, but which subset of posix? Is it just the shared subset of all the unix-like OSes currently supported? When a new posix-y OS is added does that mean it can't be added to posix if it's missing one thing the other posix OSes support or do the set of guaranteed features lose a member? Since posix is so vast and as far as I know in practice little agreed upon, that seems the most problematic.

Looking at the stdlib, there seems to be very little agreement in practice among which OSes to include, even in *_unix.go and *_posix.go files. Of course, that doesn't preclude having build tags like // +build unix,!freebsd when needed.

Looking to the future, if the list is not proactive, as Brad put it, that means a new OS could suddenly match with a new version of Go when it should not. That may be worrisome but then again currently we have the situation where a new GOOS doesn't match anything without manual intervention. Of course, even a future proofed list is only as good as targets currently known and can't contain as yet uninvented OSes, and then the same issues arise as with posix.

@j7b

This comment has been minimized.

Show comment
Hide comment
@j7b

j7b May 15, 2017

From a user code perspective it shouldn't matter if the "unix" is bsd, lunix, solaris or other. I'd consider mandating cgo for test purposes for "unix" candidates and having a cgo-centric test suite to make sure all the unix-type-things for a given Go toolchain aren't at odds with the C toolchain as known to the Go toolchain for a given GOOS.

j7b commented May 15, 2017

From a user code perspective it shouldn't matter if the "unix" is bsd, lunix, solaris or other. I'd consider mandating cgo for test purposes for "unix" candidates and having a cgo-centric test suite to make sure all the unix-type-things for a given Go toolchain aren't at odds with the C toolchain as known to the Go toolchain for a given GOOS.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc May 15, 2017

Contributor

This is basically a straight duplicate of #6325, as @dominikh points out. The problem here, as there, is that "unix" changes based on various details - sometimes it includes Linux, sometimes not, for example. It's not clear we want to be in the business of defining it exactly one way. Also, this should not affect users terribly much, given that we try to provide OS-independent APIs to the largest extent possible.

Contributor

rsc commented May 15, 2017

This is basically a straight duplicate of #6325, as @dominikh points out. The problem here, as there, is that "unix" changes based on various details - sometimes it includes Linux, sometimes not, for example. It's not clear we want to be in the business of defining it exactly one way. Also, this should not affect users terribly much, given that we try to provide OS-independent APIs to the largest extent possible.

@rsc rsc changed the title from proposal: have the go command interpret *_unix.go as Unix-specific code. to proposal: cmd/go: add *_unix.go recognition (and presumably unix build tag) May 15, 2017

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike May 16, 2017

Contributor

You're skirting the point here, which is just to avoid maintaining the huge list of Unix variants Go supports. It's to discriminate the basic difference between Windows and Unix. Windows has many versions, but there is only one predefined "windows" tag, in the sense the foo_windows.go works. I'm asking for the same courtesy for Unix.

Contributor

robpike commented May 16, 2017

You're skirting the point here, which is just to avoid maintaining the huge list of Unix variants Go supports. It's to discriminate the basic difference between Windows and Unix. Windows has many versions, but there is only one predefined "windows" tag, in the sense the foo_windows.go works. I'm asking for the same courtesy for Unix.

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche

jimmyfrasche May 16, 2017

Member

Ignoring ios and android for the moment, as they're essentially defined to be "variants" of darwin and linux, and only looking at the GOOS build tags, the complete list, as of 1.8, is:

windows plan9 nacl linux solaris darwin dragonfly openbsd netbsd freebsd

The goal, as I understand it, is to be able to use *_GOOS.go as much as possible, by defining one or more "super-GOOS" tags that match a large subset of the above tags.

I'm going to ignore the implications of defining unix or posix and just look at what's in the stdlib so there's some data. (Apologies if I made any mistakes).

Looking at the *_posix.go files in the $GOROOT/src† the definition of posix uses seems to be almost equivalent to !plan9, with occasion to additionally specify !nacl (3/15 of the files) and once adding !solaris.

(Windows is not what most people think of when they thing posix. An alternative would be to allow something like x_not-T.go to be equivalent to a // +build !T directive and rename most of those as _not-plan9.go.)

† using

find -type f -name "*_posix.go" -exec grep "^// [+]build " {} \; | cut -d" " -f3- | sort

For the *_unix.go files,I took the list of tags, removed two that, despite the name, only had to do with cgo, took out all arches, android build tags, etc., manually expanded the negations, reordered them so all the tags were in the same order and then counted the occurrences of each unique subset of GOOSes:

18: darwin dragonfly freebsd linux nacl netbsd openbsd solaris
12: darwin dragonfly freebsd linux netbsd openbsd solaris
3: darwin dragonfly freebsd linux nacl netbsd openbsd plan9 solaris
2: darwin dragonfly freebsd linux netbsd openbsd
2: darwin dragonfly freebsd linux nacl netbsd openbsd solaris windows
1: dragonfly freebsd linux nacl netbsd openbsd solaris
1: dragonfly freebsd linux netbsd openbsd
1: dragonfly linux netbsd openbsd solaris
1: freebsd linux netbsd

Looking at that it seems !windows,!plan9 would get the most existing *_unix.go files, fit most people's expected definition, but often require an additional // +build !nacl directive.

In practice how build tags are written seems to be somewhat ad hoc. Maybe gofmt could sort them in a standard way or golint warn if they're not alphabetical? Perhaps a new tool that deals solely with build constraints?

Member

jimmyfrasche commented May 16, 2017

Ignoring ios and android for the moment, as they're essentially defined to be "variants" of darwin and linux, and only looking at the GOOS build tags, the complete list, as of 1.8, is:

windows plan9 nacl linux solaris darwin dragonfly openbsd netbsd freebsd

The goal, as I understand it, is to be able to use *_GOOS.go as much as possible, by defining one or more "super-GOOS" tags that match a large subset of the above tags.

I'm going to ignore the implications of defining unix or posix and just look at what's in the stdlib so there's some data. (Apologies if I made any mistakes).

Looking at the *_posix.go files in the $GOROOT/src† the definition of posix uses seems to be almost equivalent to !plan9, with occasion to additionally specify !nacl (3/15 of the files) and once adding !solaris.

(Windows is not what most people think of when they thing posix. An alternative would be to allow something like x_not-T.go to be equivalent to a // +build !T directive and rename most of those as _not-plan9.go.)

† using

find -type f -name "*_posix.go" -exec grep "^// [+]build " {} \; | cut -d" " -f3- | sort

For the *_unix.go files,I took the list of tags, removed two that, despite the name, only had to do with cgo, took out all arches, android build tags, etc., manually expanded the negations, reordered them so all the tags were in the same order and then counted the occurrences of each unique subset of GOOSes:

18: darwin dragonfly freebsd linux nacl netbsd openbsd solaris
12: darwin dragonfly freebsd linux netbsd openbsd solaris
3: darwin dragonfly freebsd linux nacl netbsd openbsd plan9 solaris
2: darwin dragonfly freebsd linux netbsd openbsd
2: darwin dragonfly freebsd linux nacl netbsd openbsd solaris windows
1: dragonfly freebsd linux nacl netbsd openbsd solaris
1: dragonfly freebsd linux netbsd openbsd
1: dragonfly linux netbsd openbsd solaris
1: freebsd linux netbsd

Looking at that it seems !windows,!plan9 would get the most existing *_unix.go files, fit most people's expected definition, but often require an additional // +build !nacl directive.

In practice how build tags are written seems to be somewhat ad hoc. Maybe gofmt could sort them in a standard way or golint warn if they're not alphabetical? Perhaps a new tool that deals solely with build constraints?

@jimmyfrasche

This comment has been minimized.

Show comment
Hide comment
@jimmyfrasche
Member

jimmyfrasche commented May 16, 2017

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc May 23, 2017

Contributor

It sounds like the real interesting use would be outside the standard library. Can someone survey what usages look like outside, like in the go corpus files?

Contributor

rsc commented May 23, 2017

It sounds like the real interesting use would be outside the standard library. Can someone survey what usages look like outside, like in the go corpus files?

@j7b

This comment has been minimized.

Show comment
Hide comment
@j7b

j7b May 23, 2017

Maybe the best expression of "unix" is !windows,!android,!plan9,!nacl as it's much easier to express why a GOOS is !unix and the build tag waters aren't muddied when user code has to pander to particular variants of things that aren't !unix.

j7b commented May 23, 2017

Maybe the best expression of "unix" is !windows,!android,!plan9,!nacl as it's much easier to express why a GOOS is !unix and the build tag waters aren't muddied when user code has to pander to particular variants of things that aren't !unix.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jun 5, 2017

Contributor

On hold for someone to present evidence about the need for this outside the standard library.

Contributor

rsc commented Jun 5, 2017

On hold for someone to present evidence about the need for this outside the standard library.

@rsc rsc added the Proposal-Hold label Jun 5, 2017

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Jun 5, 2017

Contributor

Any code that needs to support variant operating systems would need this. For example, Upspin has

errors_unix.go:5:// +build darwin linux freebsd netbsd

Is that the right list? No. What is? I don't know, and the code will go out of date as new variants arise. The code in that file is Unix-generic, but what is Unix from Go's point of view?

It seems to me that if we can define golang.org/x/sys/unix then we can define build tag unix and _unix.go.

Contributor

robpike commented Jun 5, 2017

Any code that needs to support variant operating systems would need this. For example, Upspin has

errors_unix.go:5:// +build darwin linux freebsd netbsd

Is that the right list? No. What is? I don't know, and the code will go out of date as new variants arise. The code in that file is Unix-generic, but what is Unix from Go's point of view?

It seems to me that if we can define golang.org/x/sys/unix then we can define build tag unix and _unix.go.

@vmarkovtsev

This comment has been minimized.

Show comment
Hide comment
@vmarkovtsev

vmarkovtsev Jan 22, 2018

I have gathered some stats on the GitHub dataset in BigQuery.

First I extracted all the "// +build" lines:

SELECT C.id as id, MIN(C.content) as content, MIN(F.path) as path, MIN(C.size) as size, MIN(F.repo_name) as repo
FROM `bigquery-public-data.github_repos.contents` as C
JOIN `bigquery-public-data.github_repos.files` as F
ON C.id = F.id AND F.path LIKE '%.go'
GROUP BY id

Query complete (251.7s elapsed, 2.26 TB processed)


SELECT line, COUNT(*) as n
FROM (
  SELECT SPLIT(content, '\n') as line
  FROM [eminent-airport-191509:github.go_content]
  HAVING LEFT(line, 9) = '// +build'
)
GROUP BY line
ORDER BY n DESC

Query complete (10.0s elapsed, 11.2 GB processed)

The raw result is attached as lines.csv.gz

Then I calculated the cooccurrences using a Neugram script
bake.txt and the result is attached as
result.csv.

Finally, I normalized the co-occurrence matrix by column (sum=1) and plotted it using
a Python script:

unix

How to interpret: OS Y -> OS X cell is whiter if X occurs with Y more often than with the rest of the column. E.g. Plan9 is mentioned together with Windows more often than with anything else.
We can see that BSDs, Linux and Solaris tend to go together.

vmarkovtsev commented Jan 22, 2018

I have gathered some stats on the GitHub dataset in BigQuery.

First I extracted all the "// +build" lines:

SELECT C.id as id, MIN(C.content) as content, MIN(F.path) as path, MIN(C.size) as size, MIN(F.repo_name) as repo
FROM `bigquery-public-data.github_repos.contents` as C
JOIN `bigquery-public-data.github_repos.files` as F
ON C.id = F.id AND F.path LIKE '%.go'
GROUP BY id

Query complete (251.7s elapsed, 2.26 TB processed)


SELECT line, COUNT(*) as n
FROM (
  SELECT SPLIT(content, '\n') as line
  FROM [eminent-airport-191509:github.go_content]
  HAVING LEFT(line, 9) = '// +build'
)
GROUP BY line
ORDER BY n DESC

Query complete (10.0s elapsed, 11.2 GB processed)

The raw result is attached as lines.csv.gz

Then I calculated the cooccurrences using a Neugram script
bake.txt and the result is attached as
result.csv.

Finally, I normalized the co-occurrence matrix by column (sum=1) and plotted it using
a Python script:

unix

How to interpret: OS Y -> OS X cell is whiter if X occurs with Y more often than with the rest of the column. E.g. Plan9 is mentioned together with Windows more often than with anything else.
We can see that BSDs, Linux and Solaris tend to go together.

@trex58

This comment has been minimized.

Show comment
Hide comment
@trex58

trex58 May 30, 2018

There is a new GOOS value to take into account now: aix .
We have ported GCC Go on AIX 7 and we are working now on porting golang.
Nearly all changes dealing with AIX and GCC Go have been submitted and merged.
We are now working on golang.org/x and it appears that it contains a lot of OS cases. Moreover, colleagues who are porting Go applications on AIX are facing many cases where they have to manage the OS (adding aix) though the code is far from the Operating System.

About golang.org/x, here are some recent (2018/05/30) data about how it depends on "linux", goos, and goarch . That provides an idea about the amount of work required for adding a new operating system or a new architecture.

Number of times the expression: '// +build ...linux...' appears in the sources of each package:
arch: 0
benchmarks: 3
blog: 0
build: 3
crypto: 8
exp: 14
image: 0
mobile: 34
net: 54
oauth2: 0
perf: 0
sync: 0
sys: 94
talks: 0
text: 0
tools: 8
vgo: 5

Number of times the key-word: 'linux' appears in the sources of each package:
arch: 1
benchmarks: 3
blog: 5
build: 488
crypto: 9
exp: 18
image: 0
mobile: 57
net: 153
oauth2: 0
perf: 12
sync: 0
sys: 283
talks: 15
text: 0
tools: 46
vgo: 70

Number of times the key-word: 'goos' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 1
build: 35
crypto: 0
exp: 0
image: 0
mobile: 19
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 8
talks: 0
text: 0
tools: 6
vgo: 27

Number of times the key-word: 'goarch' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 0
build: 37
crypto: 0
exp: 0
image: 0
mobile: 8
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 5
talks: 0
text: 0
tools: 6
vgo: 28

trex58 commented May 30, 2018

There is a new GOOS value to take into account now: aix .
We have ported GCC Go on AIX 7 and we are working now on porting golang.
Nearly all changes dealing with AIX and GCC Go have been submitted and merged.
We are now working on golang.org/x and it appears that it contains a lot of OS cases. Moreover, colleagues who are porting Go applications on AIX are facing many cases where they have to manage the OS (adding aix) though the code is far from the Operating System.

About golang.org/x, here are some recent (2018/05/30) data about how it depends on "linux", goos, and goarch . That provides an idea about the amount of work required for adding a new operating system or a new architecture.

Number of times the expression: '// +build ...linux...' appears in the sources of each package:
arch: 0
benchmarks: 3
blog: 0
build: 3
crypto: 8
exp: 14
image: 0
mobile: 34
net: 54
oauth2: 0
perf: 0
sync: 0
sys: 94
talks: 0
text: 0
tools: 8
vgo: 5

Number of times the key-word: 'linux' appears in the sources of each package:
arch: 1
benchmarks: 3
blog: 5
build: 488
crypto: 9
exp: 18
image: 0
mobile: 57
net: 153
oauth2: 0
perf: 12
sync: 0
sys: 283
talks: 15
text: 0
tools: 46
vgo: 70

Number of times the key-word: 'goos' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 1
build: 35
crypto: 0
exp: 0
image: 0
mobile: 19
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 8
talks: 0
text: 0
tools: 6
vgo: 27

Number of times the key-word: 'goarch' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 0
build: 37
crypto: 0
exp: 0
image: 0
mobile: 8
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 5
talks: 0
text: 0
tools: 6
vgo: 28

@mortdeus

This comment has been minimized.

Show comment
Hide comment
@mortdeus

mortdeus May 30, 2018

dang I thought this was going to be a proposal to port Go to run on the old PDP unix made at bell labs. :(

Also, I explored thinking about this awhile back but how hard would it be to port Go to run on top of a virtual OS layer like inferno?

mortdeus commented May 30, 2018

dang I thought this was going to be a proposal to port Go to run on the old PDP unix made at bell labs. :(

Also, I explored thinking about this awhile back but how hard would it be to port Go to run on top of a virtual OS layer like inferno?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment