spec: reference-like properties of channels, slices, and maps not well specified

[robpike]

Although other documents such as Effective Go are clear on the subject, the spec is
unclear on the sharing that occurs when channels, maps and slices are assigned to other
variables such as through regular assignment and function calls. For instance, I cannot
find an explanation that changed made by a function with a map argument will be visible
in the caller. It's even more important to specify this for slices, since the header is
not shared but the array is. Channels would never work without this sharing property,
but again it's not well specified.

[griesemer]

Comment 1:

Owner changed to @griesemer.

[griesemer]

Comment 1:

Owner changed to @griesemer.

[griesemer]

Comment 2:

Issue #6529 has been merged into this issue.

[griesemer]

Comment 2:

Issue #6529 has been merged into this issue.

[rsc]

Comment 3:

Labels changed: added go1.3maybe.

[rsc]

Comment 3:

Labels changed: added go1.3maybe.

[rsc]

Comment 4:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 4:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 5:

Labels changed: added repo-main.

[rsc]

Comment 5:

Labels changed: added repo-main.

[nathany]

Comment 6:

Someone once explained them to me as types containing _unexported pointers_. That seems
to align with the sliceHeader example in http://blog.golang.org/slices.

[nathany]

Comment 6:

Someone once explained them to me as types containing _unexported pointers_. That seems
to align with the sliceHeader example in http://blog.golang.org/slices.

[adonovan]

Concluding a recent thread https://groups.google.com/d/topic/golang-nuts/86N_xtgcZqo/discussion, I think the the burden is on the spec to make clear

• that a 'map' value is a reference to an opaque data structure
• that make(map) creates an instance of this data structure and returns a reference to it
• that copying a 'map' value creates a new alias to the same instance of that data structure
• that the effect of m[k]=v updates made via one reference are visible to all aliases
• that a 'map' value may be safely represented by an unsafe.Pointer (as rsc told me)

and on the memory model to state

• that this data structure is analogous to a variable
• that m[k] and range operations on a map are analogous to reads of that variable
• and that m[k]=v and delete operations are analogous to writes.

[adonovan]

Concluding a recent thread https://groups.google.com/d/topic/golang-nuts/86N_xtgcZqo/discussion, I think the the burden is on the spec to make clear

• that a 'map' value is a reference to an opaque data structure
• that make(map) creates an instance of this data structure and returns a reference to it
• that copying a 'map' value creates a new alias to the same instance of that data structure
• that the effect of m[k]=v updates made via one reference are visible to all aliases
• that a 'map' value may be safely represented by an unsafe.Pointer (as rsc told me)

and on the memory model to state

• that this data structure is analogous to a variable
• that m[k] and range operations on a map are analogous to reads of that variable
• and that m[k]=v and delete operations are analogous to writes.

[ianlancetaylor]

I remain unconvinced that we should enforce in the language spec that a map value may be represented by an unsafe.Pointer. We don't specify the internal format of slices or interface values or channels, and I don't think we should specify maps.

[ianlancetaylor]

I remain unconvinced that we should enforce in the language spec that a map value may be represented by an unsafe.Pointer. We don't specify the internal format of slices or interface values or channels, and I don't think we should specify maps.

[adonovan]

I understand your concern. The (reflect.Value).Pointer method seems to suggest that a map value is just a pointer, but it's not clear what it actually commits us to.

It would be nice if the language provided a reliable and portable way for programs to detect cycles in arbitrary object graphs. This requires a robust notion of identity for maps, channels, functions, and slices. (reflect.Value).Pointer + unsafe.Pointer is sufficient for the first three; slices require runtime.SliceHeader + unsafePointer, which is not portable.

[adonovan]

I understand your concern. The (reflect.Value).Pointer method seems to suggest that a map value is just a pointer, but it's not clear what it actually commits us to.

It would be nice if the language provided a reliable and portable way for programs to detect cycles in arbitrary object graphs. This requires a robust notion of identity for maps, channels, functions, and slices. (reflect.Value).Pointer + unsafe.Pointer is sufficient for the first three; slices require runtime.SliceHeader + unsafePointer, which is not portable.

[alercah]

After running into this one again myself, here's a thought. I think the issue is actually most significant for maps and channels, because they refer to programmer-invisible common state. While slices and pointers refer to common state as well, this state is much more explicitly exposed to the programmer for these types, while map and channel state is completely opaque. Functions are in practice similar as well, but because there are no modifying operations on functions, this is irrelevant (an implementation which actually passed around executable code would perhaps have authors in need of a stern talking to, but would not otherwise be an issue).

So I think this could be resolved by:

1. Invent the concept of an underlying map/channel to which a map and channel value refer (except for nil). I chose "underlying" because it is already in use for slices.
2. Specify that map and channel operations other than assignment operate on the underlying object. make returns a value pointing to a new underlying object.
3. Clarify pointers by specifying that a non-nil pointer points to an addressable value, called the underlying variable. Explicitly specify that taking the address of a composite literal is a shorthand for declaring it as a variable and taking its address, to ensure that the value is always addressable.
4. Clearly specify the effect of assignment:
   a. For numeric, boolean, and string types, direct value assignment.
   b. For array and struct types, elementwise assignment per these rules.
   c. For interface types, assignment of the value per these rules (provide an example like https://play.golang.org/p/Eh3BwAWYI3 since this behaviour can be nonintuitive).
   d. For map, pointer, slice, and channel types, setting the underlying object and, in the case of a slice, also length and capacity.
   e. Function types should probably be dinstinguished on their own, and we vaguely say that after assignment, the variable assigned to refers to the same function as the value being assigned.

Thoughts?

[alercah]

After running into this one again myself, here's a thought. I think the issue is actually most significant for maps and channels, because they refer to programmer-invisible common state. While slices and pointers refer to common state as well, this state is much more explicitly exposed to the programmer for these types, while map and channel state is completely opaque. Functions are in practice similar as well, but because there are no modifying operations on functions, this is irrelevant (an implementation which actually passed around executable code would perhaps have authors in need of a stern talking to, but would not otherwise be an issue).

So I think this could be resolved by:

1. Invent the concept of an underlying map/channel to which a map and channel value refer (except for nil). I chose "underlying" because it is already in use for slices.
2. Specify that map and channel operations other than assignment operate on the underlying object. make returns a value pointing to a new underlying object.
3. Clarify pointers by specifying that a non-nil pointer points to an addressable value, called the underlying variable. Explicitly specify that taking the address of a composite literal is a shorthand for declaring it as a variable and taking its address, to ensure that the value is always addressable.
4. Clearly specify the effect of assignment:
   a. For numeric, boolean, and string types, direct value assignment.
   b. For array and struct types, elementwise assignment per these rules.
   c. For interface types, assignment of the value per these rules (provide an example like https://play.golang.org/p/Eh3BwAWYI3 since this behaviour can be nonintuitive).
   d. For map, pointer, slice, and channel types, setting the underlying object and, in the case of a slice, also length and capacity.
   e. Function types should probably be dinstinguished on their own, and we vaguely say that after assignment, the variable assigned to refers to the same function as the value being assigned.

Thoughts?

[griesemer]

@alercah Thanks for your suggestions! I can't respond in a meaningful way at the moment as this is a non-urgent issue and I haven't spent any time thinking about it more seriously. Just wanted to acknowledge that your feedback is appreciated.

[griesemer]

@alercah Thanks for your suggestions! I can't respond in a meaningful way at the moment as this is a non-urgent issue and I haven't spent any time thinking about it more seriously. Just wanted to acknowledge that your feedback is appreciated.

[alercah]

For sure! Some follow-up clarifications:

• I found the section about composite literal being a shorthand for declaring a variable and taking its address, it just wasn't where I expected it so I missed it when writing the above comment.
• For interface assignment, it should also obviously assign the type too.
• See also my comments in #20181

[alercah]

For sure! Some follow-up clarifications:

• I found the section about composite literal being a shorthand for declaring a variable and taking its address, it just wasn't where I expected it so I missed it when writing the above comment.
• For interface assignment, it should also obviously assign the type too.
• See also my comments in #20181