Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upproposal: Go 2: read-only types #22876
Comments
jba
added
Go2
Proposal
labels
Nov 25, 2017
gopherbot
added this to the Proposal milestone
Nov 25, 2017
bradfitz
added
the
LanguageChange
label
Nov 25, 2017
ianlancetaylor
changed the title from
proposal: read-only types
to
proposal: Go 2: read-only types
Nov 26, 2017
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Nov 26, 2017
Contributor
I understand the desire for this kind of thing, but I am not particularly fond of this kind of proposal. This approach seems very similar to the const qualifier in C, with the useful addition of permission genericity. I wrote about some of my concerns with const in https://www.airs.com/blog/archives/428.
You've identified the problems well: this does not provide immutability, and it does not avoid data races. I would like to see a workable proposal for immutability, and I would love to see one that avoids data races. This is not those proposals.
Using ro in a function parameter amounts to a promise that the function does not change the contents of that argument. That is a useful promise, but it is one of many possible useful promises. Is there a reason beyond familiarity with C that we should elevate this promise into the language? Go programs often rely on documentation rather than enforcement. There are many structs with exported fields with documentation about who is permitted to modify those fields. Similarly we document that a Write method that implements io.Writer may not modify its argument slice. Why put one promise into the language but not the other?
In general this is an area where experience reports can help guide Go 2 development. Does this proposal help with real problems that Go programmers have encountered?
|
I understand the desire for this kind of thing, but I am not particularly fond of this kind of proposal. This approach seems very similar to the You've identified the problems well: this does not provide immutability, and it does not avoid data races. I would like to see a workable proposal for immutability, and I would love to see one that avoids data races. This is not those proposals. Using In general this is an area where experience reports can help guide Go 2 development. Does this proposal help with real problems that Go programmers have encountered? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Nov 26, 2017
I would like to see a workable proposal for immutability, and I would love to see one that avoids data races. This is not those proposals.
I'm continuing to think about those things, but I wanted to get this proposal out there for two reasons. One, I think any proposal for immutability will have this as a subset. ro T is a subtype of both T and im T, so it will likely show up in any reasonable proposal involving im. (Permission genericity gets around using ro for functions, but you still might want it for data. Consider a data structure that wants to store both T and im T.) It's probably not an accident that Rust, Pony and Midori all have read-only types in addition to immutable ones.
The second reason I wanted to share this is that it serves as a counterexample to anyone who thinks adding read-only types to Go is just a matter of adding a keyword.
Does this proposal help with real problems that Go programmers have encountered?
Yes. At the recent Google-internal Go conference, @thockin specifically asked for const, citing bugs in Kubernetes due to inadvertent modification of values returned from caches. I think Alex Turcu also mentioned that he wanted something like this for an internal video ads system.
jba
commented
Nov 26, 2017
I'm continuing to think about those things, but I wanted to get this proposal out there for two reasons. One, I think any proposal for immutability will have this as a subset. The second reason I wanted to share this is that it serves as a counterexample to anyone who thinks adding read-only types to Go is just a matter of adding a keyword.
Yes. At the recent Google-internal Go conference, @thockin specifically asked for |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Nov 28, 2017
Contributor
What do you think of a builtin freeze function that returns an immutable shallow copy of an object? That would fix the cache problem without modifying the type system. (The returned value would be immutable in that any attempt to modify it would cause a run time panic.)
|
What do you think of a builtin |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Nov 28, 2017
Out of curiosity, how does that work? And how does it detect modification of a nested value?
jba
commented
Nov 28, 2017
|
Out of curiosity, how does that work? And how does it detect modification of a nested value? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Nov 28, 2017
Contributor
I don't know exactly how it works, which is why I haven't written a proposal for it. One conceivable implementation would be to mmap a new page, copy the object in, and then to mprotect that page, but the difficulties are obvious.
For a nested value, you use freeze multiple times, as desired.
|
I don't know exactly how it works, which is why I haven't written a proposal for it. One conceivable implementation would be to For a nested value, you use |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
neild
Nov 28, 2017
Contributor
I'm not following the distinction between values and variables in this proposal. Why is the modification of the value stored in a permitted below?
var a ro int
a = 1 // Modifying an ro int via a variable.
var b *ro int := &a
*b = 1 // Modifying an ro int via a pointer.A nitpick: Pointers-to-constants are entirely orthogonal to the rest of this proposal and (IMO) distract from the meat of it. Go already has syntax for constructing non-zero pointers to compound types; providing a similar facility for non-compound types does not require the addition of read-only values to the language. e.g., #19966.
|
I'm not following the distinction between values and variables in this proposal. Why is the modification of the value stored in var a ro int
a = 1 // Modifying an ro int via a variable.
var b *ro int := &a
*b = 1 // Modifying an ro int via a pointer.A nitpick: Pointers-to-constants are entirely orthogonal to the rest of this proposal and (IMO) distract from the meat of it. Go already has syntax for constructing non-zero pointers to compound types; providing a similar facility for non-compound types does not require the addition of read-only values to the language. e.g., #19966. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willfaught
Nov 29, 2017
Contributor
@jba Could you accomplish the same thing with overriding type operations? Is it important that the read-only property be at the type level? For example, string (basically a read-only []byte) could be defined as something like this:
type string []byte
func (s string) []=(index int) byte {
panic("not supported")
}This doesn't require any changes to the type system, and seems to be backward-compatible at first glance.
|
@jba Could you accomplish the same thing with overriding type operations? Is it important that the read-only property be at the type level? For example, string (basically a read-only []byte) could be defined as something like this: type string []byte
func (s string) []=(index int) byte {
panic("not supported")
}This doesn't require any changes to the type system, and seems to be backward-compatible at first glance. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Nov 29, 2017
@neild ro int is the same thing as int (actually, I disallow it, but that could go either way). ints are already immutable: you can't modify an int, only copy and change it. So your code is equivalent to
var a int
a = 1 // Modifying an ro int via a variable.
var b *int := &a
*b = 1 // Modifying an ro int via a pointer.and of course both of those assigments are equally legal. The assignment in
var c ro *int = &a
*c = 1would not be, but c itself could be changed.
I'm trying to avoid proposing both a type modifier and what C would call "storage class,", out of hygiene. (See Ian's blog post that he linked to above for a criticism of how C const conflates those.)
jba
commented
Nov 29, 2017
|
@neild var a int
a = 1 // Modifying an ro int via a variable.
var b *int := &a
*b = 1 // Modifying an ro int via a pointer.and of course both of those assigments are equally legal. The assignment in var c ro *int = &a
*c = 1would not be, but I'm trying to avoid proposing both a type modifier and what C would call "storage class,", out of hygiene. (See Ian's blog post that he linked to above for a criticism of how C |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Nov 29, 2017
@willfaught I don't think operator overloading is a good fit for Go. One of the nice things about the language is that every bit of syntax has a fixed meaning.
jba
commented
Nov 29, 2017
|
@willfaught I don't think operator overloading is a good fit for Go. One of the nice things about the language is that every bit of syntax has a fixed meaning. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
willfaught
Nov 29, 2017
Contributor
It seems identical to how methods and embedding work. Like the selector a.b, the operation a[b] could also be overridden. It would simplify things for operators to just be methods (that can be aggressively inlined).
|
It seems identical to how methods and embedding work. Like the selector |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
neild
Nov 30, 2017
Contributor
@jba Your proposal says:
Transitivity increases safety, and it can also simplify reasoning about read-only types. For example, what is the difference between ro *int and *ro int? With transitivity, the first is equivalent to ro *ro int, so the difference is just the permission of the full type.
The existence of *ro int implies the existence of ro int, doesn't it? If not, why not and what is the type of *p where p is a *ro int?
You also say:
It is a compile-time error to modify a value of read-only type,
I can't square this with it being legal to modify the value of c in your example:
var c ro *int = &aThe variable c has type ro *int. The value contained within c is a value of read-only type. Why can it be modified?
|
@jba Your proposal says:
The existence of You also say:
I can't square this with it being legal to modify the value of var c ro *int = &aThe variable |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Nov 30, 2017
Contributor
@willfaught Operator overloading is a very different idea that should be discussed in a separate proposal, not this one.
|
@willfaught Operator overloading is a very different idea that should be discussed in a separate proposal, not this one. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Dec 4, 2017
Transitivity increases safety, and it can also simplify reasoning about read-only types. For example, what is the difference between ro *int and *ro int? With transitivity, the first is equivalent to ro *ro int, so the difference is just the permission of the full type.
The existence of *ro int implies the existence of ro int, doesn't it? If not, why not and what is the type of *p where p is a *ro int?
That's a bug in my proposal. I chose a poor example. Replace int with []int.
It is a compile-time error to modify a value of read-only type,
I can't square this with it being legal to modify the value of c in your example:
var c ro *int = &aThe variable c has type ro *int. The value contained within c is a value of read-only type. Why can it be modified?
The it in your last sentence refers to the value of read-only type, the ro *int. That value cannot be modified; *c = 3 is illegal. But you can change the binding of c. There is nothing in my proposal that restricts the semantics of variable bindings.
The situation is analagous to
var s string = "x"
s = "y"
The value is immutable, but the variable binding is not.
jba
commented
Dec 4, 2017
That's a bug in my proposal. I chose a poor example. Replace
The it in your last sentence refers to the value of read-only type, the The situation is analagous to
The value is immutable, but the variable binding is not. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
neild
Dec 4, 2017
Contributor
It is possible that I have misunderstood the spec, but this is not consistent with my understanding of variable assignment. s = "y" does not change the binding of s; it changes the value of the variable bound to s.
|
It is possible that I have misunderstood the spec, but this is not consistent with my understanding of variable assignment. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Dec 5, 2017
I guess I'm using the word "binding" wrong. I was thinking variables are bound to their values, and you're saying identifiers are bound to variables, which have values. Anyway, you can change variable-value associations, but some values cannot be modified.
jba
commented
Dec 5, 2017
|
I guess I'm using the word "binding" wrong. I was thinking variables are bound to their values, and you're saying identifiers are bound to variables, which have values. Anyway, you can change variable-value associations, but some values cannot be modified. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Spriithy
Dec 27, 2017
Why not reuse the already existing const keyword to ease readability and stick with Go's spirit of not obfuscating intent ?
My point here is that ro is an obfuscating keyword that hides intent to non-aware readers. Again, as you stated earlier, this is merely a suggestion and syntax comes last.
Other point, say I have a read only type for ints. Is such type declarable (as in type T = ro int) ?
If yes, do I declare an instance of such type using the var or const const keyword since it is a non modifiable type ?
type T = ro int
var x T = 55
// or
const y T = 98Moreover, wouldn't it be enough to allow constant pointers ?
Other point, what about compound types ? Say, using these declarations
type S struct {
Exported int
}
type RoS = ro SDoes this snippet compile ? If not, what errors are thrown ? If yes, what is the expected behavior ? Does it panic ? If yes, how does the runtime detects this ?
func main() {
ros := &RoS{Exported: 55}
p := &ros.Exported
*p = 98
}What about this one ?
func main() {
ros := &RoS{Exported: 55}
p := (*int)(unsafe.Pointer(&ros.Exported))
*p = 98
}
Spriithy
commented
Dec 27, 2017
|
Why not reuse the already existing My point here is that Other point, say I have a read only type for ints. Is such type declarable (as in If yes, do I declare an instance of such type using the type T = ro int
var x T = 55
// or
const y T = 98Moreover, wouldn't it be enough to allow constant pointers ? Other point, what about compound types ? Say, using these declarations type S struct {
Exported int
}
type RoS = ro SDoes this snippet compile ? If not, what errors are thrown ? If yes, what is the expected behavior ? Does it panic ? If yes, how does the runtime detects this ? func main() {
ros := &RoS{Exported: 55}
p := &ros.Exported
*p = 98
}What about this one ? func main() {
ros := &RoS{Exported: 55}
p := (*int)(unsafe.Pointer(&ros.Exported))
*p = 98
} |
pciet
referenced this issue
Dec 28, 2017
Closed
proposal: Go 2: capability based security via stateless packages #23267
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaekwon
Dec 29, 2017
I just want to point to two proposals, one for immutable slices and one for pointerized structs that I think in combination amounts to a simpler set of language changes than what is proposed here. Please take a look!
What is needed is a constraint of the form "T is either []S or ro []S", that is, permission genericity.
Check out the any modifier in the immutable slices proposal.
Pointerized structs
Here's a concrete example. Here is one way to control write-access to structs. Copying is trivial, you can just do var g Foo = f from anywhere, even outside the module that declares Foo.
type Foo struct {
value interface{}
}
func (f *Foo) SetValue(interface{}) {...}
func (f Foo) GetValue() interface{} {...}The other way is to protect the struct with a mutex:
type Foo struct {
mtx sync.RWMutex
value interface{}
}
func (f *Foo) SetValue(interface{}) {...} // Lock/Unlock
func (f *Foo) GetValue() interface{} {...} // RLock/RUnlockHere's a full pointerized struct version:
type foo struct* {
Value interface{}
}
func (f foo) GetValue() interface{} {...}
type Foo struct {
mtx sync.RWMutex
foo
}
func (f *Foo) SetValue(interface{}) {...} // Lock/Unlock
func (f *Foo) GetValue() interface{} {...} // RLock/RUnlock
f = Foo{...}
f.SetValue(...) // ok, f is addressable
g := f
g.SetValue(...) // ok, g is addressable
func id(f Foo) Foo { return f } // returns a non-addressable copy
id(g).SetValue(...) // compile-error, not addressable.
id(g).GetValue(...) // calls foo.GetValue, mtx not needed
Q: So why readonly slices? It seems natural to create a "view" into an arbitrary-length array of objects without copying. For one, it's a required performance optimization. Second, there's no way to mark any items of a slice to be externally immutable, as can be done with private struct fields. For these reasons, readonly slices appear to be natural and necessary (for lack of any alternative).
jaekwon
commented
Dec 29, 2017
•
|
I just want to point to two proposals, one for immutable slices and one for pointerized structs that I think in combination amounts to a simpler set of language changes than what is proposed here. Please take a look!
Check out the Pointerized structsHere's a concrete example. Here is one way to control write-access to structs. Copying is trivial, you can just do type Foo struct {
value interface{}
}
func (f *Foo) SetValue(interface{}) {...}
func (f Foo) GetValue() interface{} {...}The other way is to protect the struct with a mutex: type Foo struct {
mtx sync.RWMutex
value interface{}
}
func (f *Foo) SetValue(interface{}) {...} // Lock/Unlock
func (f *Foo) GetValue() interface{} {...} // RLock/RUnlockHere's a full pointerized struct version: type foo struct* {
Value interface{}
}
func (f foo) GetValue() interface{} {...}
type Foo struct {
mtx sync.RWMutex
foo
}
func (f *Foo) SetValue(interface{}) {...} // Lock/Unlock
func (f *Foo) GetValue() interface{} {...} // RLock/RUnlock
f = Foo{...}
f.SetValue(...) // ok, f is addressable
g := f
g.SetValue(...) // ok, g is addressable
func id(f Foo) Foo { return f } // returns a non-addressable copy
id(g).SetValue(...) // compile-error, not addressable.
id(g).GetValue(...) // calls foo.GetValue, mtx not needed
Q: So why readonly slices? It seems natural to create a "view" into an arbitrary-length array of objects without copying. For one, it's a required performance optimization. Second, there's no way to mark any items of a slice to be externally immutable, as can be done with private struct fields. For these reasons, readonly slices appear to be natural and necessary (for lack of any alternative). |
jaekwon
referenced this issue
Dec 29, 2017
Closed
Proposal: Make Go2 better implement the Object-capability model for security. #23157
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
wora
Dec 29, 2017
I think this design would lead to significant complexity in practice, similar to C++ const. A couple of key issues:
- The caller is free to modify the value while it looks like a constant to the callee.
- If you read a field of ro T, what is the type of the field value? F or ro F?
- Having libraries to consistently use this new feature can be very challenging and costly.
One cheap alternative is to introduce a documentary type annotation, which just document the value should not be changed. There is no enforcement, but it offers a design contract between caller and callee. Go doesn't provide in-process security anyway, a bad library can do arbitrary damage. I am not sure whether we need to guard it at language level.
wora
commented
Dec 29, 2017
|
I think this design would lead to significant complexity in practice, similar to C++
One cheap alternative is to introduce a documentary type annotation, which just document the value should not be changed. There is no enforcement, but it offers a design contract between caller and callee. Go doesn't provide in-process security anyway, a bad library can do arbitrary damage. I am not sure whether we need to guard it at language level. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Jan 3, 2018
Why not reuse the already existing const keyword to ease readability and stick with Go's spirit of not obfuscating intent ?
const is about the identifier-value binding, while ro modifies types. I think it would be more confusing to conflate the two.
.. do I declare an instance of [an ro type] using the var or const const keyword since it is a non modifiable type ?
var, because the variable can still be set to a different value.
Moreover, wouldn't it be enough to allow constant pointers ?
No, ro is useful for anything that has structure, like maps and slices. You might want to return a map from a function without worrying that your callers will modify it, for example.
Does this snippet compile ? If not, what errors are thrown ? If yes, what is the expected behavior ? Does it panic ? If yes, how does the runtime detects this ?
func main() {
ros := &RoS{Exported: 55}
p := &ros.Exported
*p = 98
}
It fails to compile. p has type ro *int, so the assignment *p = 98 is illegal.
What about [using
unsafe]?
Of course, all bets are off with unsafe.
jba
commented
Jan 3, 2018
No,
It fails to compile.
Of course, all bets are off with |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Jan 3, 2018
Check out the
anymodifier in the immutable slices proposal.
I don't see how any actually works. Say I have x, which may be an roPeeker or an rwPeeker. Now I do
if y, ok := x.(interface{ Peek(int) any []byte }); ok {
b := y.Peek(3)
b[1] = 17 // ???
}
Can I assign to elements of b or not? Hopefully the compiler somehow knows and reports an error just in case x was an roSeeker. But I don't see how it knows that.
Here is one way to create immutable structs:
type Foo struct {
value interface{}
}
func (f *Foo) SetValue(interface{}) {...}
func (f Foo) GetValue() interface{} {...}
I don't understand this. What is immutable? Certainly not Foo—you can set its value field. (The field may as well be exported.) Is the thing I put in value immutable? Maybe; depends what I put there:
var f Foo
f.SetValue([]int{1})
x := f.GetValue()
x.([]int)[0] = 2 // Nope, not immutable.
jba
commented
Jan 3, 2018
I don't see how
Can I assign to elements of
I don't understand this. What is immutable? Certainly not
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Jan 3, 2018
I think this design would lead to significant complexity in practice, similar to C++
const.
I think it's a little less complex, but yes, I basically agree.
The caller is free to modify the value while it looks like a constant to the callee.
It doesn't look like a constant, it looks like a readonly value.
If you read a field of
ro T, what is the type of the field value?Forro F?
ro F
jba
commented
Jan 3, 2018
I think it's a little less complex, but yes, I basically agree.
It doesn't look like a constant, it looks like a readonly value.
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jaekwon
Jan 9, 2018
I don't see how
anyactually works. Say I havex, which may be anroPeekeror anrwPeeker. Now I doif y, ok := x.(interface{ Peek(int) any []byte }); ok { b := y.Peek(3) b[1] = 17 // ??? }Can I assign to elements of
bor not? Hopefully the compiler somehow knows and reports an error just in casexwas anroSeeker. But I don't see how it knows that.
No, you can't. any means it might be read-only, so first you must cast to a writeable.
y := x.(interface{ Peek(int) any []byte }) if wy, ok := x.(interface{ Peek(int) []byte }); ok { b := wy.Peek(3) b[1] = 17 }
Here is one way to create immutable structs:
type Foo struct { value interface{} } func (f *Foo) SetValue(interface{}) {...} func (f Foo) GetValue() interface{} {...}I don't understand this. What is immutable? Certainly not Foo—you can set its value field. (The field may as well be exported.)
I meant, you pass by value (e.g. copy) to prevent others from writing to it. Immutable is an overloaded word... I was using it to refer to pass-by-copy semantics.
f := &Foo{value: "somevalue"}
f.SetValue("othervalue") // `f` is a pointer
g := *f
g.SetValue("another") // can't, g is a readonly copy.The use-cases for ro struct{} overlap significantly for use-cases for g := *f, and the latter already exists. We don't need transitive ro as long as all field values are non-pointer types.
But I also acknowledge that Golang1 isn't perfectly suited for this kind of usage, because it forces you to write verbose and type-unsafe syntax to get the behavior you want... Here's an example with an (immutable) tree-like structure:
type Node interface {
AssertIsNode()
}
type node struct {
Left Node
Right Node
}
func (_ node) AssertIsNode() {}
// Using the struct is cumbersome, but overall this has the behavior we want.
// Interfaces are pointer-like in how its represented in memory,
// copying is quick and efficient.
var tree Node = ...
maliciousCode(tree) // cannot mutate my copy
// But using this as a struct is cumbersome and type-unsafe.
var leftValue = tree.(node).Left.(node).ValueMaybe one way to make this easier is to declare a struct to be "pointerized"...
type Node struct* {
Left Node
Right Node
}
var n Node = nil // not the same as a zero value.
n.Left = ... // runtime error
n = Node{}
n.Left = ...
n.Right = ...
var n2 = n
n2.Left = ... // This won't affect `n`.
n2.Left.Left = ... // compile-time error, n2.Left is not addressable.
n.Left = n // circular references are OK.Please check out #23162
@jba Please check out the update to the last comment: #22876 (comment)
jaekwon
commented
Jan 9, 2018
•
No, you can't.
I meant, you pass by value (e.g. copy) to prevent others from writing to it. Immutable is an overloaded word... I was using it to refer to pass-by-copy semantics. f := &Foo{value: "somevalue"}
f.SetValue("othervalue") // `f` is a pointer
g := *f
g.SetValue("another") // can't, g is a readonly copy.The use-cases for But I also acknowledge that Golang1 isn't perfectly suited for this kind of usage, because it forces you to write verbose and type-unsafe syntax to get the behavior you want... Here's an example with an (immutable) tree-like structure: type Node interface {
AssertIsNode()
}
type node struct {
Left Node
Right Node
}
func (_ node) AssertIsNode() {}
// Using the struct is cumbersome, but overall this has the behavior we want.
// Interfaces are pointer-like in how its represented in memory,
// copying is quick and efficient.
var tree Node = ...
maliciousCode(tree) // cannot mutate my copy
// But using this as a struct is cumbersome and type-unsafe.
var leftValue = tree.(node).Left.(node).ValueMaybe one way to make this easier is to declare a struct to be "pointerized"... type Node struct* {
Left Node
Right Node
}
var n Node = nil // not the same as a zero value.
n.Left = ... // runtime error
n = Node{}
n.Left = ...
n.Right = ...
var n2 = n
n2.Left = ... // This won't affect `n`.
n2.Left.Left = ... // compile-time error, n2.Left is not addressable.
n.Left = n // circular references are OK.Please check out #23162 @jba Please check out the update to the last comment: #22876 (comment) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
andlabs
Jan 31, 2018
Contributor
Has anyone listed all the existing proposals for declaring a block of data to be stored in read-only memory?
|
Has anyone listed all the existing proposals for declaring a block of data to be stored in read-only memory? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
iand
Mar 6, 2018
Contributor
This recent blog post on const in C++ and D has some relevant discussion of the difficulties of implementing useful const/immutable concepts in programming languages.
|
This recent blog post on const in C++ and D has some relevant discussion of the difficulties of implementing useful const/immutable concepts in programming languages. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Mar 27, 2018
Contributor
We are not certain about adding a type qualifier to the language. Go in general has a very simple type system. There is only one type qualifier at present: marking a channel as send-only or receive-only.
Also, I don't think anybody has address my comment from above:
Using ro in a function parameter amounts to a promise that the function does not change the contents of that argument. That is a useful promise, but it is one of many possible useful promises. Is there a reason beyond familiarity with C that we should elevate this promise into the language? Go programs often rely on documentation rather than enforcement. There are many structs with exported fields with documentation about who is permitted to modify those fields. Similarly we document that a Write method that implements io.Writer may not modify its argument slice. Why put one promise into the language but not the other?
Still, there is something to the ideas here, and the general concept might be worth pursuing. Keeping this proposal open for further discussion.
|
We are not certain about adding a type qualifier to the language. Go in general has a very simple type system. There is only one type qualifier at present: marking a channel as send-only or receive-only. Also, I don't think anybody has address my comment from above:
Still, there is something to the ideas here, and the general concept might be worth pursuing. Keeping this proposal open for further discussion. |
ianlancetaylor
added
the
NeedsInvestigation
label
Mar 27, 2018
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Mar 29, 2018
Ian, let me try address your comment—why this promise and not another?—while also putting this proposal in context. This is an expansion of my earlier comment.
There's a lot of interest in type systems that offer data-race freedom. (Russ even mentioned them in his Go Resolutions for 2017). The languages Rust, Pony and Midori all have different ways of eliminating data races, but they share the idea of using type modifiers that restrict access to values.
The trick is picking the right set of modifiers so that programs are both expressive and not too painful to write. For example, if you chose to add just an immutability modifier to an imperative language like Go, you'd find that it is hard to construct immutable values that have structure. How would you create an immutable slice containing non-zero values, or an immutable linked list?
So these languages all have a few different modifiers, that when used together in certain patterns let you prove little theorems (at least, that's how I think of it). For example, if I create a reference to a value from fresh memory at a point in the program, only put immutable values into it, and never let a copy of the reference out of scope, then I can convert that reference to an immutable reference. That particular theorem lets me create an immutable []int containing any values I like.
But that theorem is weaker than it needs to be. If I create a reference to a value from fresh memory, only put immutable values into it, and never let a writable copy of the reference out of scope, then I can still convert that reference to an immutable reference. That extra power lets me pass my reference to other functions while I build it up, provided they promise not to modify it.
So "can't modify this value" seems to be a useful building block in the machinery of constructing data-race free programs. Of course, my one example doesn't prove that, but it is interesting that all three of those languages, Rust, Pony and Midori, have something like the ro of this proposal.
In short, while this proposal might not add enough value to pull its weight, I think it's a necessary stepping stone to data-race freedom.
jba
commented
Mar 29, 2018
|
Ian, let me try address your comment—why this promise and not another?—while also putting this proposal in context. This is an expansion of my earlier comment. There's a lot of interest in type systems that offer data-race freedom. (Russ even mentioned them in his Go Resolutions for 2017). The languages Rust, Pony and Midori all have different ways of eliminating data races, but they share the idea of using type modifiers that restrict access to values. The trick is picking the right set of modifiers so that programs are both expressive and not too painful to write. For example, if you chose to add just an immutability modifier to an imperative language like Go, you'd find that it is hard to construct immutable values that have structure. How would you create an immutable slice containing non-zero values, or an immutable linked list? So these languages all have a few different modifiers, that when used together in certain patterns let you prove little theorems (at least, that's how I think of it). For example, if I create a reference to a value from fresh memory at a point in the program, only put immutable values into it, and never let a copy of the reference out of scope, then I can convert that reference to an immutable reference. That particular theorem lets me create an immutable But that theorem is weaker than it needs to be. If I create a reference to a value from fresh memory, only put immutable values into it, and never let a writable copy of the reference out of scope, then I can still convert that reference to an immutable reference. That extra power lets me pass my reference to other functions while I build it up, provided they promise not to modify it. So "can't modify this value" seems to be a useful building block in the machinery of constructing data-race free programs. Of course, my one example doesn't prove that, but it is interesting that all three of those languages, Rust, Pony and Midori, have something like the In short, while this proposal might not add enough value to pull its weight, I think it's a necessary stepping stone to data-race freedom. |
bcmills
referenced this issue
Mar 30, 2018
Open
proposal: spec: generic programming facilities #15292
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
See also #20443. |
ianlancetaylor
referenced this issue
Apr 16, 2018
Closed
proposal: Go 2: parameter tags for static analysis tools #24889
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
wora
Apr 17, 2018
I think the concept of read-only is much more general other other promises. Anyone who uses computer, not just developers, has good understanding of read-only files or read-only documents. People have very little problem dealing with such things in their daily life.
Extending the concept to programming language is not a big cost for developers. C++ added constexpr must later in its lifecycle and it works reasonably well.
wora
commented
Apr 17, 2018
|
I think the concept of read-only is much more general other other promises. Anyone who uses computer, not just developers, has good understanding of read-only files or read-only documents. People have very little problem dealing with such things in their daily life. Extending the concept to programming language is not a big cost for developers. C++ added |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
bcmills
Apr 17, 2018
Member
Reading through @rsc's critique of @bradfitz's 2013 proposal, I'm struck by just how many of the issues Russ raised boil down to parametricity and/or metaprogramming.
The TrimSpace example in “Duplication and triplication”
func bytes.TrimSpace(s []byte) []byte func strings.TrimSpace(s string) stringcannot be replaced by
func strings.TrimSpace(s readonly []byte) readonly []byte
seems like it would be fixed by using parametricity instead:
func [T] TrimSpace(s T) TThe Join example
func bytes.Join(x [][]byte, sep readonly []byte) []byte func strings.Join(x []string, sep readonly []byte) string func robytes.Join(x []readonly []byte, sep readonly []byte) []byte
is similar, and requires only that readonly string collapse to string:
func [T] Join(x []readonly T, sep readonly []byte) TThe issue of “Immutability and memory allocation” is more difficult: a notion of “read-only” without some stronger notion of “strict” or “unowned” does not suffice to prevent subtle aliasing bugs. (The race detector will catch a few of those, but certainly not all.) That hints at a stronger (and larger) version of this proposal, but the stronger and larger the proposal, the less likely it is to fit into the scale of changes for Go 2.
On the other hand, given that Go programs are already susceptible to subtle aliasing bugs, that strikes me as kind of a silly reason to reject an incremental safety improvement.
In the “Loss of generality” section, the TrimSpace/ToUpper example for seems too trivial (even more so if #21498 is accepted): it's easy enough to make the types match up using a function literal.
var convert = robytes.TrimSpace
if wackyContrivedExample {
convert = func(x []byte) []byte { return robytes.ToUpper(x) }
}We could even encode that directly in the type system by treating functions and methods that accept a readonly T as a subtype of functions that accept a T (and treating functions that return a T as a subtype of functions that return a readonly T). That would also address the example of Bytes and Peek methods in interfaces. We'd just need to be careful to limit covariance to functions (and not repeat the Java array-covariance mistake).
The final “Loss of generality” example, sort.IntSlice, is moot because of sort.SliceIsSorted, which would work with read-only slices as-is. The sort API seems to break with readonly slices only because it is already overspecified: it uses the same type for both reading and writing.
That said, it could also be addressed with a bit more (IMO simple) metaprogramming: a compile-time conditional to provide Swap only if the slice type is mutable.
type [T] CmpSlice T
func (x CmpSlice) Less(i, j int) bool { return x[i] < x[j] }
func (x CmpSlice) Len() int { return len(x) }
[if Mutable(T)] func (x CmpSlice[T]) Swap(i, j int) { x[i], x[j] = x[j], x[i] }Where I'm going with all this is that, in my opinion, the decision for this proposal should depend on the outcome of https://golang.org/issue/15292. If Go 2 has workable generics, many of the problems Russ observed will become moot; if Go 2 has workable generics with compile-time reflection or other metaprogramming, then the only remaining issue will be that this proposal does not go far enough.
|
Reading through @rsc's critique of @bradfitz's 2013 proposal, I'm struck by just how many of the issues Russ raised boil down to parametricity and/or metaprogramming. The
seems like it would be fixed by using parametricity instead: func [T] TrimSpace(s T) TThe
is similar, and requires only that func [T] Join(x []readonly T, sep readonly []byte) TThe issue of “Immutability and memory allocation” is more difficult: a notion of “read-only” without some stronger notion of “strict” or “unowned” does not suffice to prevent subtle aliasing bugs. (The race detector will catch a few of those, but certainly not all.) That hints at a stronger (and larger) version of this proposal, but the stronger and larger the proposal, the less likely it is to fit into the scale of changes for Go 2. On the other hand, given that Go programs are already susceptible to subtle aliasing bugs, that strikes me as kind of a silly reason to reject an incremental safety improvement. In the “Loss of generality” section, the var convert = robytes.TrimSpace
if wackyContrivedExample {
convert = func(x []byte) []byte { return robytes.ToUpper(x) }
}We could even encode that directly in the type system by treating functions and methods that accept a The final “Loss of generality” example, That said, it could also be addressed with a bit more (IMO simple) metaprogramming: a compile-time conditional to provide type [T] CmpSlice T
func (x CmpSlice) Less(i, j int) bool { return x[i] < x[j] }
func (x CmpSlice) Len() int { return len(x) }
[if Mutable(T)] func (x CmpSlice[T]) Swap(i, j int) { x[i], x[j] = x[j], x[i] }Where I'm going with all this is that, in my opinion, the decision for this proposal should depend on the outcome of https://golang.org/issue/15292. If Go 2 has workable generics, many of the problems Russ observed will become moot; if Go 2 has workable generics with compile-time reflection or other metaprogramming, then the only remaining issue will be that this proposal does not go far enough. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jba
Apr 17, 2018
@bcmills, my "permission genericity" was an attempt to address Russ's points without invoking full genericity. If we had full genericity, then permission genericity might not be necessary—but Midori found that both were needed. See http://joeduffyblog.com/2016/11/30/15-years-of-concurrency, search for "generic parameterization over permissions".
jba
commented
Apr 17, 2018
|
@bcmills, my "permission genericity" was an attempt to address Russ's points without invoking full genericity. If we had full genericity, then permission genericity might not be necessary—but Midori found that both were needed. See http://joeduffyblog.com/2016/11/30/15-years-of-concurrency, search for "generic parameterization over permissions". |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
ianlancetaylor
Apr 17, 2018
Contributor
@wora I don't really accept that kind of argument from analogy in general, but in this case I think the analogy is flawed anyhow. This proposal is not about constexpr (which is a kind of immutability, not a read-only type qualifier) and it's not about documents that can not be changed. It's about saying that a value can not be changed using a specific reference to that value, but the value can still be changed using other references.
@bcmills I have to say that I think it is extremely unlikely that Go 2 will provide generics with compile-time reflection or metaprogramming. Yours is the only generics proposal I've ever seen with anything close to that. All of my proposals have specifically not included it, which I personally regard as a feature.
|
@wora I don't really accept that kind of argument from analogy in general, but in this case I think the analogy is flawed anyhow. This proposal is not about @bcmills I have to say that I think it is extremely unlikely that Go 2 will provide generics with compile-time reflection or metaprogramming. Yours is the only generics proposal I've ever seen with anything close to that. All of my proposals have specifically not included it, which I personally regard as a feature. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
bcmills
Apr 18, 2018
Member
@jba, I guess part of my point is that this proposal can be made quite a bit simpler if we already have some form of generics in the language anyway.
|
@jba, I guess part of my point is that this proposal can be made quite a bit simpler if we already have some form of generics in the language anyway. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
bcmills
Apr 18, 2018
Member
@ianlancetaylor Yes, I would be surprised if Go 2 had generics with any sort of general-purpose metaprogramming mechanism. However, I think this case is worth mentioning for two reasons:
- Even without metaprogramming, simple generics address many of Russ's objections to Brad's original proposal.
- This use-case, and potentially others like it, may be considerations when we decide how much (if any) metaprogramming to support (either in Go 2, or further in the future if we do not choose a design that precludes them in Go 2).
|
@ianlancetaylor Yes, I would be surprised if Go 2 had generics with any sort of general-purpose metaprogramming mechanism. However, I think this case is worth mentioning for two reasons:
|
This was referenced Apr 19, 2018
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
celestiallake
May 22, 2018
You have to take a look at closures. Go supports them widely. Don't know if there's any nice use for const keyword since that's just an expensive compile-time check.
celestiallake
commented
May 22, 2018
|
You have to take a look at closures. Go supports them widely. Don't know if there's any nice use for |
jba commentedNov 25, 2017
I propose adding read-only types to Go. Read-only types have two related benefits:
An additional minor benefit is the ability to take the address of constants.
This proposal makes significant changes to the language, so it is intended for Go 2.
All new syntax in this proposal is provisional and subject to bikeshedding.
Basics
All types have one of two permissions: read-only or read-write. Permission is a property of types, but I sometimes write "read-only value" to mean a value of read-only type.
A type preceded by
rois a read-only type. The identifierrois pronounced row. It is a keyword. There is no notation for the read-write permission; any type not marked withrois read-write.The
romodifier can be applied to slices, arrays, maps, pointers, structs, channels and interfaces. It cannot be applied to any other type, including a read-only type:ro ro Tis illegal.It is a compile-time error to
append,A value of read-only type may not be immutable, because it may be referenced through another type that is not read-only.
Examples:
The compiler guarantees that the bytes of
datawill not be altered bytransmit.This proposal is concerned exclusively with avoiding modifications to values, not variables. Thus it allows assignment to variables of read-only type.
One could imagine a companion proposal that also used
ro, but to restrict assignment:I don't pursue that idea here.
Conversions
There is an automatic conversion from
Ttoro T. For instance, an actual parameter of type[]intcan be passed to a formal parameter of typero []int. This conversion operates at any level: a[][]intcan be converted to a[]ro []intfor example.There is an automatic conversion from
stringtoro []byte. It does not apply to nested occurrences: there is no conversion from[][]stringto[]ro []byte, for example.(Rationale:
rodoes not change the representation of a type, so there is no cost to addingroto any type, at any depth. A constant-time change in representation is required to convert fromstringtoro []bytebecause the latter is one word larger. Applying this change to every element of a slice, array or map would require a complete copy.)Transitivity
Permissions are transitive: a component retrieved from a read-only value is treated as read-only.
For example, consider
var a ro []*int. It is not only illegal to assign toa[i]; it is also illegal to assign to*a[i].Transitivity increases safety, and it can also simplify reasoning about read-only types. For example, what is the difference between
ro *intand*ro int? With transitivity, the first is equivalent toro *ro int, so the difference is just the permission of the full type.The Address Operator
If
vhas typero T, then&vhas type*ro T.If
vhas typeT, thenro &vhas typero *T. This bit of syntax simplifies constructing read-only pointers to struct literals, likero &S{a: 1, b: 2}.Taking the address of constants is permitted, including constant literals. If
cis a constant of typeT, then&cis of typero *Tand is equivalent toRead-Only Interfaces
Any method of an interface may be preceded by
ro. This indicates that the receiver of the method must have read-only type.If
Iis an interface type, thenro Iis effectively the sub-interface that contains just the read-only methods ofI. If typeTimplementsI, then typero Timplementsro I.Read-only interfaces can prevent code duplication that might otherwise result from the combination of read-only types and interfaces. Consider the following code from the
sortpackage:We would like to allow
IntsAreSortedto accept a read-only slice, since it does not change its argument. But we cannotcast
ro []inttoIntSlice, because theSwapmethod modifies its receiver. It seems we must copy code somewhere.The solution is to mark the first two methods of the interface as read-only:
Now we can write
IsSortedin terms of the read-only sub-interface:and call it on a read-only slice:
Permission Genericity
One of the problems with read-only types is that they lead to duplicate functions. For example, consider this trivial function, ignoring its obvious problem with zero-length slices:
We cannot call
tail1on values of typero []int, but we can take advantage of the automatic conversion to writeThanks to the conversion from read-write to read-only types,
tail2can be passed an[]int. But it loses type information, because the return type is alwaysro []int. So the first of these calls is legal but the second is not:If we had to write two variants of every function like this, the benefits of read-only types would be outweighed by the pain they cause.
To deal with this problem, most programming languages rely on overloading. If Go had overloading, we would name both of the above functions
tail, and the compiler would choose which to call based on the argument type. But we do not want to add overloading to Go.Instead, we can add generics to Go—but just for permissions. Hence permission genericity.
Any type inside a function, including a return type, may be preceded by
ro?instead ofro. Ifro?appears in a function, it must appear in the function's argument list.A function with an
ro?argumentamust type-check in two ways:ahas typero Tandro?is treated asro.ahas typeTandro?is treated as absent.In calls to a function with a return type
ro? T, the effective return type isTif thero?argumentais a read-write type, andro Tifais a read-only type.Here is
tailusing this feature:tailtype-checks because:xdeclared asro []int, the slice expression can be assigned to the effective return typero []int.xdeclared as[]int, the slice expression can be assigned to the effective return type[]int.This call succeeds because the effective return type of
tailisro []intwhen the argument isro []int:This call also succeeds, because
tailreturns[]intwhen its argument is[]int:Multiple, independent permissions can be expressed by using
ro?,ro??, etc. (If the only feasible type-checking algorithm is exponential, implementations may restrict the number of distinctro?...forms in the same function to a reasonable maximum, like ten.)In an interface declaration,
ro?may be used before the method name to refer to the receiver.There are no automatic conversions from function signatures using
ro?to signatures that do not usero?. Such conversions can be written explicitly. Examples:Permission genericity can be implemented completely within the compiler. It requires no run-time support. A function annotated with
ro?requires only a single implementation.Strengths of This Proposal
Fewer Bugs
The use of
roshould reduce the number of bugs where memory is inadvertently modified. There will be fewer race conditions where two goroutines modify the same memory. One goroutine can still modify the memory that another goroutine reads, so not all race conditions will be eliminated.Less Copying
Returning a reference to a value's unexported state can safely be done without copying the state, as shown in Example 2 above.
Many functions take
[]bytearguments. Passing a string to such a function requires a copy. If the argument can be changed toro []byte, the copy won't be necessary.Clearer Documentation
Function documentation often states conditions that promise that the function doesn't modify its argument, or that extracts a promise from the caller not to modify a return value. If
roarguments and return types are used, those conditions are enforced by the compiler, so they can be deleted from the documentation. Furthermore, readers know that in a well-designed function, a non-roargument will be written along at least one code path.Better Static Analysis Tools
Read-only annotations will make it easier for some tools to do their job. For example, consider a tool that checks whether a piece of memory is modified by a goroutine after it sends it on a channel, which may indicate a race condition. Of course if the value is itself read-only, there is nothing to do. But even if it isn't, the tool can do its job by checking for writes locally, and also observing that the value is passed to other functions only via read-only argument. Without
roannotations, the check would be difficult (requiring examining the code of functions not in the current package) or impossible (if the call was through an interface).Less Duplication in the Standard Library
Many functions in the standard library can be removed, or implemented as wrappers over other functions. Many of these involve the
stringand[]bytetypes.If the
io.Writer.Writemethod's argument becomes read-only, thenio.WriteStringis no longer necessary.Functions in the
stringspackage that do not return strings can be eliminated if the correspondingbytesmethod usesro. For example,strings.Index(string, string) intcan be eliminated in favor of (or can trivially wrap)bytes.Index(ro []byte, ro []byte) int. This amounts to 18 functions (includingReplacer.WriteString). Also, thestrings.Readertype can be eliminated.Functions that return
stringcannot be eliminated, but they can be implemented as wrappers around the correspondingbytesfunction. For example,bytes.ToLowerwould have the signaturefunc ToLower(s ro? []byte) ro? []byte, and thestringsversion could look likeThe conversion to
stringinvolves a copy, butToLoweralready contains a conversion from[]bytetostring, so there is no change in efficiency.Not all
stringsfunctions can wrap abytesfunction with no loss of efficiency. For instance,strings.TrimSpacecurrently does not copy, but wrapping it aroundbytes.TrimSpacewould require a conversion from[]bytetostring.Adding
roto the language without some sort of permission genericity would result in additional duplication in thebytespackage, since functions that returned a[]bytewould need a corresponding function returningro []byte. Permission genericity avoids this additional duplication, as described above.Pointers to Literals
Sometimes it's useful to distinguish the absence of a value from the zero value. For example, in the original Google protobuf implementation (still used widely within Google), a primitive-typed field of a message may contain its default value, or may be absent.
The best translation of this feature into Go is to use pointers, so that, for example, an integer protobuf field maps to the Go type
*int. That works well except for initialization: without pointers to literals, one must writeor use a helper function.
In Go as it currently stands, an expression like
&3cannot be permitted because assignment through the resulting pointer would be problematic. But if we stipulate that&3has typero *int, then assignment is impossible and the problem goes away.Weaknesses of This Proposal
Loss of Generality
Having both
Tandro Tin the language reduces the opportunities for writing general code. For example, an interface method with a[]intparameter cannot be satisfied by a concrete method that takesro []int. A function variable of typefunc() ro []intcannot be assigned a function of typefunc() []int. Supporting these cases would start Go down the road of covariance/contravariance, which would be another large change to the language.Problems Going from
stringtoro []byteWhen we change an argument from
stringtoro []byte, we may eliminate copying at the call site, but it can reappear elsewhere because the guarantee is weaker: the argument is no longer immutable, so it is subject to change by code outside the function. For example,os.Openreturns an error that contains the filename. If the filename were not immutable, it would have to be copied into the error message. Data structures like caches that need to remember their methods' arguments would also have to copy.Also, replacing
stringwithro []bytewould mean that implementers could no longer compare via operators, range over Unicode runes, or use values as map keys.Subsumed by Generics
Permission genericity could be subsumed by a suitably general design for generics. No such design for Go exists today. All known constraints on generic types use interfaces to express that satisfying types must provide all the interface's methods. The only other form of constraint is syntactic: for instance, one can write
[]T, whereTis a generic type variable, enforcing that only slice types can match. What is needed is a constraint of the form "Tis either[]Sorro []S", that is, permission genericity. A generics proposal that included permissions would probably drop the syntax of this proposal and use identifiers for permissions, e.g.Missing Immutability
This proposal lacks a permission for immutability. Such a permission has obvious charms: immutable values are goroutine-safe, and conversion between strings and immutable byte slices would work in both directions.
The problem is how to construct immutable values. Literals of immutable type would only get one so far. For example, how could a program construct an immutable slice of the first N primes, where N is a parameter? The two easy answers—deep copying, or letting the programmer assert immutability—are both unpalatable. Other solutions exist, but they would require additional features on top of this proposal. Simply adding an
imkeyword would not be enough.Does Not Prevent Data Races
A value cannot be modified through a read-only reference, but there may be other references to it that can be modified concurrently. So this proposal prevents some but not all data races. Modern languages like Rust, Pony and Midori have shown that it is possible to eliminate all data races at compile time. But the cost in complexity is high, and the value unclear—there would still be many opportunities for race conditions. If Go wanted to explore this route, I would argue that the current proposal is a good starting point.
References
Brad Fitzpatrick's read-only slice proposal
Russ Cox's evaluation of the proposal. This document identifies the problem with the
sortpackage discussed above, and raises the problem of loss of generality as well as the issues that arise in moving fromstringtoro []byte.Discussion on golang-dev