New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: spec: add read-only slices and maps as function arguments #20443

Open
henryas opened this Issue May 21, 2017 · 54 comments

Comments

Projects
None yet
@henryas
Copy link

henryas commented May 21, 2017

Background

I recently stumbled upon a bug and after hours of searching, I managed to find the culprit, which is a function that accidentally modifies the slice argument passed to it. The problem is that as I took several subsets of the original slice and gave them new identities, I forgot that they were still pointing to the original slice. I don't think this is an uncommon issue, and hence the proposal below.

Proposal

I would like to suggest the idea of defining a read-only slice and map in the function arguments. For example, we can define a function as follows:

//function definition
func MyFunction(const mySlice []byte, const myMap map[string][]byte) []byte

//usage
mySlice := []byte {0x00, 0x01} //normal slice definition. There is no weird const whatsoever.
myMap := make(map[string][]byte) //normal map definition.

//you are still using slice and map as you normally would. Slice and map are still modifiable,
//but MyFunction cannot alter the original variables.
MyFunction(mySlice, myMap)

//you can still make changes here
myMap["Me"] = 0x01

or in the interface definition:

type MyInterface { MyFunction(const mySlice []byte) []byte }

Why slices and maps?

With structs, interfaces and built-in types, it is easy to tell whether you want them to be modifiable or not. With slices and maps, it is not so easy to tell. Maps are probably less prone to accidental changes, but slices are tricky, especially when they get passed around many functions.

Implementation Options

Read-only slices and maps are shallow-copied upon being passed to the function. Hence, any accidental change is local to the function. I understand that this will not prevent interior mutability, but exterior mutability is good enough. I think this implementation option is probably less intrusive and there are fewer breaking changes to existing codes (if any) - possibly none.

Alternatively, we may have a more elaborate system where the compiler will not compile if a function tries to modify a read-only slice/map. However, this leads to a very complex solution. What if a new sub-slice is taken from an existing read-only slice, should it be immutable too? Now, what if that sub-slice is passed into another function, should the compiler check whether the other function preserves the integrity of the sub-slice?

I personally tend to lean with the first option.

Syntax

In order not to add any new keyword, I am thinking of reusing const, and it is also more intuitive to those familiar with C-family programming languages. However, I am open to suggestions.


Let me know what you think.

Thanks

@gopherbot gopherbot added this to the Proposal milestone May 21, 2017

@gopherbot gopherbot added the Proposal label May 21, 2017

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented May 21, 2017

This has been discussed before, and such big changes to the language won't be made in 1.x. I believe there were proposals to also make string be a constant []byte.

@dgryski

This comment has been minimized.

Copy link
Contributor

dgryski commented May 22, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented May 22, 2017

Russ Cox's evaluation applies to the second part of my possible implementation options. While I welcome full-blown read-only types, I do agree that it is a complex issue that must be thoroughly analyzed.

However, what I originally required is nothing revolutionary. See the first implementation option. It can be done in Go 1.x now, but it requires boilerplate for each type. See illustration. It would be nice to have a keyword to do automatic shallow-copying for slice and map for all types without the boilerplates. The objective is to protect the original variables against accidental changes. After slicing and re-slicing and several functions later, it was crazy to track all those mutated slices to find who did what that caused the bug.

@dnfield

This comment has been minimized.

Copy link

dnfield commented Jun 16, 2017

I think this is a great idea, but I don't like the idea of shallow copying. There's nothing stopping you from creating your own types over a slice that do this, and that way at least it's very transparent to you and consumers that they're actually getting copies of data.

I'd love to see go support read only types that are enforced at compile time. Slices in particular - simply have a compile error if someone tries to invoke code that would involve assigning to a slice that's const or read only. I understand it'd be a bit more involved than that, but it'd be very helpful.

@rsc rsc changed the title Proposal: Read Only Slices and Maps proposal: spec: add read-only slices and maps Jun 16, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 17, 2017

Go's slices are way too powerful and flexible that people rarely have to deal with arrays directly any more. In my opinion, a slice should be a read-only view into an array. So if people want to change the content of an array, they would need to deal with the array and not its slice.

If we are going into that direction, then there is no need for new keywords to indicate immutability.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 17, 2017

@dnfield

This comment has been minimized.

Copy link

dnfield commented Jun 17, 2017

I do not agree that slices should become exclusively read only. Being able to modify slices is an attractive feature, and changing that would be a very radical breaking change.

Appending to a read only slice should throw a compiler error, like any other operation that involves assignment

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 17, 2017

How about this? You can still work with slices as usual. You can slice, re-slice, append, make it point to different values, etc. However, regardless what you do to the slice, it won't change the backing arrays and slices. You can think of a slice like a pointer. You can make it point to different values at any time, but in this case you can't change the value you are pointing at.

@leiser1960

This comment has been minimized.

Copy link

leiser1960 commented Jun 17, 2017

What is the goal? Better documentation, or race free code?
Your proposal goes for the first, I assume.
You give the the guarantee that your function will have no way to alter the underlaying array.
But you do not get the guarantee that your function is race free in respect of changes to this array, because other goroutines may modify the underlaying array concurrently.
Adding immutability to golang should go for both goals, i.e. full value semantics as we have it for strings. This changes assignment as well as comparison semantics. But would help significantly reasoning about non sequential program behaviour.
Adding a simple form of immutibility to the type system of go is big challenge, both for language design and implementation. But please go for it!

@dnfield

This comment has been minimized.

Copy link

dnfield commented Jun 17, 2017

@leiser1960

This comment has been minimized.

Copy link

leiser1960 commented Jun 17, 2017

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 17, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 18, 2017

@dave: the final value of x should remain unchanged, which is {1,2,3}

Anyhow, please do not take this proposal strictly as is. The idea is there, but implementation-wise I am not so certain, which is why I welcome everybody's inputs on this matter. Some form of control over a variable's mutability will certainly be beneficial to Go.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 18, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 18, 2017

@dave: y is a new slice that points to X's first two values. X remains as it is. Y is just a "pointer" to x. Z is a new slice that points to Y's values and 5, but it doesn't change x and y. Even if you were to do z[0] = 7, z at index 0 just points to a new value of 7, but it doesn't change x and y.

But I welcome any other idea. It was just something that came to me as I was driving home from work yesterday.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 18, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 18, 2017

Nope. Not a copy. If someone are to change x, the change will cascade down to y and z as long as they are still pointing to any value of x.

Think of a slice like pointers to some values. When the values change, the slice will change too. However, when the slice is changed, it just points to the new values. The old values remain unaffected.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 18, 2017

@henryas

This comment has been minimized.

Copy link

henryas commented Jun 18, 2017

Think about the table of contents (TOC) of a book. The TOC is the slice, the content of the book are the actual values. If you need to find something, you look up the TOC, find the page it refers to, and find the page and read the information.

When the content of a page changes, naturally people who looks up via TOC will see the change. Hence, changes in x will cascade down to y and z.

However, when you change the entries in the TOC, eg. changing the page number a TOC entry points to, people use the TOC will see the new page, but the content of the book remains unchanged. Hence, changes in z does not cascade up to x and y.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 18, 2017

@dnfield

This comment has been minimized.

Copy link

dnfield commented Jun 18, 2017

It seems to me that we have at least a few proposals here:

  1. Most basically, support for some kind of read-only/const declaration of slices (and perhaps maps), to trigger a compiler error if you try to do anything to that slice that would modify it. Thus all assignments/appends would fail, as would any attempt to create a writable (non-const) sub-slice from it. This should be a non-breaking change, as effectively a read-only slice would be a new type that is not currently used.
  2. Immutable types. While this could be done without exposing the plumbing for it to users, it's hard to say how that would really help (particularly if you want to be able to send users a read-only slice to read from without creating a copy of the memory). So, for example, point 1 might enable something like:
    func (const []byte) AppendBytes(const []byte bytes) const []byte
  3. Completely changing the nature of slices in a non-compatible way. I'm struggling to see the value of this - users are used to reasoning about mutable slices (even if it's complicated to do so and error prone). There's lots and lots of code out there that uses mutable slices. I strongly oppose the idea that slices should automatically allocate new memory (or, more new memory than the minimum needed for the pointer/len/capacity) - the whole point of slices is that they avoid these allocations.

For what it's worth, I see this as parallel to some of the work going on in .NET Core 2.0 around Span<T> and ReadOnlySpan<T>, with slice as is being parallel to Span<T>. I also see this as a very valuable tool for use in concurrency scenarios where it's critical to avoid unnecessary allocations and data copying for read only usage. I'm less concerned with mistakes someone might make from passing slices when they really should be using arrays (which seems to be a bigger concern for @henryas )

@leiser1960

This comment has been minimized.

Copy link

leiser1960 commented Jun 18, 2017

The proposal here is not about immutable types, it is about your point 1 or 2
@davecheney question was, if y is a const slice (which is my interpretation) i.e.
x := int{1,2,3}
var y const []int = x[:2]
z := append(y, 5)
What is z pointing to? Will the last line compile?

You may prohibit the last line completely, or allow it. In the second case z must point to a new array, and be of type []int. append applied to a const slice behaves as if len==cap. This would be a consistent extension, a const slice simply has no cap at all.

My concerns are about the question:
x := int{1,2,3}
var y const []int = x[:2]
z := x[:2]
x[0] = 9. // or z[0] = 9 which is the same
b := y[0] == z[0]
Is b true or false?
This proposal says true.
Immutability would require it to be false. Reasoning about the program with immutability is much simpler because immutable values are race free. Implementing it efficiently may be a bit harder.

The question:
Does z and y point to x?
Is not a relevant question for immutable types. It must not be detectable! (It may as long as no one changes x as in my example.)
But it is of big importance for the const proposal and race detection.

@creker

This comment has been minimized.

Copy link

creker commented Jun 18, 2017

I think simple const keyword like in C could be naturally (and probably within Go 1.x) extended to a shallow immutable solution. Instead of throwing compile errors, on any change to the const slice compiler could emit a shallow copy operation. Changing an element in a const slice - make a copy and make the change to that copy. Appending - make a copy and append to that. Crossing any boundary between const and non-const should make a copy. Of course, these copy operations must be thread-safe. Otherwise it's useless.

The big problem with that is hidden copy operations and change of semantics. You're no longer sure exactly how your assignment/append will work without looking up the type of the slice. We kind of have the same problem with methods with value/pointer receivers but it's less obvious here.

@davecheney

This comment has been minimized.

Copy link
Contributor

davecheney commented Jun 18, 2017

@cespare

This comment has been minimized.

Copy link
Contributor

cespare commented Jun 18, 2017

Read-only slice types seem like a more or less reasonable proposal -- which is why Russ's evaluation of such a proposal (linked to by @dgryski earlier) was very in-depth and involved a real implementation of the feature. By contrast, I think that immutable types which do full copies of themselves transparently are a fairly unserious feature in a language where performance is even a vague concern.

Immutable types in other languages allow you to "modify" them (i.e. get a new object that's a modification of the old object while leaving the old object untouched) without doing full copies all over the place. For example, in Clojure, the persistent types such as vectors and maps are backed by tree structures which allow multiple objects to share common structure. When you append something to a vector or associate a new key/value into a map, the cost of that operation and the amount of extra space used is ~constant, not linear in the size of the vector/map.

That said, I doubt that any form of immutable/persistent data structures like these fit well in the Go language (though I look forward to Russ's evaluation of immutability in the future).

@henryas

This comment has been minimized.

Copy link

henryas commented Jul 13, 2017

How about turning the built-in arrays, slices, and maps into 'objects' with methods attached to them? That way we can use interface to create their read-only types and the compiler can enforce their intended usage as defined by the interface. In fact, the user can define their own version of restricted maps, slices, and arrays.

With this approach, there is no need for any new keyword. We can further slim down Go's specs by removing the array, slices, and maps related syntax. The behavior of the array, slices and maps will now be defined by their methods.

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Jul 20, 2017

I don't have any specific comments on this proposal; rather just wanted to point out an experiment with immutable data structures that I've put together: https://github.com/myitcv/immutable

The principal focus of this experiment has been the interface to the code-generated immutable data structures; there are clearly countless optimisations etc that could be made to the underlying implementation.

Given we don't have any language-level support for immutable data structures, in our approach we're limited to implementing immutability via types and their methods. This gives the overall interface a rather bloated feel, but despite this, as others have commented above, in certain situations it becomes significantly easier to reason about things knowing you are working with immutable data structures (nothing new in that statement obviously).

Like @cespare, I'm interested to hear the result of @rsc's evaluation of immutability.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Jul 20, 2017

@myitcv I skimmed your wiki page, and it's not obvious to me that you are describing what I would call immutable data structures. I may well have missed something but it looks more like C-style const references--that is, references to a data structure such that you can not change the data structure through that reference, but the data structure can be changed in other ways through other non-const references.

If I'm right about that, then I think that is a reasonable thing to discuss, but I don't think we should use the name "immutable" for it. To me "immutable" connotes a data structure that can not change after it has been initialized.

My apologies if I misunderstand.

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Jul 20, 2017

@ianlancetaylor - I might well be using the term incorrectly, so any apology is just as likely required from my side!

The wiki doesn't do a great job of providing a motivating example, so I'll try and provide one here (and then refine the wiki). And if we conclude my use of the term "immutable" is incorrect, then I'll fix that too, but bear with its use for what follows.

immutableGen is a go generate generator that creates immutable struct, map and slice type declarations from template type declarations.

For example, consider the declaration of an immutable struct type Person:

// person.go
package example

import "fmt"

//go:generate immutableGen

// via go generate, this template is code generated into the immutable Person
// struct within the same package
type _Imm_Person struct {
	Name string
	Age  int
}

// Hence we can then define methods on *Person (methods can only be defined on
// a pointer receiver)
func (p *Person) String() string {
	return fmt.Sprintf("Person{ Name: %q, Age: %v}", p.Name(), p.Age())
}

The type Person is declared in the generated code

A quick test of the resulting types:

package example_test

import (
	"fmt"
	"testing"

	"myitcv.io/immutable/example"
)

func TestThis(t *testing.T) {
	// the zero value of Person is immutable
	p1 := new(example.Person)

	fmt.Printf("p1: %v\n", p1)
	fmt.Println()

	// hence setting the name on the Person pointed to by p1 leaves that Person
	// unchanged and instead returns a new Person with the name set (notice the
	// code generated "setter" for Name)
	p2 := p1.SetName("Paul")

	fmt.Printf("p1: %v\n", p1)
	fmt.Printf("p2: %v\n", p2)
	fmt.Println()

	p3 := p1.WithMutable(func(p *example.Person) {
		// WithMutable is used whre multiple mutations are required... but again
		// the mutations are applied to a copy (the p passed to this function is
		// mutable copy of p1, which when WithMutable returns is marked
		// immutable)
		p.SetName("Monty Python")
		p.SetAge(42)
	})

	fmt.Printf("p1: %v\n", p1)
	fmt.Printf("p2: %v\n", p2)
	fmt.Printf("p3: %v\n", p3)
}

gives the output:

p1: Person{ Name: "", Age: 0}

p1: Person{ Name: "", Age: 0}
p2: Person{ Name: "Paul", Age: 0}

p1: Person{ Name: "", Age: 0}
p2: Person{ Name: "Paul", Age: 0}
p3: Person{ Name: "Monty Python", Age: 42}

Once initialized, a *Person value cannot be modified; any attempt to do so returns a copy that has been modified (let me repeat the aforementioned caveat of a very inefficient implementation at this stage). Same goes for immutable slices and maps generated via immutableGen.

Correct usage of methods on these generated immutable types is enforced by immutableVet)

The interface to such immutable data structures is, as you can see, quite bloated. But again, this is just an experiment based on the existing Go language.

So I think that fits with the definition of immutable data structures?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Jul 20, 2017

I see, so you have a set of methods that return a copy that can be modified. OK, I agree that this does seem like an immutable type. (I think I would just provide a single method that returns a copy.) Thanks for the explanation.

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Jul 20, 2017

I see, so you have a set of methods that return a copy that can be modified

Almost; the methods return a modified copy, but the returned value is itself immutable. So any attempt to "modify" the returned value (via a "setter") results in another copy.

Hence (p2 as before):

p4 := p2.SetAge(42)

fmt.Printf("p1: %v\n", p1)
fmt.Printf("p2: %v\n", p2)
fmt.Printf("p4: %v\n", p4)

gives:

p1: Person{ Name: "", Age: 0}
p2: Person{ Name: "Paul", Age: 0}
p4: Person{ Name: "Paul", Age: 42}

(WithMutable and AsMutable/AsImmutable allow for working with a mutable receiver where multiple changes are required)

@henryas

This comment has been minimized.

Copy link

henryas commented Jul 21, 2017

If you turn Person into objects, what you are trying to do is already achievable without needing code generation. See https://play.golang.org/p/TUaq76pB98

I do think that by wrapping them into objects, we can take advantage of Go's interface to solve this problem. The same goes with array, slices, and maps. However, the problem with the built-in array, slices, and maps is that they have no behavior (no methods) and thus one cannot directly apply interface to them. You need to create new objects by wrapping those built-in collections. On one hand, you can say it is a good practice to wrap a collection because you are giving them a domain identity. On the other hand, it involves additional boilerplate code. Without generics, you end up creating many different collection objects for each different type.

I believe we should look into using Go's interface instead of looking for new magic keywords.

@myitcv

This comment has been minimized.

Copy link
Member

myitcv commented Jul 21, 2017

@henryas thanks, I think I get what @ianlancetaylor was referring to now by the "single method that returns a copy". So we're using the same definition for immutability, it's just different interfaces/approaches.

Of course there is nothing that requires code generation; we introduced it because it was easy and it saved writing tonnes of boilerplate as you say. Taking your example, code generation would allow you to "explode":

type personImpl struct {
	name string
	age  int
}

into the full implementation you have in your linked example.

Writing code to write less code as someone once said 👍

@Azareal

This comment has been minimized.

Copy link

Azareal commented Jul 23, 2017

Something which would be useful in addition to const maps and slices are const keys for maps. In other words, you declare all the keys the map can have and it'll be enforced by the compiler. While it overlaps with structs, it has the benefits of a map. E.g. Calling the field stored in a string without using reflection.

And for full const maps, I don't think any of the keys or contents should be editable.

var blahblah map[const string]string
Constant keys.

var blahblah map[string]string
Normal map.

const blahblah map[string]string
Full constant map.

Perhaps, something like this. This would help eliminate bugs (e.g. calling the wrong key), and to give the compiler more chances to speed things up. For things which can be detected at compile time, it would error and halt compilation, otherwise it would panic.

@hooluupog

This comment has been minimized.

Copy link

hooluupog commented Nov 19, 2017

Just give my two cents,
I'd love to be able to send immutable / read-only data(a very large slice,map,struct,etc) back and forth to goroutines through channel.There are three use cases,

  • Read-only access the data by different goroutines without needing to do a full copy. It's safe and performant to share immutable data.
  • Read-write access the data by different goroutines. Then copy-on-write is inevitable.
  • Send data[1] to only one receiver(Only receiver has the right to modify the data once sent out). The sent data is not copied, just the ownership of the data is transferred to the receiver.So Something like Transferable Objects [2] would be helpful.
    [1]the sent data can be parts of origin data. For example,The sender created a large list and sent subLists(different list segments) to many receivers simultaneously.Each receiver just got the ownership of its own sublist.
    [2]https://developers.google.com/web/updates/2011/12/Transferable-Objects-Lightning-Fast
@henryas

This comment has been minimized.

Copy link

henryas commented Dec 3, 2017

I may have a change of opinion about this proposal since I first wrote it. I would now be very cautious about introducing mutability/immutability (or something similar) into Go. Being able to tell the compiler exactly what you want so that it can work smarter and help you catch errors is always ideal. Mutability/immutability embodies one of those ideals. However, implementation-wise, it introduces a whole lot of other problems.

In order to be able to specify a type as a mutable/immutable, you also need to be able to mark their methods as such. In Go, you may do this by adding an extra keyword to the receivers. You may also need to be able to mark the method signature in an interface as mutable/immutable. Once you have mutability qualifiers all over the place, you have to give extra thought when designing your code because if you are not careful, you may hinder your code future usability due to the unnecessary constraints you place in your codes. Even after some careful consideration, you may still find yourself in an awkward situation due to the mutability constraints. In Rust, they have a number of special pointers to bail you out when everything else fails. One such smart pointer is the Cell/RefCell, which is a way to allow mutability in an immutable object. When I first learned about it, I thought I would never going to need it. It turns out that there are quite a number of occasions when I had to use it. One such example is when I needed to create a mock object. The methods being mocked specify that the receiver must be immutable (because the actual object mustn't change anything), and yet the mock object needs to update itself in order to keep track of the way the mock object is used. Hence, the need for the smart pointers.

I am now of the opinion that Go's current approach, although it is far from perfect, is simpler and less mentally taxing. I would wait until something better comes along.

@leiser1960

This comment has been minimized.

Copy link

leiser1960 commented Dec 3, 2017

@henryas:

I would now be very cautious about introducing mutability/immutability (or something similar) into Go.

Being cautious is always a good idea.
But sending a mutable value over a channel is risky.
Adding immutability is tricky,
but locating race conditions is difficult.
Your proposal is a step in the right direction.

@dnfield

This comment has been minimized.

Copy link

dnfield commented Dec 4, 2017

I think immutability in Go is still a great thing to pursue. I still think we have a few different good ideas in here that would be of varying utility and difficulty to implement.

The use case that drew me to this has to do with unsafe/syscall usage. I would like to be able to memory map a file in Go and pass around slices with confidence that receivers cannot modify them (without going unsafe themselves). This is more or less having an interface that utilizes something like C const, and is really just a compile time constraint. The benefit is that it could allow services to process large amounts of read-only data more efficiently. And this is less about preventing race conditions rather than protecting memory/data integrity - if I'm working with a read-only slice, I know I don't have to worry about data read/write race conditions, and I also know that none of my consumers can mess with that.

I can also see a great case for having runtime immutability, which would almost certainly involve some of the copying mechanisms that have been discussed above. That certainly seems like a bigger issue and something that would require some new features. It's different than the use case I mention above though.

@cznic

This comment has been minimized.

Copy link
Contributor

cznic commented Dec 4, 2017

I would like to be able to memory map a file in Go and pass around slices with confidence that receivers cannot modify them (without going unsafe themselves).

For that you don't need language-level immutability and it can be done today by passing only the syscall.PROT_READ flag and omitting the syscall.PROT_WRITE flag in the call to syscall.Mmap. The backing array of the byte slice viewing the file will be R/O even for crafted unsafe.Pointers.

@dnfield

This comment has been minimized.

Copy link

dnfield commented Dec 4, 2017

@cznic, that's fine for a runtime error, I'd like it to be able to fail at compile time. It's possible to do that in other languages, and it's a great feature when sharing code across packages/development teams/etc.

For instance, I really like what .NET core is trying to do with Span and ReadOnlySpan: https://github.com/dotnet/corefxlab/blob/master/docs/specs/span.md

@jaekwon

This comment has been minimized.

Copy link

jaekwon commented Dec 16, 2017

Here is an extension of Brad's proposal for readonly slices, by way of replies to Russ Cox's evaluation of Brad's proposal. It is part of a set of proposals to improve Golang's security model.

The main linguistic motivation is to be able to write functions that operate equally well on []byte and string. The change delivers on this promise for functions that use strings only as input.

There is another motivation for readonly []byte without making it an alias of string.

Quote from parent proposal:

Slices are used everywhere in Golang, but there's no native language feature to help with safety. It's not always immediately obvious whether it's safe to pass a slice, or whether that slice needs to be copied before being passed in or returned. You need to think about the performance tradeoffs of copying or not, and often that blinds you to the concern of security/mutability.

Here's an example that doesn't need global immutability as in strings, yet benefits from increased security and compile-time type-checking w/ readonly slices.

type Listener func(msg readonly []byte)
func broadcast(msg []byte, listeners []Listener) {
    for listener := range listeners {
        listener(readonly(msg))
    }
}

Without the readonly type modifier, we'd be forced to duplicate the msg byte-slice before sending to listeners, under the proposed security requirement. By adding the readonly/any modifiers, we're just making what would otherwise be "documentation-level" contracts into explicit contracts that the compiler can check for us. It would probably help write programs faster by catching bugs sooner.

Duplicated functions returning subslices of their input cannot be merged. For example,
func bytes.TrimSpace(s []byte) []byte
func strings.TrimSpace(s string) string
cannot be replaced by
func strings.TrimSpace(s readonly []byte) readonly []byte
because often the caller really does need a []byte, for future mutation, or string, for use with string operations like ==, <, >, or +. A readonly []byte is not good enough, so the duplication must be preserved.

In this proposal, readonly []byte isn't a string, so this doesn't apply. Some other critiques and the section Immutability and memory allocation also no longer apply.

there are a few common methods in the Go tree that involve returning slices. At the least:
Bytes() []byte
Peek(n int) []byte
Different implementations might choose to make the result readonly or not. Packages testing for the interface will only find it if the readonly bits match.
x, ok := x.(interface{ Peek(int) []byte })
will not find the readonly form, and vice versa. To the extent that such bifurcation happens, the libraries become less powerful.

Perhaps by introducing any?

type Peeker interface {
    Peek(int) any []byte
}

type roPeeker struct {}
func (_ roPeeker) Peek(n int) readonly []byte {...}

type rwPeeker struct {}
func (_ rwPeeker) Peek(n int) []byte {...}

xro := roPeeker{}
_, ok := xro.(interface{ Peek(int) []byte })          // not ok
_, ok  = xro.(interface{ Peek(int) readonly []byte }) // ok
_, ok  = xro.(interface{ Peek(int) any []byte })      // ok

xrw := rwPeeker{}
_, ok  = xrw.(interface{ Peek(int) []byte })          // ok
_, ok  = xrw.(interface{ Peek(int) readonly []byte }) // ok
_, ok  = xrw.(interface{ Peek(int) any []byte })      // ok

This should also work in unsurprising ways when there are multiple slices in the method signature. Here's a complete list of match rules for arguments:

need \ provide x.(func([]byte)) x.(func(any[]byte)) x.(func(readonly[]byte))
func([]byte) ok NOT ok NOT ok
func(any []byte) ok ok ok
func(readonly []byte) ok ok ok

And return values:

provide \ need x.(func()[]byte) x.(func()any[]byte) x.(func()readonly[]byte)
func() []byte ok ok ok
func() any []byte NOT ok ok ok
func() readonly []byte NOT ok ok ok

When a writeable slice is need, it must be provided.

If we introduce read-only slices, it makes sense to change IntsAreSorted to take a readonly []int. However, the readonly []int cannot be converted to the non-read-only IntSlice. Instead, a new ReadOnlyIntSlice must be defined, adding duplication, and either ReadOnlyIntSlice.Swap will need to panic, or IsSorted will need to be changed to take a new ReadOnlyInterface so that ReadOnlyIntSlice need not implement Swap, adding either a magic panic or more duplication.

I prefer the duplication approach which in the example introduces 2 new lines:

type ReadInterface interface {
	Less(i, j int) bool
	Len() int
}

type SortInterface interface {
	Swap(i, j int)
}

func Sort(data SwapInterface) bool {
	… code using Less, Len, and Swap …
}

func IsSorted(data ReadInterface) bool {
	… code using only Less and Len …
}

type ReadIntSlice readonly []int
func (x ReadIntSlice) Less(i, j int) bool { return x[i] < x[j] }
func (x ReadIntSlice) Len() int { return len(x) }

type SortIntSlice []int
func (x SortIntSlice) Less(i, j int) bool { return ReadIntSlice(x).Less(i, j) }
func (x SortIntSlice) Len() int { return ReadIntSlice(x).Len()) }
func (x SortIntSlice) Swap(i, j int) { x[i], x[j] = x[j], x[i] }

func Ints(a []int) { // invoked as sort.Ints
	Sort(SortIntSlice(a))
}

func IntsAreSorted(a []int) bool {
	return IsSorted(ReadIntSlice(a))
}

We can add sugar and do:

type SortIntSlice +ReadIntSlice
func (x SortIntSlice) Swap(i, j int) { x[i], x[j] = x[j], x[i] }

Going back to the original problem statement,

Slices are used everywhere in Golang, but there's no native language feature to help with safety. It's not always immediately obvious whether it's safe to pass a slice, or whether that slice needs to be copied before being passed in or returned. You need to think about the performance tradeoffs of copying or not, and often that blinds you to the concern of security/mutability.

With this proposal, if you receive a []byte, then it's mutable and the implicit contract is that the slice is OK to modify, or that there is a correct way to modify it. If you receive an any []byte, you'd need to do a runtime type-check to modify it, or you'd need to copy it, or use it as a readonly []byte. If you want to prevent others from modifying your slice, you'd send a readonly []byte. If you don't know, then you can always send an any []byte, but only when the receiver expects an any []byte or a readonly []byte.

@jaekwon

This comment has been minimized.

Copy link

jaekwon commented Dec 30, 2017

Syntax-wise, it seems like a 1-character prefix to the []T might be better to legibility.

_[]T for read-only, e.g. var bz = _[]byte
?[]T for any
[]T for writeable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment