Mutability qualifier limited to package

[networkimprov]

I hope this helps. I'm sympathetic because you believe passionately in this concept, and have invested a great deal of time documenting it. I've poured considerable time into a major Go2 proposal myself. That said, I can't engage in a deep discussion.

Go won't admit a const qualifier. The designers considered it at length before Go 1.0, and have since been badgered for it repeatedly by users. They always draw the same conclusion. But happily, const is not the solution you truly want, just a compromise for the sake of compatibility.

Escape the compatibility trap by making mut a per-package option, and allowing mutable-aware packages to call APIs or be called by programs which lack mutability awareness. Within a mutable-aware package, unqualified data is immutable. Beyond the package, its unqualified data is unprotected -- i.e. when returned or passed to a legacy package.

It's not the safe world you sought, but it might get airborne. The current draft cannot fly.

[romshark]

First of all, thank you for the kind words and the feedback! I really appreciate it!

Personal Motivation

I think I should have described my personal motivation better. You understood correctly that const is not the solution I aspire! const is a dirty workaround for backward-compatibility reasons. What I strive for is a way of making Go code clearer and remove the ambiguousness that's often followed by misinterpretation, which is often followed by nasty bugs.

I believe this feature should be an integral part of the language because it perfectly fits both the goals of the Google Go team and the language philosophy.

When programming turns into software engineering you use tools to help you find bugs earlier and
you look for ways to make programs as clear as possible so that bugs are less likely. Nearly all of Go's distinctive design decisions were motivated by concerns about software engineering.

- Russ Cox at the GopherConSG, May 2018

I do agree that forcing Go devs to immutability is not an option, but providing an option to write safer code, in my opinion, is necessary because it would make writing open source packages and complex software in Go a lot safer. I do agree, that we need to find a suitable compromise to seamlessly integrate the proposed concept into the regular Go workflow. Immutable types should be given to those who care and remain transparent to those, who don't.

Immutability-aware packages

I'm not sure how you propose to make packages immutability-aware. I mostly do understand your idea in general, but could you please still describe your idea in a little more detail and possibly with code examples?

[romshark]

First of all, thank you for the kind words and the feedback! I really appreciate it!

Personal Motivation

I think I should have described my personal motivation better. You understood correctly that const is not the solution I aspire! const is a dirty workaround for backward-compatibility reasons. What I strive for is a way of making Go code clearer and remove the ambiguousness that's often followed by misinterpretation, which is often followed by nasty bugs.

I believe this feature should be an integral part of the language because it perfectly fits both the goals of the Google Go team and the language philosophy.

When programming turns into software engineering you use tools to help you find bugs earlier and
you look for ways to make programs as clear as possible so that bugs are less likely. Nearly all of Go's distinctive design decisions were motivated by concerns about software engineering.

- Russ Cox at the GopherConSG, May 2018

I do agree that forcing Go devs to immutability is not an option, but providing an option to write safer code, in my opinion, is necessary because it would make writing open source packages and complex software in Go a lot safer. I do agree, that we need to find a suitable compromise to seamlessly integrate the proposed concept into the regular Go workflow. Immutable types should be given to those who care and remain transparent to those, who don't.

Immutability-aware packages

I'm not sure how you propose to make packages immutability-aware. I mostly do understand your idea in general, but could you please still describe your idea in a little more detail and possibly with code examples?

[networkimprov]

There would be a flag in each package file to enable the mut qualifier. I haven't given it much thought, but maybe one of...

package main --immutable /* compiler flag on package line;
                            could be turned off on command line */

//go:immutable           /* the go:generate pattern */

var iData = T{...}                    // immutable within package,
                                      // but mutable outside the package, e.g.
func f() *T {
   err := json.Unmarshal(..., &iData) // modifies iData
   return &iData                      // caller could modify iData
}

I neglected to mention previously that unqualified/immutable data should be protected when passed or returned to another package that's also mutable-aware.

The software-engineering motivation for immutability is perfectly valid, but there's another good rationale... Enabling it makes the language acceptable to a wider audience!

EDIT:
I strongly recommend getting buy-in from the Go gods on this concept before re-working the proposal accordingly.

[networkimprov]

There would be a flag in each package file to enable the mut qualifier. I haven't given it much thought, but maybe one of...

package main --immutable /* compiler flag on package line;
                            could be turned off on command line */

//go:immutable           /* the go:generate pattern */

var iData = T{...}                    // immutable within package,
                                      // but mutable outside the package, e.g.
func f() *T {
   err := json.Unmarshal(..., &iData) // modifies iData
   return &iData                      // caller could modify iData
}

I neglected to mention previously that unqualified/immutable data should be protected when passed or returned to another package that's also mutable-aware.

The software-engineering motivation for immutability is perfectly valid, but there's another good rationale... Enabling it makes the language acceptable to a wider audience!

EDIT:
I strongly recommend getting buy-in from the Go gods on this concept before re-working the proposal accordingly.

[romshark]

Sounds interesting, I'd call it The Go Immutability Experiment that can be enabled by the mentioned compiler flag.

I'll try to think it through. Working with mutability-aware packages from within mutability-unaware packages and vice-versa must be elaborated.

Advantages

1. It doesn't break backward-compability of Go 1.x and could introduce the experimental feature without having to wait until a hypothetical Go 2.x release.
2. No const keyword overloading is ever necessary, mut and immut can be seemlessly integrated into the existing ecosystem.
3. The verbosity of immutable type declaration can be reduced by mutability qualification propagation which was otherwise impossible with const overloading.
4. It doesn't force anyone to use immutable types but rather offers it as an experimental option.
5. If the experiment happens to be canceled - code written with immutable types in mind will continue to work with immutability being disabled.

Disadvantage

• Code from immutability-aware packages would look a little strange compared to regular Go code ¹.

¹ if we make immutability-aware packages mutable by default while providing explicit immutability through immut then the code will mostly look familiar.

Required Changes

• We will most likely have to give it up the concept of "immutability of all types by default" to ensure that the code works with the experiment being disabled. We'll have to return to explicit immtuability through immut qualification (and mut propagation cancelation).

[romshark]

Sounds interesting, I'd call it The Go Immutability Experiment that can be enabled by the mentioned compiler flag.

I'll try to think it through. Working with mutability-aware packages from within mutability-unaware packages and vice-versa must be elaborated.

Advantages

1. It doesn't break backward-compability of Go 1.x and could introduce the experimental feature without having to wait until a hypothetical Go 2.x release.
2. No const keyword overloading is ever necessary, mut and immut can be seemlessly integrated into the existing ecosystem.
3. The verbosity of immutable type declaration can be reduced by mutability qualification propagation which was otherwise impossible with const overloading.
4. It doesn't force anyone to use immutable types but rather offers it as an experimental option.
5. If the experiment happens to be canceled - code written with immutable types in mind will continue to work with immutability being disabled.

Disadvantage

• Code from immutability-aware packages would look a little strange compared to regular Go code ¹.

¹ if we make immutability-aware packages mutable by default while providing explicit immutability through immut then the code will mostly look familiar.

Required Changes

• We will most likely have to give it up the concept of "immutability of all types by default" to ensure that the code works with the experiment being disabled. We'll have to return to explicit immtuability through immut qualification (and mut propagation cancelation).

[networkimprov]

All language changes are Go2 features, even if backward-compatible. If you're dependent on the Go team to implement this (vs providing patches yourself) it would come after error handling and generics in the Go2 queue, and wouldn't land for 2-3 years.

We want mutable-aware code to be immutable by default. Looking different is an advantage. In C if you forget a const qualifier, it compiles and a later patch can wrongly modify the data. In Go-mut if you forget a mut qualifier, it doesn't compile!

Disabling the experimental feature means making mut a no-op. Abandoning the feature means a quick go-fmt pass to delete mut keywords.

I would expect this to be adopted by projects building apps and not used much for libraries. The stdlib might never adopt it.

[networkimprov]

All language changes are Go2 features, even if backward-compatible. If you're dependent on the Go team to implement this (vs providing patches yourself) it would come after error handling and generics in the Go2 queue, and wouldn't land for 2-3 years.

We want mutable-aware code to be immutable by default. Looking different is an advantage. In C if you forget a const qualifier, it compiles and a later patch can wrongly modify the data. In Go-mut if you forget a mut qualifier, it doesn't compile!

Disabling the experimental feature means making mut a no-op. Abandoning the feature means a quick go-fmt pass to delete mut keywords.

I would expect this to be adopted by projects building apps and not used much for libraries. The stdlib might never adopt it.

[romshark]

@networkimprov

We want mutable-aware code to be immutable by default. Looking different is an advantage. In C if you forget a const qualifier, it compiles and a later patch can wrongly modify the data. In Go-mut if you forget a mut qualifier, it doesn't compile!

Yeah, I've described it in 3.1. Benefits and I personally would also prefer immutability by default.

But many seem to hate the idea of having Go code where everything's immutable by default, people want to be able to read both kinds of packages (aware and unaware ones) similarly and I totally understand that. If I would have to work with different packages where some are immutable by default and some are not I might get very confused as well! That's why I proposed it for a backward-incompatible Go 2 only, where any Go 1 code would have to be rewritten anyway (but we can't expect it any time soon)

I would expect this to be adopted by projects building apps and not used much for libraries. The stdlib might never adopt it.

Even though it's unrealistic to expect every non-standard library out there to immediately jump on the immutability train (which would be desirable though) - the standard library will need to be made immutability-aware for the reasons described in 2.12. Standard Library. I'm not sure whether we can fix this problem without making the std. lib. aware of immutable types, but in this issue we discuss just that: making unaware and aware packages compatible and interoperable.

P.S.

All language changes are Go2 features

Just to make sure: when I'm talking about Go2 I'm usually referring to Go 2.0.0, the technical semantic version, which is to be considered backward-incompatible with Go 1.x. The "Go2" they usually mean is probably just a marketing name for Go 1.13 and beyond.

[romshark]

@networkimprov

We want mutable-aware code to be immutable by default. Looking different is an advantage. In C if you forget a const qualifier, it compiles and a later patch can wrongly modify the data. In Go-mut if you forget a mut qualifier, it doesn't compile!

Yeah, I've described it in 3.1. Benefits and I personally would also prefer immutability by default.

But many seem to hate the idea of having Go code where everything's immutable by default, people want to be able to read both kinds of packages (aware and unaware ones) similarly and I totally understand that. If I would have to work with different packages where some are immutable by default and some are not I might get very confused as well! That's why I proposed it for a backward-incompatible Go 2 only, where any Go 1 code would have to be rewritten anyway (but we can't expect it any time soon)

I would expect this to be adopted by projects building apps and not used much for libraries. The stdlib might never adopt it.

Even though it's unrealistic to expect every non-standard library out there to immediately jump on the immutability train (which would be desirable though) - the standard library will need to be made immutability-aware for the reasons described in 2.12. Standard Library. I'm not sure whether we can fix this problem without making the std. lib. aware of immutable types, but in this issue we discuss just that: making unaware and aware packages compatible and interoperable.

P.S.

All language changes are Go2 features

Just to make sure: when I'm talking about Go2 I'm usually referring to Go 2.0.0, the technical semantic version, which is to be considered backward-incompatible with Go 1.x. The "Go2" they usually mean is probably just a marketing name for Go 1.13 and beyond.

[networkimprov]

A program with package p --immutable which needs stdlib functions that modify their arguments should call a mutable-aware shim package. Leave stdlib alone.

[networkimprov]

A program with package p --immutable which needs stdlib functions that modify their arguments should call a mutable-aware shim package. Leave stdlib alone.

[networkimprov]

Go2 changes under discussion will break some Go1 programs by introducing new keywords. There will never be a GoX that breaks all programs.

[networkimprov]

Go2 changes under discussion will break some Go1 programs by introducing new keywords. There will never be a GoX that breaks all programs.

[romshark]

A program with package p --immutable which needs stdlib functions that modify their arguments should call a mutable-aware shim package. Leave stdlib alone.

Even if we were to write shim-package for the standard library (which is terrifying enough if you think about this) we still couldn't solve the compatibility problem described in section 2.12.

If we take strings.Join(a []string, sep string) string for example then we can't use it like this:

import (
  "strings"
)

var Separator immut string = ","

func main() {
  var immutSlice immut []string = []string {"hello", "world"}
  strings.Join(immutSlice, Separator) // Compile-time error
}

Because immutSlice is an immutable immut []string (const-analogue: const [] const string) and cannot be cast to a mutable []string, just as Separator cannot be cast from immut string to string.

I guess we have no other option but to allow casting immutable to mutable types between mutability-aware and mutability-unaware packages.

Go2 changes under discussion will break some Go1 programs by introducing new keywords. There will never be a GoX that breaks all programs.

Is there a reliable source stating that the Go team is considering to add new keywords to Go? (not Go 2.x)?

[romshark]

A program with package p --immutable which needs stdlib functions that modify their arguments should call a mutable-aware shim package. Leave stdlib alone.

Even if we were to write shim-package for the standard library (which is terrifying enough if you think about this) we still couldn't solve the compatibility problem described in section 2.12.

If we take strings.Join(a []string, sep string) string for example then we can't use it like this:

import (
  "strings"
)

var Separator immut string = ","

func main() {
  var immutSlice immut []string = []string {"hello", "world"}
  strings.Join(immutSlice, Separator) // Compile-time error
}

Because immutSlice is an immutable immut []string (const-analogue: const [] const string) and cannot be cast to a mutable []string, just as Separator cannot be cast from immut string to string.

I guess we have no other option but to allow casting immutable to mutable types between mutability-aware and mutability-unaware packages.

Go2 changes under discussion will break some Go1 programs by introducing new keywords. There will never be a GoX that breaks all programs.

Is there a reliable source stating that the Go team is considering to add new keywords to Go? (not Go 2.x)?

[networkimprov]

we have no other option but to allow casting immutable to mutable types ...

See example code in #23 (comment)

New keywords are proposed for error handling; otherwise it's backwards compatible:
https://go.googlesource.com/proposal/+/master/design/go2draft.md

Note: type string is immutable; immut string looks odd.

[networkimprov]

we have no other option but to allow casting immutable to mutable types ...

See example code in #23 (comment)

New keywords are proposed for error handling; otherwise it's backwards compatible:
https://go.googlesource.com/proposal/+/master/design/go2draft.md

Note: type string is immutable; immut string looks odd.

[romshark]

New keywords are proposed for error handling; otherwise it's backwards compatible:
https://go.googlesource.com/proposal/+/master/design/go2draft.md

I see. The Go team doesn't bother adding new keywords like check (which are, most likely, abundant in the current Go 1.x code bases), so why even bother about immut and mut which should be occurring relatively rarely compared to check?

But well, keywords are not really the problem, we'll still need to differentiate between immut-aware packages and regular packages to deal with legacy packages that don't care about immutable types.

Immut-aware packages should really only be used with other immut-aware packages whenever possible. Immutable- to mutable type conversions are bad and wrong, we don't want them. We'd only tolerate them for compatibility reasons to have an option to either slowly adapt the standard library and ecosystem package by package or not rewrite them at all (which would be a shame because we want both the standard library and the ecosystem to be safe and dependable, thus they should describe their APIs clearly)

package example

//go:immutable

import (
  "strings"
  "github.com/immut/unaware"
)

var ImmutGlobal immut []string = []string{
  "b1",
  "a2",
}

func ExampleFunc() {
  ImmutGlobal[0] = "third" // Compile-time error

  // This is okay, strings.Join usually doesn't mutate "a"
  strings.Join(ImmutGlobal, ",") // Implicit cast from immutable to mutable (cross-package)
  log.Print(ImmutGlobal) // ["b1", "a2"]

  // But this is not! this voids the entire concept of immutability in this package!
  unaware.SortStrings(ImmutGlobal) // Implicit cast from immutable to mutable (cross-package)

  log.Print(ImmutGlobal) // ["a2", "b1"]
}

If we mix unaware and aware packages too much we'll have the opposite of safety, in fact, we'll have the illusion of safety.

Note: type string is immutable; immut string looks odd.

The byte array beneath the string is immutable, but the string itself is reassignable, thus immut string is necessary to prevent Separator from being reassigned to another string. See here for more details.

[romshark]

New keywords are proposed for error handling; otherwise it's backwards compatible:
https://go.googlesource.com/proposal/+/master/design/go2draft.md

I see. The Go team doesn't bother adding new keywords like check (which are, most likely, abundant in the current Go 1.x code bases), so why even bother about immut and mut which should be occurring relatively rarely compared to check?

But well, keywords are not really the problem, we'll still need to differentiate between immut-aware packages and regular packages to deal with legacy packages that don't care about immutable types.

Immut-aware packages should really only be used with other immut-aware packages whenever possible. Immutable- to mutable type conversions are bad and wrong, we don't want them. We'd only tolerate them for compatibility reasons to have an option to either slowly adapt the standard library and ecosystem package by package or not rewrite them at all (which would be a shame because we want both the standard library and the ecosystem to be safe and dependable, thus they should describe their APIs clearly)

package example

//go:immutable

import (
  "strings"
  "github.com/immut/unaware"
)

var ImmutGlobal immut []string = []string{
  "b1",
  "a2",
}

func ExampleFunc() {
  ImmutGlobal[0] = "third" // Compile-time error

  // This is okay, strings.Join usually doesn't mutate "a"
  strings.Join(ImmutGlobal, ",") // Implicit cast from immutable to mutable (cross-package)
  log.Print(ImmutGlobal) // ["b1", "a2"]

  // But this is not! this voids the entire concept of immutability in this package!
  unaware.SortStrings(ImmutGlobal) // Implicit cast from immutable to mutable (cross-package)

  log.Print(ImmutGlobal) // ["a2", "b1"]
}

If we mix unaware and aware packages too much we'll have the opposite of safety, in fact, we'll have the illusion of safety.

Note: type string is immutable; immut string looks odd.

The byte array beneath the string is immutable, but the string itself is reassignable, thus immut string is necessary to prevent Separator from being reassigned to another string. See here for more details.

[networkimprov]

It's up to the //go:immutable user to shim any unaware packages he needs. A stdlib shim would be provided. Shim authors only have to write those functions that need mut arguments or immut returns; the rest can be generated via go/doc or similar.

As immutability is only enabled for specific packages, try proposing by-default first, and fall back to by-definition if it doesn't fly.

The Go gods may come around to this slowly, i.e. years. They rejected many suggestions to simplify error handling before drafting check/handle. (Which ironically has poor reviews; see the feedback wiki.)

[networkimprov]

It's up to the //go:immutable user to shim any unaware packages he needs. A stdlib shim would be provided. Shim authors only have to write those functions that need mut arguments or immut returns; the rest can be generated via go/doc or similar.

As immutability is only enabled for specific packages, try proposing by-default first, and fall back to by-definition if it doesn't fly.

The Go gods may come around to this slowly, i.e. years. They rejected many suggestions to simplify error handling before drafting check/handle. (Which ironically has poor reviews; see the feedback wiki.)

[romshark]

There would be a flag in each package file to enable the mut qualifier. I haven't given it much thought, but maybe one of...

package main --immutable /* compiler flag on package line;
                            could be turned off on command line */

//go:immutable           /* the go:generate pattern */

Is there no other way than having this flag?

I'm worried about it because this flag would actually affect the entire package including all its files, but it wouldn't be obvious that a package uses immutability if this flag is defined in only one of the files.

Will we have to force the developer to add the flag in all files which won't otherwise compile? (Would be pretty annoying)

Making the package name special like mypack.immut or mypack_immut (because we already do stuff like this in case of _test.go files) is probably not the best solution (at all) but yet another option. Go Vet complains about packages with underscore names anyway, so this could be an option, but a rather bad one. A developer who's not yet familiar with immutable packages could accidentally remove the postfix making the code very unsafe (because it was written with everything being immutable in mind), which is less likely with in-code flags.

also package mypack --immutable looks kinda odd. why not something like package arguments?

package mypack (immutable)

[romshark]

There would be a flag in each package file to enable the mut qualifier. I haven't given it much thought, but maybe one of...

package main --immutable /* compiler flag on package line;
                            could be turned off on command line */

//go:immutable           /* the go:generate pattern */

Is there no other way than having this flag?

I'm worried about it because this flag would actually affect the entire package including all its files, but it wouldn't be obvious that a package uses immutability if this flag is defined in only one of the files.

Will we have to force the developer to add the flag in all files which won't otherwise compile? (Would be pretty annoying)

Making the package name special like mypack.immut or mypack_immut (because we already do stuff like this in case of _test.go files) is probably not the best solution (at all) but yet another option. Go Vet complains about packages with underscore names anyway, so this could be an option, but a rather bad one. A developer who's not yet familiar with immutable packages could accidentally remove the postfix making the code very unsafe (because it was written with everything being immutable in mind), which is less likely with in-code flags.

also package mypack --immutable looks kinda odd. why not something like package arguments?

package mypack (immutable)

[networkimprov]

Flag in all files, yes. We need package and import stmts in all files; I don't think it's a burden.

//go:immutable is probably best. I doubt the Go gods will smile on a package-line argument.

[networkimprov]

Flag in all files, yes. We need package and import stmts in all files; I don't think it's a burden.

//go:immutable is probably best. I doubt the Go gods will smile on a package-line argument.