proposal: Go 2: Index on any

[iFrozenPhoenix]

Author background

• Would you consider yourself a novice, intermediate, or experienced Go programmer?
• intermediate
• What other languages do you have experience with?
• Typescript, Python, (Java in the past)

Related proposals

• Has this idea, or one like it, been proposed before?
  I have not found a related or similar proposal
• Does this affect error handling?
  No
• Is this about generics?
  No

Proposal

• What is the proposed change?

• Allow an index on any / interface{} to be capable to access map[string]any maps on deeper levels more easily. This proposal addresses situations where the map is defined in previous, i.e. where type inference is possible as well as where it is not possible, e.g. unmarshalling json values.

• Who does this proposal help, and why?
  Anyone who works with undefined data structures in golang. The proposed change helps to access complex data structures more easily.

• Please describe as precisely as possible the change to the language.
  Allowing to access an any value with an index.
  If the key is indexable the value is returned. If the key is not indexable an error is returned that the key is not indexable.
  Currently one can do a type assertion on an any type. If the type is not the asserted one it results in a runtime error.
  The resulting behavior if the any key is not indexable would stay the same, i.e. a runtime error.

• What would change in the language spec?

• Allowing to index any

• Please also describe the change informally, as in a class teaching Go.

• Is this change backward compatible?

• Yes. Normal type assertion is not affected

  • Breaking the Go 1 compatibility guarantee is a large cost and requires a large benefit.
    Show example code before and after the change.
  • Before

m := map[string}any{
  "name": "Marvin",
  "attributes": map[string]any {
    "place_of_birth": "Mardrid",
    "age": 20,
  },
}

age := m["attributes"].(map[string]any)["age"]
placeOfBirth := m["attributes"].(map[string]any)["place_of_birth"]

• After

m := map[string}any{
  "name": "Marvin",
  "attributes": map[string]any {
    "place_of_birth": "Mardrid",
    "age": 20,
  },
}

age := m["attributes"]["age"]
placeOfBirth := m["attributes"]["place_of_birth"]

• Orthogonality: how does this change interact or overlap with existing features?
  It would extend the current capabilities of accessing any values with type assertion with the capability to access these values without a prior type assertion. This would enhance readability and the access to nested data structures more easily.
• Is the goal of this change a performance improvement?
  No

Costs

• Would this change make Go easier or harder to learn, and why?
  It would make it easier because it reduces the code and is more intuitive. Many people have at least basic knowledge in scripting languages like JS, TS or Python. The proposed solution would help them to use the benefits of golang with the simplicity that they already know.
• What is the cost of this proposal? (Every language change has a cost).
• How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?
  2 (Vet and GOPLS)
• What is the compile time cost?
  Probably none, max minimal
• What is the run time cost?
  None
• Can you describe a possible implementation?
  No
• Do you have a prototype? (This is not required.)
  No

[apparentlymart]

I'd like to check that I'm understanding correctly what you're proposing, by restating it in some different words.

The new behavior is that, if the index operator [...] is applied to a value of interface type:

• Find the interface's dynamic type.
• If the dynamic type is not a type that would normally support indexing, then panic.
• Otherwise, apply the index operator to the underlying value and return its result.

Is that a correct understanding?

If so, I have some follow-up questions:

1. Is this need specific to the index operator?

   Could the same argument of convenience suggest that it should also be possible to apply the field access operator to an interface value in case its dynamic type is a struct type, or to apply the addition operator if the dynamic type is something that would support that operator? If not, what makes indexing special?

2. Can you identify anything else in Go that you consider comparable to this?

   I see this as a sort of "automatic type assertion" behavior, where the compiler behaves approximately as if a type assertion had been present, and I can't think of other situations where Go allows that sort of implicit behavior, but perhaps I'm forgetting something.

My initial instinct was that the type assertion is helpful to make it clear that this operation is fallible if the type isn't what's expected, but it's already possible for indexing to panic dynamically in some other situations and so perhaps that is one plausible argument that indexing is "special" compared to some other operations that can be proven not to panic at compile time. Still, this feels quite "un-Go-like", so unless there's some existing precedent in the language that I'm not remembering this feels like a more substantial design decision.

[seankhliao]

Rather than a language change, a package seems to be a better fit for these kinds of operations, especially if you want things like defaults or error handling

example package: https://pkg.go.dev/github.com/k0sproject/dig

[seankhliao]

somewhat related to navigating unknown structures #42847

[iFrozenPhoenix]

@apparentlymart your understanding is correct.

Concerning your first question I think that this proposal is not suitable or applicable for structs because this would nearly completely allow an untyped behavior. This is not the intention.
But I think maps are something special in this case. If one has a defined data structure he uses structs if one has to deal with unknown or dynamic data structures one uses a map.

Currently I also don't know something similar in the current language. But nevertheless there are attempts and intentions to make the language more easy. E.g. if anonymous structs are defined currently one has to write struct {... and the definition again. At least for struct slices one can now directly define the values. This is also some kind of an indirect behavior.

Concerning your assumption that it makes it more clear that this operation can fail. Yes, at least somehow. But what is the benefit? The result is the same. The error handling should also be the same like for type assertions, i.e. an optional second return that indicates ok or not.

[iFrozenPhoenix]

@seankhliao this is not really related. The proposal you mentioned describes that a nil pointer should be accessible and should not result in a panic. I definitely want that the attempt to access a nil pointer results in a panic. My proposal is only about the type assertion of an any value because the compiler cannot check if the runtime value is indeed what the type assertion implies. This can only be evaluated at runtime. So the result of trying to access a non indexable any value with an index or a type assertion that is incorrect is the same, a runtime error.

[apparentlymart]

Thanks for the extra detail, @iFrozenPhoenix.

The extra part you mentioned about having a two-result form is interesting.

v, ok = m["foo"]

In today's Go, ok is false if there's no "foo" element in the map. If m is a nil map then it behaves as if it has no elements, and so ok is false in that case too. In all other cases, ok will be true.

I think you're suggesting that if m is an interface value then ok being false could also mean that the dynamic value of the interface isn't indexable at all, which seems to overload a single boolean result with two different meanings. With that said, I can see that if someone does want to distinguish those two cases then they would still have the option of using an explicit type assertion.

---

You added another comment while I was thinking about your response to me in which you asserted that nil would still panic under your proposal. That does seem consistent with the idea of it panicking if the underlying value isn't something indexable (there is no underlying value in a nil interface), but would you want the v, ok form to also panic in that case, or should that also cause ok to be set to false?

[iFrozenPhoenix]

@apparentlymart I think that the proposed behavior should be consistent with the current index access behavior, i.e. if the index value is not found and only one return is expected, then panic. If the value and ok is expected then also the solution in this proposal should not panic, regardless if any is indexable or if the key is not found. Yeah, you're right, the case where someone wants to distinguish between any not indexable and index not found can't be handled due to the fact that ok is a bool and can only represent true or false. The only solution for this would be to return an error but I don't would recommend this and stay compatible with the current behavior of an index access and only return true or false.

[iFrozenPhoenix]

@apparentlymart if someone really needs not know if the reason was that any is not indexable one could use reflection to check if the type is indexable (Slice, Map, String) or not.

[apparentlymart]

After some further pondering, I did think of one situation in today's Go that you could potentially argue as similar to this proposal:

It's valid to compare interface values using ==, which will panic at runtime if the underlying dynamic types are not comparable. If you squint you could imagine this as being an implicit type assertion to comparable, although it's not implemented that way both because this behavior of == predates the comparable concept and because it isn't possible to make a a value of type comparable anyway (it's only used as a constraint for type parameters).

I'm not sure I'm very convinced that these two situations are similar enough for this to serve as a precedent, but since I posed the question in the first place I thought I'd at least try to answer it. 😀

[iFrozenPhoenix]

@bcmills any comment why you don't think the language and the ecosystem could benefit of this?

[bcmills]

It's not consistent with Go's general design as a statically-typed programming language: it would add more run-time failure modes that are not obvious to readers of the code.

[atdiar]

That's an interesting proposal.
It tries to make Go more dynamic by providing a new, map access operation to interfaces.
It would be faillible the way type assertions are via the comma, ok idiom.

For more flexible datastructures maybe.

For more rigid structures, it's always possible to define a map of map type with it's own getter and setters but obviously the nesting level is fixed.

Interesting.

[iFrozenPhoenix]

@bcmills Thank you for your answer.

Ok, I haven't expect that. For example if we look at JS or Python, both are largely used, also at Google, and are completely untyped and are capable to be used in large scale projects and result in a reliable runtime behavior.
Golang has components of different paradigms like functional and object orientated programming, but data structure handling is one thing that is quite unpleasant.
In my personal opinion there would be not many changes needed to mike Golang capable to completely replace Python for Data Science. One thing is data handling (Access, definition).
With this proposal one part of the data access is addressed. The other part is the open proposal from 2015 for type inference on composite literals #12854.
More dynamic features Golang does not need. For a good reason on someday TS was invented and mostly replaced JS in Dev. We all have the need for static typing, but now and then we also need to deal with dynamic or unknown data structures.

[atdiar]

@iFrozenPhoenix isn't it possible to define your own function if these are strings? Instead of using the map access operator?

func MustGetFrom(obj map[string]any, labels ...string) any{//...}

func GetFrom(obj map[string]any, labels ...string) (any, error) {//...}

[griesemer]

While JS and Python are used in large systems, being dynamically typed leads to many problems for new team members (readability), for refactoring (much harder without types), and general maintenance. These are reasons why Go's dynamic parts of the language are restricted to interfaces. Changing that would fundamentally change the character of the language in ways that are directly counter the original design philosophy of Go.

As an aside, much of what you would like to do can be implemented in Go (if need be using reflection) and encapsulated for ease of use.

The emoji voting is also not in favor.

Therefore, this is a likely decline. Leaving open for four weeks for final comments.

[iFrozenPhoenix]

Ok. Understood.
But @bcmills @griesemer please do me a favor and look at https://opensearch.org/docs/latest/clients/go/. This is the official implementation of the opensearch API in golang. Especially look how the query for searches is formed... A string literal with plain json.
I know why it is implemented this way. Try to implement it in golang. I tried 3 ways. Classic type structs, nested anonymous structs and maps. The maps were the less horrible way. If you use type structs then you end up with about 10 different structs (If you only implement the basic queries) that each have to be initialized to perform a query. If you use anonymous nested structs, then yeah, I think you know how it looks like to initialize them...
And for the response, yes, There is the same problem, + that you have to deal with unstructured data that you can't even map somehow, because the response is the same for every query, if it is successful, but the documents are the different types / structs / maps. So you also need to use generics.
Therefore i understand why they implemented the query this way. But sincerely, how is the readability or type safety ensuring correctness in such a case. This is just one example. The problem is that the way data structures are defined and accessed is not really ergonomic in golang. I don't talk about giving up type safety, but to make it more natural to use the data. It makes absolutely no sense to define a nested anonymous struct and then write the same thing again to initialize it. It also does not make sense to force a type assertion that potentially can fail instead of allowing to directly access a value.

l

[griesemer]

I may not have been clear enough before, so let me state it more explicitly:

What you want to achieve can be trivially done with a small helper function, at least if the data structure is represented with maps as in your example (playground):

// get implements "dynamic" field access:
// get(m, k1, k2, ... kn) is the same as "m[k1][k2]...[kn]".
func get(data any, keys ...string) any {
	for _, key := range keys {
		if m, _ := data.(map[string]any); m != nil {
			data = m[key]
			if data == nil {
				break
			}
		}
	}
	return data
}

And then you can just write:

age := get(m, "attributes", "age")

which is almost as short as the proposed

age := m["attributes"]["age"]

and just as easy if not easier to read.

A language change is certainly not justified if the same effect can be achieved by writing some code.
I note that more complex data representations using not just maps, but any composite data type incl. structs, arrays, slices, etc. can be accessed "dynamically" this way. The accessor function (get) will obviously have to be a bit smarter and may need to use reflection (e.g., to access struct fields by name), but again, this is straight-forward Go code.

The bar for a language change is very high. At a minimum it must address a problem that would possibly improve a lot of programs (such as error handling), cannot be done without a language change (generics), or would otherwise significantly improve comfort and/or readability. None of these criteria are met here.

This proposal remains a likely decline*.

[griesemer]

I will add that @seankhliao was suggesting essentially the same with a package. If you want specific behavior such as a nil panic, that's obviously trivial to implement as well, with minor changes to the suggested get function.