Implement typed dictionaries

[Repiteo]

This aims to add the ability to bind typed dictionaries to script & GDScript. The implementation takes heavy inspiration from the existing typed array format. The primary difference is the ability to specify type for just a key, leaving the value as typeless much like current dictionaries. Both key and value need their types specified for typed dictionaries, with Variant allowing for a typeless equivalent.

Syntax

var typed_key_value: Dictionary[int, String] = { 1: "first value", 2: "second value", 3: "etc" }
var typed_key: Dictionary[int, Variant] = { 0: "any value", 10: 3.14, 100: null }
var typed_value: Dictionary[Variant, int] = { "any value": 0, 123: 456, null: -1 }

Editor

Issues

All core issues are resolved!
There are a fair number of areas that still require fixes, in addition to whatever lingering bugs/errors there are. In particular:

• No way currently to make a typeless key and a typed value
• Variant text appears as white instead of green in GDSCript if used as the value:
  
• Editor fails to update creation key/value sections if changing type
• Editor incorrectly perceives the initial typed key/value as null
• Editor fails to setup exported dictionaries during runtime
• Editor won't recognize a typed key passed with the incorrect type as invalid in a GDScript function (that is, when using the [] operator); it throws an error if attempting to do so in runtime & does recognize invalid keys when creating a dictionary initially
• Editor will treat any Object variant as a "Resource" regardless of provided type
• Dictionary fails to recognize local script types in curly-bracket initialization during runtime
• Potential memory leak with container_element_type being a list now instead of a single item. (This is inexperience with the engine on my end, dunno if it needs something special to memdelete array entries)
• Implementation of GDScript tests

Closes godotengine/godot-proposals#56

[lufog]

I would prefer:

var my_dictionary: Dictionary[int, Variant]
var my_dictionary: Dictionary[Variant, int]

Just Dictionary[int] a little confusing.

[Repiteo]

@lufog Great point! I adjusted the code to handle exactly that, but it revealed some other pitfalls in the process (mainly the process of transfering container types from one DataType to another) so I'll have to iron that out before showing off a new version

In the meantime I've added documentation descriptions & made it possible to use a constructor to make a typed dictionary in GDScript, much like typed arrays

[Repiteo]

GDScript tests implemented! They're largely lifted from the tests given to typed Arrays so more specialization will be required, but they're a solid starting point to see what's awry off the bat

Namely, typed arrays seem to be instantiating with null keys & values, though they're recognized as typed. I'm not quite sure yet where the incorrect implementation is happening, but that's my current focus as it very well may be the linchpin of the editor errors

[Bromeon]

With typed arrays, we have the variance problem:

var typed: Array[int] = [1, 2, 3, 4]
var as_untyped: Array = typed

as_untyped[0] = "string" # silently fails at runtime

I guess the same problem exists here, but for both keys and values?
How do we deal with that?

[dalexeev]

With typed arrays, we have the variance problem

The exception for Array (aka Array[Variant]) was intentionally added as a convenience to users of untyped GDScript.

var a: Array[Node2D]
var b: Array[Node] = a # Error
var c: Array[Variant] = a # Unsafe, but ok.
var d: Array = a # Unsafe, but ok.

[Bromeon]

The exception for Array (aka Array[Variant]) was intentionally added as a convenience to users of untyped GDScript.

Yes, but it's still a hole in the type system. I understand the desire to pass typed arrays to APIs accepting untyped ones, and for reading, this is completely sound. The covariance violation occurs when writing to an Array that points to a Array[T]. This cannot be checked at parse time, it's a runtime check (and IIRC it's silent do-nothing, no error at the moment).

My question is, is the same covariance conversion ("upcast") planned for dictionaries on each of the generic parameters? That is:

• Dictionary[int, String] -> Dictionary[int, Variant]
• Dictionary[int, String] -> Dictionary[Variant, String]
• Dictionary[int, Variant] -> Dictionary[Variant, Variant] == Dictionary

And if yes, are we fully aware of the pitfalls?

[dalexeev]

This cannot be checked at parse time, it's a runtime check (and IIRC it's silent do-nothing, no error at the moment).

However, this is not directly related to the PR. PRs of this kind tend to garner large numbers of comments that are difficult to navigate. If you'd like to discuss this, I suggest the #gdscript channel on Rocket Chat.

[Bromeon]

Thanks! My point was not to discuss typed arrays, but the covariance problem regarding dictionaries, and what conversions we allow (see above). As such it's quite on-topic I believe.

[Repiteo]

The holes in the type system certainly have been a bit of a headache to navigate, but attempting to overhaul that system as a whole is probably a bit beyond this PR (at least as I initially envisioned it). This is more about setting up the system that currently exists to dictionaries, warts and all. If the current implementation were to somehow inhibit the ability to implement typed dictionaries in the first place, then changes to the system would make sense

On that note, I've figured out why variants were appearing as null & not changing when swapping types, so both those fixes are in place. As it stands, the two biggest remaining issues are:

• The editor treats any Object variant as "Resource", regardless of the provided type (need to figure out how typed arrays handle this)
• A dictionary fails to properly parse provided key/value if there's unorthodox syntax (eg: typed_dictionary_assign_wrong_to_typed.out has the wrong error, thinks an entirely different type is passed on as the value (from reading the key?))

[Jakepc007]

Would it be in the scope (or goal) of this PR to add type information to existing dictionaries? Such as intersect_ray?

var some_func():
  var result := space_state.intersect_ray(query)
  handle_result(result)

func handle_result(result: 🤔):
  if result.collider:
    pass

I would imagine this would have some overlap with #76843 if such is the case. As those properties aren't even guaranteed.

[AThousandShips]

It would break compatibility to do so imo so not guaranteed

[YuriSizov]

Would it be in the scope (or goal) of this PR to add type information to existing dictionaries?

It wouldn't be in scope for this PR, but it can be discussed as a follow-up to it.

[IntangibleMatter]

Would it be in the scope (or goal) of this PR to add type information to existing dictionaries? Such as intersect_ray?

var some_func():
  var result := space_state.intersect_ray(query)
  handle_result(result)

func handle_result(result: 🤔):
  if result.collider:
    pass

I would imagine this would have some overlap with #76843 if such is the case. As those properties aren't even guaranteed.

For Dictionaries with a precise type schema, but with multiple different types, a struct would likely make more sense. godotengine/godot-proposals#7903

[DaloLorn]

Looks like the C# editor is failing to build?

[IntangibleMatter]

Looks like all tests are passing... Can someone review this ASAP? It'd be awesome to have in 4.3, afaik feature freeze hasn't hit yet

[Bromeon]

There are still open variance problems with Array[T] for which no one has a solution, see my response here and problem elaboration here. The same would happen for Dictionary[K, V] but for both key/value.

I'm not sure if it's wise to rush an implementation as long as we don't have a proper strategy for this (unless people agree that this is not an issue worth addressing).

[DaloLorn]

I'd say "not worth addressing". I'm biased, though: I'm positive I'm not going to have trouble with it, and I desperately want to be able to tell the inspector what kind of object I want to put into my dictionaries.

(The proposal this PR is addressing is one of several prospective language upgrades I've been watching with great interest because they eliminate a lot of unnecessary busywork and complexity from my work on Fleet Ops and should allow me to continue development.)

[IntangibleMatter]

I agree with @DaloLorn . This feels like the kind of thing that's just... part of how the language works, and if/when it does get addressed, it'd make more sense to do at the same time as Arrays. There are exponentially more problems solved short-term and long-term than if we wait for every possible issue to be resolved. The covariance thing feels really nitpicky, and not like the kind of think 90% of users will worry about.

IMO it should be pushed as it is, Covariance is its own issue that needs to be resolved in its own discussion. This is just about getting statically typed dictionaries to finally be in the engine

[ajreckof]

Only checked the part that I know best (ie EditorPropertyDictionary)