Skip to content

proposal: maps: new package to provide generic map functions (discussion) #47330

proposal: maps: new package to provide generic map functions (discussion) #47330
Jul 21, 2021 · 22 comments · 100 replies

This is a discussion that led to the proposal #47649.

This proposal is for use with #43651. We propose defining a new package, maps, that will provide functions that may be used with maps of any type. If this proposal is accepted, the new package will be included with the first release of Go that implements #43651 (we currently expect that that will be Go 1.18).

This description below is focused on the API, not the implementation. In general the implementation will be straightforward.

See also the slices proposal at #45955 (discussion at #47203).

// Package maps defines various functions useful with maps of any type.
package maps

// Keys returns the keys of the map m.
// The keys will be an indeterminate order.
func Keys[M constraints.Map[K, V], K comparable, V any](m M) []K

// Values returns the values of the map m.
// The values will be in an indeterminate order.
func Values[M constraints.Map[K, V], K comparable, V any](m M) []V

// Equal reports whether two maps contain the same key/value pairs.
// Values are compared using ==.
func Equal[M1, M2 constraints.Map[K, V], K, V comparable](m1 M1, m2 M2) bool

// EqualFunc is like Equal, but compares values using cmp.
// Keys are still compared with ==.
func EqualFunc[M1 constraints.Map[K, V1], M2 constraints.Map[K, V2], K comparable, V1, V2 any](m1 M1, m2 M2, cmp func(V1, V2) bool) bool

// Clear removes all entries from m, leaving it empty.
func Clear[M constraints.Map[K, V], K comparable, V any](m M)

// Clone returns a copy of m.  This is a shallow clone:
// the new keys and values are set using ordinary assignment.
func Clone[M constraints.Map[K, V], K comparable, V any](m M) M

// Copy copies all key/value pairs in src adding them to dst.
// When a key in src is already present in dst,
// the value in dst will be overwritten by the value associated
// with the key in src.
func Copy[M constraints.Map[K, V], K comparable, V any](dst, src M)

// DeleteFunc deletes any key/value pairs from m for which del returns true.
func DeleteFunc[M constraints.Map[K, V], K comparable, V any](m M, del func(K, V) bool)

Replies

22 comments
·
100 replies

[ Resolved -- changed to DeleteIf ]

Filter seems out of place since it was removed from the proposed slices package but is much easier to implement for maps.

I could see a func Remove[K comparable, V any](m map[K]V, keys ...K) for bulk deletions similar to the proposed Set package.

14 replies
@rsc

One of the awkward things about Filter for slices is the allocation story.
It seems much less awkward for maps.Filter to be defined to edit in place,
since that's just what you do with maps, far more often than with slices.

@Cyberax

Many users familiar with other languages would expect methods like Filter to create a copy.

A better name for a mutating operation would be RetainIf.

@fzipp

Keep?

[ Resolved -- these are just maps, we don't need to define performance beyond that ]

Same performance note as in sets: #47331 (comment)

0 replies

[ Resolved -- changed to Copy ]

I have a mild preference for renaming Add to Copy. The latter gives a clear analog to copy and io.Copy indicating the order of arguments, which is less clear for Add.

21 replies
@cespare

To me, "copy" suggests replacement more than addition. In the abstract, I think I'd prefer the name Merge.

However, I think we should also consider what happens with container/set. Right now, that proposal has us using the name AddSet for the corresponding operation in that package (and Union for the non-destructive function version), so I think that going in a new direction in this package would be a mistake. Therefore, if we keep those names for container/set, I think this one should probably stay as Add (or else some variant: AddMap, AddAll, ...)

@deanveloper

Adding onto this - I think there isn't really an analog to copy here. In my eyes, copy works similarly to an assignment (assign right value to left). However in this case, I feel that it is more analogous to a method call (similar to myMap.Add(otherMap)).

I don't like the plain name Add though since I feel that it sounds singular. I definitely like AddMap/AddAll

@brunnre8

what about Update()? After all that's what it does

[ Resolved - Filter changed to DeleteIf ]

Other language have .map(), .reduce(), .filter() and often this can seem confusing for newbie when learning in functional paradigm.

I have a preference to choose Collect, Pick or Drop over Filter.

2 replies
@bcmills

Prune, maybe?

@bcmills

Actually, I think this thread substantially overlaps with #47330 (comment).

I can see the benefit to Keys and Values, but I wonder if we're going to have support for more general iterators in the future.

Thinking ahead to that, should we name these functions KeySlice and ValueSlice instead, so that we can retains the nicer Keys and Values names for iterators?

9 replies
@bcmills

(Compare to the awkward Python migration from iterkeys to keys, because they initially used the name keys for the method that makes an O(N) copy: https://www.python.org/dev/peps/pep-0469/.)

@DeedleFake

I don't think that it would be too weird to add KeysIter() and ValuesIter() methods later instead. I do think that a discussion should be started about an iter package or something sooner rather than later, though, as it seems that the lack of that design has been coming up through most of these container-type design discussions.

@colin-sitehost

I like the idea, but think that a real iterator system will shake how people write code too much to meaningfully try to fit for where we think things are going. unless we resist it, I expect things to drift in a rusty direction. I would also think that iter.From(maps.Keys(m)) is not the end of the world, as that is what all existing slice code will need to do anyway.

Would you mind adding an example of what it would look like to use it?

1 reply
@ianlancetaylor

This comment didn't get threaded and I no longer remember what it referred to. Do you remember?

[ Resolved -- leave for later ]

Along with the container/set discussion, an Any/Peek or Pop function would be nice. As said in the other discussion, I often do recursive algorithms with a base case of len(m)==1. Any/Peek would be best but a Pop would suffice too. Currently the only way to get the sole element of a map is to range, which is pretty inconvenient

0 replies

[ Resolved -- leave for later ]

How about an Entries() method that returns a slice of some pair type of the keys and values? That pair type could be either something new in this package, such as just type Entry[K comparable, V any] struct { Key K; Value V }, or it could be some more general Pair type implemented somewhere else.

If that gets added, it might also make sense to add func Of[K comparable, V any](v ...Entry[K, V]) map[K]V for parity with both that and set.Of(). That seems less useful to me with a maps.Entry type rather than a generalized pair type, though, as it would specifically require a slice of Entrys. It could still be useful paired with some kind of iterator collection function, though.

2 replies
@colin-sitehost

While I agree, and I think a compelling case can be made for only adding pair [recte two tuple], it is not special and the need for a n tuples will begin to develop with generics. I do not want to block anything, but this is probably a good time to reconsider #32941, since imo this reads better:

func Entries[K comparable, V any](m map[K]V) [](K, V)

func Of[K comparable, V any](v ...(K, V)) map[K]V

Though, if we do not; I think the following convention is fairly readable:

func Entries[K comparable, V any](m map[K]V) []struct{K; V}

func Of[K comparable, V any](v ...struct{K; V}) map[K]V
@rsc

Generally speaking, we are trying to limit the initial API to "clearly should be in this package", not just "probably okay".
This API strikes me as more the latter than the former.

[ Resolved -- the existence of modules doesn't mean that the standard library is not needed ]

Now as Go supports modules I wonder why do you want to put these generic slices and maps packages into standard library. Why not putting it to golang.org/x/...?

1 reply
@rsc

Because they will be used by essentially all Go programs. They are core to Go.

[ Resolved -- no ]

Would this also work with sync.Map?

3 replies
@Merovius

No. There is no way to write a generic function that works both on sync.Map and map[K]V.

@hherman1

I wish we could figure out a way to make interchanging user built hash maps and std maps smooth before adding these utils, else we’ll probably see the same code rewritten many times for these things. I think we will probably end up with a GenericMap if we’re not careful that wraps built in map awkwardly just to fit in with user defined code that wants to be impl agnostic

@Merovius

@hherman1 I think the way this will go at first is that if your package takes an interface for a map-like datastructure, you also provide a wrapper-type type Map[K comparable, V any] map[K]V which implements that interface using a builtin map. Users could then call your code as somepkg.F(somepkg.Map(m)). Not super ergonomic, but IMO okay. If we ever add an interface for a map-like datastructure in the stdlib, we can then also put such a wrapper-type alongside it.

FWIW for sync.Map specifically, I don't think the functions that are proposed here even make a lot of sense. Generally, sync.Map can be modified concurrently, so using operations that consider "the entire map" don't make a lot of sense for it.

Retracted


I would like to propose adding

// Range returns a range iterator over m.
//
// Call Next to advance the iterator, and Key/Value to access each entry. Next
// returns false when the iterator is exhausted. Range follows the same
// iteration semantics as a range statement.
func Range[K comparable, V any, M constraints.Map[K, V]](m M) *Iter[K, V]`

// An Iter is an iterator for ranging over a map. See Range.
type Iter[K comparable, V any] struct {
    // Contains unexported fields
}

func (it *Iter[K, V]) Next() bool

func (it *Iter[K, V]) Key(p *K)

func (it *Iter[K, V]) Value(p *V)

Range and Iter would be type-safe versions of (reflect.Value).MapRange and reflect.MapIter respectively. The implementation would call into runtime for their implementation (and could hopefully be inlined, to be just as efficient as range).

On its own, this isn't very useful - we can just use range directly. However, it would enable us to get efficient, leak-free iterators of maps, which can be resumed and passed around. More importantly, this could be used by container/set to replace Do with a better API - allowing the user to use break and return early as needed without extra hoops to jump through and not having to worry about the closure escaping or anything like this.

It would also lay the groundwork for an iteration API - for example, a package could define

type Iter[V any] interface {
    Next() bool
    Value(p *V)
}

to be able to iterate over either a map or a container/set.Set and it would be an interface easily implemented by third-party collections as well.

Lastly, this is something that can't really be implemented outside of the stdlib. It requires interaction with the runtime and its implementation details.

10 replies
@Merovius

I chose the signatures of Iter.{Key,Value} based on #46131. It occurs to me that this might not apply to a reflect-less API like this. If we can verify that Key() K and Value() V don't do extra allocations, I think they'd be preferable.

@rsc

I think we are explicitly trying to avoid defining iterators in this round. There are many ways that could go, and it is not clear which way is best.

@deanveloper

I think that we should possibly wait for some form of generic range-able iterator feature (#43557 or similar) before committing to something for this

[ Resolved -- leave for later ]

If there should be an EqualFunc counterpart to Equal, shouldn't there also be a CloneFunc counterpart to Clone:

func CloneFunc[K comparable, V1, V2 any](m map[K]V1, transform func(V1) V2) map[K]V2
2 replies
@deanveloper

This would essentially just be a Transform function which may be more akin to a stream/iter API.

@ianlancetaylor

This proposal is aimed to only cover widely used functionality. This one isn't common enough.

[ Resolved -- out of scope for this proposal ]

I think it would be extremely valuable if the standard library contained a standardized iterator interface, something like:

type Iterator[T any] interface {
    // Next returns true until there are no more elements in this iterator.
    func Next() bool
    // Value returns the current value of this iterator.
    func Value() T
}

It would allow to get the keys or values of a map without allocating a slice.

keys := maps.Keys(myMap);
for keys.Next() {
    fmt.Println(keys.Value())
}

If putting the values in a slice is required, a function to do this can be created easily in the slices package.

func Collect[T any](it Iterator[T]) []T {
    var result []T
    for it.Next() {
        result = append(result, it.Value())
    }
    return result
}

I don't think language support for Iterators is necessary, but I think that defining a standard interface for them is a necessity. If no official interface for iterators is provided, it's almost certain that multiple implementations will pop up in 3rd party libraries and lead to fragmentation of the ecosystem.

Plus it allows us to create potentially less memory intensive programs by avoiding slice allocations.

4 replies
@DeedleFake

There's been some discussion of this elsewhere, including in this discussion as well as in #43557. Unfortunately, creating an iterator over a map in Go without language changes is surprisingly difficult, as the only way to iterate through one is with range, and that's incompatible with a Next() method based iterator without a background thread.

@sbstp

Yeah it would require access to the functions in runtime/map.go

@Merovius

@DeedleFake FWIW it doesn't need a language-change - after all, reflect.MapIter exists. But it does require cooperation from the runtime, which is why the stdlib is the only place where it could feasibly be done.

That being said, the idea of adding an iterator interface, as well as the idea to expose a concrete iterator over maps, have been rejected decisively, for go 1.18. It is not going to happen.

DeleteIf still doesn't feel exactly right to me. It occurs to me that slices, bytes, and strings all have a precedent for this kind of thing already: IndexFunc, ContainsFunc, and so on. It sure seems like slices.Index is to slices.IndexFunc as maps.Delete is to __this function__, which suggests it should be called DeleteFunc.

Here are all the functions taking functions as arguments in the standard library:

% grep -h 'func .*, func' go/api/* | sort
pkg bytes, func FieldsFunc([]uint8, func(int32) bool) [][]uint8
pkg bytes, func IndexFunc([]uint8, func(int32) bool) int
pkg bytes, func LastIndexFunc([]uint8, func(int32) bool) int
pkg bytes, func TrimFunc([]uint8, func(int32) bool) []uint8
pkg bytes, func TrimLeftFunc([]uint8, func(int32) bool) []uint8
pkg bytes, func TrimRightFunc([]uint8, func(int32) bool) []uint8
pkg crypto, func RegisterHash(Hash, func() hash.Hash)
pkg flag, func Func(string, string, func(string) error)
pkg go/ast, func Inspect(Node, func(Node) bool)
pkg go/parser, func ParseDir(*token.FileSet, string, func(os.FileInfo) bool, Mode) (map[string]*ast.Package, error)
pkg image, func RegisterFormat(string, string, func(io.Reader) (Image, error), func(io.Reader) (Config, error))
pkg math/rand, func Shuffle(int, func(int, int))
pkg net/http, func HandleFunc(string, func(ResponseWriter, *Request))
pkg os, func Expand(string, func(string) string) string
pkg reflect, func MakeFunc(Type, func([]Value) []Value) Value
pkg runtime/pprof, func Do(context.Context, LabelSet, func(context.Context))
pkg runtime/pprof, func ForLabels(context.Context, func(string, string) bool)
pkg runtime/trace, func WithRegion(context.Context, string, func())
pkg sort, func Search(int, func(int) bool) int
pkg sort, func Slice(interface{}, func(int, int) bool)
pkg sort, func SliceIsSorted(interface{}, func(int, int) bool) bool
pkg sort, func SliceStable(interface{}, func(int, int) bool)
pkg strings, func FieldsFunc(string, func(int32) bool) []string
pkg strings, func IndexFunc(string, func(int32) bool) int
pkg strings, func LastIndexFunc(string, func(int32) bool) int
pkg strings, func TrimFunc(string, func(int32) bool) string
pkg strings, func TrimLeftFunc(string, func(int32) bool) string
pkg strings, func TrimRightFunc(string, func(int32) bool) string
pkg testing, func AllocsPerRun(int, func()) float64
pkg time, func AfterFunc(Duration, func()) *Timer
% 

Obviously many of these aren't analogous, but the ones are analogous overwhelmingly use FooFunc.
And there's nothing at all ending in a conjunction like DeleteIf.
In particular it's not ContainsIf or IndexIf.

I suggest we use DeleteFunc.

2 replies
@deanveloper

I’m largely in support of this, but I think my only opposition to this is that the “XyzFunc” pattern typically just means “the exact same as Xyz but uses a function”. However, DeleteFunc isn’t the exact same as delete, since it may be (and will likely be used for) deleting more than one element.

@bcmills

@deanveloper, you could file a proposal to make the delete built-in variadic. 😁

I believe it will be very valuable to add a function like PutIf(Key, Value) (bool), It'll update only if the key is not present already.

0 replies

Adding a method to create immutable readonly map replica could be useful, it'll have the same application as passing <- chan to any method. Easier to expose/access data between apis/packages.

1 reply
@tw-ayush

Could be as simple as returning an Iterator or returning a map with const set of keys.

Just realized: The top-comment doesn't use constraints.Map. That should probably be fixed.

26 replies
@ianlancetaylor

Thanks. I fixed Clone to use constraints.Map. I think that's the only case where it makes a difference.

@Merovius

FWIW I've been talking to @rogpeppe and there is one situation where it also makes a difference if you don't have to return the type: In case you need to instantiate the function and get a func of the right type. It's a bit of a stretch, but for example, you might want to pass maps.Equal[MyMapType] to slices.EqualFunc[MyMapType, MyMapType]. It's also a bit strange to have to instantiate some functions as F[K, V] and some as F[map[K]V, K, V].

IMHO, every function which is intended to work on "any slice/map/chan" type should use constraint type inference - if nothing else, then just for uniformity.

@fzipp

IMHO, every function which is intended to work on "any slice/map/chan" type should use constraint type inference - if nothing else, then just for uniformity.

I'm becoming increasingly concerned about the infestation of every single function signature related to slices, maps and channels with the unwieldy constraints package. It's definitely not beautiful:

func Clone[M constraints.Map[K, V], K comparable, V any](m M) M

The actual parameter list is atrophied and doesn't communicate anything anymore, the type parameter list completely takes over, and nothing in the whole signature looks like an actual Go map type anymore. Is there any way this can be remedied? Like

func Clone[K comparable, V any](m ~map[K]V) ~map[K]V

or similar?

This comment was marked as off-topic.

I'm concerned about the complexity of these function signatures. It's definitely better now that we can use ~map[K]V instead of constraints.Map[K, V] (see the current version of maps.go), but the function signatures, which of course will be shown in some fashion in the package docs, are rather unwieldy -- they remind me of Scala, and that's not a place I want Go to go.

For reference, here's the current signature:

func Keys[M ~map[K]V, K comparable, V any](m M) []K { ... }

The simpler signature:

func Keys[K comparable, V any](m map[K]V) []K { ... }

Will be simpler to read in the docs, and actually uses map[K]V in the main parameter list so is easier to understand conceptually. The simpler version works fine with ordinary maps and with named types whose underlying type is a map (see here).

The only downside it has is what @Merovius pointed out above -- if you need to instantiate or pass around a function of the concrete type. (An example @rogpeppe showed me is here). As Axel acknowledged above, "it's a bit of a stretch". I think this will be very rare, and I believe doesn't warrant complicating the signatures -- normal use will just be a type-inferred call like maps.Keys(m). Besides, in the rare cases you need to instantiate the concrete function type, you can write a one-line wrapper function to get it.

The only function that really needs it is Clone, as it returns a map. I don't mind keeping that one as is -- I think it would be better to have 7 simpler signatures and 1 more complex one than 8 more complex (consistent) ones. And you could even rotate the M type to the end in Clone so the K comparable, V any type parameters are first for all of them. We could add a sentence to the Clone doc to explain something like "M is constrained to ~map[K]V so that the returned map has the same type as map being cloned, even if a named map type is being used" or similar (this wasn't obvious to me at first).

There was discussion about avoiding constraints.Map above (which is no longer needed, phew), but the thread above kind of devolved into a syntax discussion, which we're understandably not going to change at this point. So I'd love it if we can discuss simplifying the signatures specifically.

0 replies

This comment has been hidden.

Having used the x/exp/maps package for a little, I feel like the type parameters for maps.Values() and maps.Keys() are a bit cumbersome when it comes to the comparable type parameters:
func Keys[M ~map[K]V, K comparable, V any](m M) []K
func Values[M ~map[K]V, K comparable, V any](m M) []V

It's not unusual to have an interface as key types in a map, which however immediately means not being able to use these functions. This requirement of having comparable keys seems unnecessary as well, given that no comparison is done in either of these functions.

I understand that this might boil down to the argument of comparable including interfaces or not, but it seems reasonable to accept any as the map keys to these functions as this comparability requirement doesn't exist for these functions. The existence of these maps guarantees that they have comparable keys anyway.

2 replies
@Merovius

See #51338 for some discussion about this problem. In particular, it would be helpful to provide use-cases of where you use interfaces as key types.

Note that the constraint has to be comparable to be able to even write map[K]V, even if you don't do any comparisons. So, at least for Go 1.18, this is where we're at - it's simply not possible to use generic maps with interface key types.

@Sandertv

I did not know a constraint had to be comparable to be able to write map[K]V. In that case I rest my case here, as this was a specific comment for those methods. Thanks for clearing that up!

it would be helpful to provide use-cases of where you use interfaces as key types.

One classic example is map[reflect.Type]T.

0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
generics Discussion-Closed