You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
proposal: maps,cmd/compile: special case and/or add function for getting an arbitrary entry from a map #66361
An alternative is the compiler could recognize that pattern: a range with a body containing only a break.
If so, the compiler could insert a call to a runtime func that takes 2 pointers (either/both of which may be nil) and updates them to a random map value if the map has non-zero length. If it's empty, the pointers are unmodified.
Reacted by Eric Lagergren, Zxilly, Oleg Kovalov, Alexey Palazhchenko, Naoki Sega, Carlo Alberto Ferraris, Robert Burke, Mauri de Souza Meneguzzo, Aliaksandr Valialkin, Emmanuel T Odeke and 7 more
And happen to implement that with a compiler optimization that recognizes the way users often write it today. That way existing users benefit, and future users have a more convenient and reliable spelling.
Reacted by Mateusz Poliwczak, Emmanuel T Odeke, jimmyfrasche, DeedleFake, Eric Lagergren, Oleg Kovalov, Oleg, James Pudney, Yuriy Nasretdinov, QuantumGhost and 7 more
An alternate, more general, approach would be to expose some way to get the nth map entry. That may be too general but it's worth exploring, if just for contrast.
It does not provide a fixed order over the map, but as long as you do not modify or iterate over the map the same n will return the same entry. It need not be O(1).
With that selecting an arbitrary element is unsafe.Entry(m, 0) and choosing a random element is unsafe.Entry(m, rand.IntN(len(m))). This would also let you draw k>1 elements from the map.
Of course if you want an arbitrary element a random one will do and you could avoid exposing it by just having a func in rand that calls the equivalent of unsafe.Entry(m, rand.IntN(len(m))) for you, but the lack of generic methods make that a bit cumbersome.
Reacted by Branden J Brown, Sean Liao, Axel Wagner, QuantumGhost, Brandon Bennett, Carlana, Aliaksandr Valialkin, Adel and jub0bs
An alternate, more general, approach would be to expose some way to get the nth map entry.
As exposed to users today, map entries are unordered, so there's no unique "nth map entry." We could change that and define some order, but that seems like a substantially broader change than proposed here, and I'm skeptical that's something we want to pursue without use cases.
Reacted by Branden J Brown, Eric Lagergren, Oleg Kovalov, QuantumGhost, Brandon Bennett, cherrymui and zhaolei
A function would be weird because it becomes O(cap(m)) on map with small ratios.
If you grow a map and remove many elements, this would be expensive operation (because we never shrink maps).
It easily do ~30ns when the fill up ratio is good, so your ~10s suggest you are doing 329 million requests (assuming the ratio is good). I know this is an extremely limited comparison, but are you doing in the range of 3b~32m requests ? If you are doing far less, your map is poorly filled up so there is gonna be lots of scanning of empty cells involved even with a compiler help (more state could be added to skip some scanning but that a change to the runtime more than the compiler).
The patch you sent make sense.
I'm saying I don't think adding a compiler fast path is gonna be a substantial improvement because a mostly empty sparse map is the wrong datastructure to pick random elements from.
How long was the profile you took ? or how many requests did it handled ? this can be used to estimate if the map is poorly filled up or not.
I think it is generally unobvious, that it's not an even distribution by taking the first element of iteration (https://go.dev/play/p/n50sJuL1O6s and for other lens it's still not even).
Do we actually want to encourage "getting random element directly from a map"? This clearly adds a confusion about the distribution
Reacted by Filippo Valsorda, Komu Wairagu, Alberto Donizetti, Brandon Bennett, Andrew Ayer, Aliaksandr Valialkin, Yuri, Leo Antunes and jub0bs
@pkositsyn I assume that the intention is to get an 'arbitrary' element (which could be the same each time) instead of a 'random' element. The specification leaves the order undefined. So, it is theoretically possible that another Go implementation (or this one in the future, albeit unlikely) implements an ordered map.
changed the title [-]cmd/compile: special case getting a random map entry?[/-][+]proposal: maps,cmd/compile: special case and/or add function for getting an arbitrary entry from a map[/+]on Mar 20, 2024
Most often when I use range m { ...; break}, I know that the map contains a single element and wish there was a library function maps.Sole:
package maps
// Sole returns the sole entry of map m. It panics if len(m) != 1.funcSole[Kcomparable, Vany](mmap[K]V) (K, V) {
ifn:=len(m); n>1 {
panic(fmt.Sprintf("map contains %d entries", n))
}
fork, v:=rangem { returnk, v }
panic("empty map")
}
There is nothing wrong with the code being fast. However, using a for loop to pick a random value from a map in the first place seems inappropriate, considering the reason why Go made map iteration random.
Before Go 1, we faced portability issues because we relied on a specific order when looping over maps. Now we are trying to rely on the property that a random value is returned when looping over maps.
I think what we really need here is a function that can quickly return a single random value, not to strengthen the for loop.
Again, I think that making code faster is a good thing. But isn't it bad to rely on the property of looping over maps?
The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next.
The language specification does not guarantee that the order will be the same for each iteration. However, it also does not guarantee that different values will be produced. It only means that the order does not have to be the same.
I think that expecting randomness by looping over a map is a bad idea that relies too heavily on internal implementation details that are not part of the language specification.
Activity
gopherbot commentedon Mar 17, 2024
Change https://go.dev/cl/572119 mentions this issue:
database/sql: optimize connection request poolmdempsky commentedon Mar 17, 2024
We could add:
And happen to implement that with a compiler optimization that recognizes the way users often write it today. That way existing users benefit, and future users have a more convenient and reliable spelling.
seankhliao commentedon Mar 17, 2024
it appears to be somewhat common https://github.com/search?q=language%3AGo+%2Ffor.*%3D.*range.*%5Cn%5Cs*break%24%2F&type=code
jimmyfrasche commentedon Mar 17, 2024
An alternate, more general, approach would be to expose some way to get the nth map entry. That may be too general but it's worth exploring, if just for contrast.
Let's say there's
It does not provide a fixed order over the map, but as long as you do not modify or iterate over the map the same
nwill return the same entry. It need not beO(1).With that selecting an arbitrary element is
unsafe.Entry(m, 0)and choosing a random element isunsafe.Entry(m, rand.IntN(len(m))). This would also let you drawk>1elements from the map.Of course if you want an arbitrary element a random one will do and you could avoid exposing it by just having a func in
randthat calls the equivalent ofunsafe.Entry(m, rand.IntN(len(m)))for you, but the lack of generic methods make that a bit cumbersome.mdempsky commentedon Mar 17, 2024
As exposed to users today, map entries are unordered, so there's no unique "nth map entry." We could change that and define some order, but that seems like a substantially broader change than proposed here, and I'm skeptical that's something we want to pursue without use cases.
Jorropo commentedon Mar 17, 2024
A function would be weird because it becomes
O(cap(m))on map with small ratios.If you grow a map and remove many elements, this would be expensive operation (because we never shrink maps).
Jorropo commentedon Mar 17, 2024
@bradfitz I'm not convinced a fast path would improve anything here, scanning the empty slots seems like a slow thing to do.
It easily do
~30nswhen the fill up ratio is good, so your ~10s suggest you are doing 329 million requests (assuming the ratio is good). I know this is an extremely limited comparison, but are you doing in the range of 3b~32m requests ? If you are doing far less, your map is poorly filled up so there is gonna be lots of scanning of empty cells involved even with a compiler help (more state could be added to skip some scanning but that a change to the runtime more than the compiler).bradfitz commentedon Mar 17, 2024
@Jorropo, it's a very heavily loaded server. I did the optimization based on data, not on a whim or for fun.
Jorropo commentedon Mar 17, 2024
The patch you sent make sense.
I'm saying I don't think adding a compiler fast path is gonna be a substantial improvement because a mostly empty sparse map is the wrong datastructure to pick random elements from.
How long was the profile you took ? or how many requests did it handled ? this can be used to estimate if the map is poorly filled up or not.
pkositsyn commentedon Mar 18, 2024
I think it is generally unobvious, that it's not an even distribution by taking the first element of iteration (https://go.dev/play/p/n50sJuL1O6s and for other lens it's still not even).
Do we actually want to encourage "getting random element directly from a map"? This clearly adds a confusion about the distribution
gophun commentedon Mar 18, 2024
@pkositsyn I assume that the intention is to get an 'arbitrary' element (which could be the same each time) instead of a 'random' element. The specification leaves the order undefined. So, it is theoretically possible that another Go implementation (or this one in the future, albeit unlikely) implements an ordered map.
bboreham commentedon Mar 18, 2024
Anecdata: I spent many hours tracking down odd behaviour in a gossip system due to the skewed distribution of this pattern.
pkositsyn commentedon Mar 18, 2024
If it is about getting "any" element I suggest renaming the issue to "getting arbitrary element entry in map"
8 remaining items
[-]cmd/compile: special case getting a random map entry?[/-][+]proposal: maps,cmd/compile: special case and/or add function for getting an arbitrary entry from a map[/+]adonovan commentedon Mar 27, 2024
Most often when I use
range m { ...; break}, I know that the map contains a single element and wish there was a library functionmaps.Sole:database/sql: optimize connection request pool
rsc commentedon Apr 3, 2024
I don't think we need to worry about giving every element exactly the same chance for the start of an iteration.
Special-casing this:
to be faster seems reasonable and does not require a proposal. Is anyone interested?
nanasi880 commentedon Jan 9, 2025
There is nothing wrong with the code being fast. However, using a for loop to pick a random value from a map in the first place seems inappropriate, considering the reason why Go made map iteration random.
Before Go 1, we faced portability issues because we relied on a specific order when looping over maps. Now we are trying to rely on the property that a random value is returned when looping over maps.
I think what we really need here is a function that can quickly return a single random value, not to strengthen the for loop.
Again, I think that making code faster is a good thing. But isn't it bad to rely on the property of looping over maps?
nanasi880 commentedon Jan 9, 2025
I have checked the Go language specification.
https://go.dev/ref/spec#For_statements
The language specification does not guarantee that the order will be the same for each iteration. However, it also does not guarantee that different values will be produced. It only means that the order does not have to be the same.
I think that expecting randomness by looping over a map is a bad idea that relies too heavily on internal implementation details that are not part of the language specification.
mateusz834 commentedon Apr 15, 2026
Well today I saw CL 745440 from Alan. And thought to myself that maybe we should have an
iter.First(). So it could be used as:maps.All(map).First().That might be a better place for such thing.
But note that this would not solve the performance, but might result with a clearer/simpler code.