slices: add Concat

This came up in the discussion of #55358 and prior discussions. It would be nice to have a simple, generic way to concatenate a slice of slices.

Here's the proposed signature:

func Concat[S ~[]E, E any](dst S, slices ...S) S

Usage:

// Join slices into a new slice
a := []int{ 1, 2, 3 }
b := []int{ 4, 5, 6 }
c := slices.Concat(nil, a, b) 
// c == int{ 1, 2, 3, 4, 5, 6 }

// overwrite an existing slice
a = slices.Concat(a[:0], c[0:1], c[2:3], c[4:5]) 
// a == int{ 1, 3, 5 }

a := []int{1, 2, 3}
b := []int{4}
c := slices.Concat(a[:2], b, a[2:])

What is the value of c? Depending on the order of events weird things could happen.

For example, consider the following (perhaps naive) implementation:

func Concat[S ~[]E, E any](dst S, slices ...S) S {
	for _, s := range slices {
		dst = append(dst, s...)
	}
	return dst
}

This will result in c being [1 2 4 4] which is probably unexpected.

Isn't append already generic?

Isn't append already generic?

Append can't concatenate multiple slices. This means you need to jump through hoops to precalculate the capacity and to make a one-liner you end up with awkward things like append(s1, append(s2, s3...)...).

This will result in c being [1 2 4 4] which is probably unexpected.

I think it's only worth doing Concat if it does one shot preallocation, like

func Concat[S ~[]E, E any](dst S, ss ...S) S {
	c := 0
	for _, s := range ss {
		c += cap(s)
	}
	dst = Grow(dst, c)
	for _, s := range ss {
		dst = append(dst, s...)
	}
	return dst
}

If you run this, you get 1, 2, 4, 3.

@carlmjohnson

@carlmjohnson I don't think it should use cap as that could greatly overestimate the size requirement. For example, slices.Concat(nil, make([]int, 0, 1000)) should not grow the dest to 1000 elements.

Really it should be len, but that still causes problems. https://go.dev/play/p/zb1d-MhD6Pj

😅 You're right!

I think at a certain point, if the user is clobbering the slice they are reading from, it's a user error. You can probably make up an example using slices.Insert that is confusing too.

@carlmjohnson

@carlmjohnson It would be all too easy for someone not to realize that the first parameter is special in that regard, especially because it isn't in other languages that have similar APIs. And due to the nature of append/slices.Grow it will sometimes appear to work the way they expect, meaning they might not even notice the problem until it causes some production incident.

What if it were a method instead of a function, to reinforce the fact that the first parameter is actually the output buffer?

type Buffer[T any] []T

func (buf Buffer[T]) Concat(ss ...[]T) Buffer[T] {
	size := 0
	for _, s := range ss {
		size += len(s)
	}
	if size == 0 {
		return buf
	}
	buf = Grow(buf, len(buf)+size)
	for _, s := range ss {
		buf = append(buf, s...)
	}
	return buf
}

(This type name is just for example purposes.)

Then it would be slices.Buffer[int](a).Concat(b, c). For people who don't care to optimize it, they can call slices.Buffer[int](nil).Concat(a, b, c).

The objection to destination slices already applies to several functions in exp/slices: Insert, Delete, the upcoming Replace, and arguably Grow and Clip (although not as confusing as the others). If you think it's causing bugs, you should open an issue to make a v2 of the package before the current slices API gets into the standard library.

Yes, it is possible to break them, but it is a bit more contrived. https://go.dev/play/p/4lLBHhxrfdN

Regardless, adding a typedef with methods does not itself necessitate a v2. It just means some (or all) of the existing functions would presumably be deprecated.

I feel like two versions of concatenation could make sense - a minimally complicated, more variadic 'Append'-like concatenation, and a profligately eager, maximally allocating 'Collect'-ing concatenation:
https://go.dev/play/p/13N05VZPjwS

It is impossible to write normal benchmarks without hacking the runtime code. Someone ever hacked it.

There is no need to create a new issue.

slices: add Concat

@ianlancetaylor

@ianlancetaylor / @earthboundkid could one of you elaborate on what this comment means and why we shouldn't be using make here?

https://go-review.googlesource.com/c/go/+/504882/3..6/src/slices/slices.go#b510

Since we don't make any promises about the capacity, I recommend
newslice := Grow[S](nil, size)
That should lift the capacity up to the next size class.

If it used make there, the capacity would be exactly the combined size of the slices, but we want it to be rounded up to the next size class, so people don't rely on len(s) == cap(s) being true, and box ourselves into a corner.

Still not quite following, what do we mean by rounded up to the next size class? Do we mean:

cap(Grow[S](nil, size)) != size

@earthboundkid

@earthboundkid
I also don't quite understand why slices.Grow should be used instead of make in this case. I tested it with Go 1.22 using the link you provided, and contrary to what the post claims, this code does not allocate memory.

func BenchmarkSortStrings(b *testing.B) {
        s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
        b.ReportAllocs()
        for i := 0; i < b.N; i++ {
                sort.Strings(s)
        }
}

Therefore, I believe that this post is no longer valid.

In this case, make seems to be superior in terms of performance, and there doesn't seem to be a significant difference in readability either. Can you explain in more detail why slices.Grow should be used in this scenario please?

This is a closed issue. Your benchmark is invalid because sort.Strings isn't relevant to slices.Concat and the slice is sorted after the first loop. If you can make a change with better performance, submit the change with a benchmark from benchstat.

@OhJuhun

@OhJuhun
It looks the APIs in the slices package try to round up memory size to the next class size, except strings.Repeat. Honestly, I don't understand the cause of the inconsistency.

The current design of the slices package lacks of custom configurations:

whether or not round up to the next memory class size.
whether or not zero freed-up elements (in Delete, Replace and Compact etc)
whether or not keep element order (such as in Delete)
...

Because of these facts, the slices package doesn't satisfy the needs of all cases.
So sometimes, you might need to write your own implementations.

Navigation Menu

Repository navigation

slices: add Concat #56353

49 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

slices: add Concat #56353

Description

Activity

rittneje commented on Oct 20, 2022

ericlagergren commented on Oct 20, 2022

earthboundkid commented on Oct 20, 2022

earthboundkid commented on Oct 20, 2022

rittneje commented on Oct 20, 2022

earthboundkid commented on Oct 21, 2022

earthboundkid commented on Oct 21, 2022

rittneje commented on Oct 22, 2022

earthboundkid commented on Oct 22, 2022

rittneje commented on Oct 22, 2022

AndrewHarrisSPU commented on Oct 23, 2022

49 remaining items

zigo101 commented on Sep 1, 2023

justinhwang commented on Feb 22, 2024

earthboundkid commented on Feb 22, 2024

justinhwang commented on Feb 23, 2024

earthboundkid commented on Feb 23, 2024

OhJuhun commented on Jul 15, 2024

earthboundkid commented on Jul 15, 2024

zigo101 commented on Jul 16, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions