Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spec: review/clarify uses of "slice of bytes" #23814

Closed
griesemer opened this issue Feb 13, 2018 · 18 comments
Closed

spec: review/clarify uses of "slice of bytes" #23814

griesemer opened this issue Feb 13, 2018 · 18 comments
Milestone

Comments

@griesemer
Copy link
Contributor

griesemer commented Feb 13, 2018

This is a follow-up issue for #23536: Comb through documentation (spec, builtins) to verify the use of the term "slice of bytes". The spec appears to mean by this any type whose underlying type is []byte. But not all implementations agree.

@mdempsky
Copy link
Member

mdempsky commented Sep 27, 2019

I just ran into this issue. I think it's valuable to at least come to a shared agreement on what we think the right behavior is, so we can start aligning the implementations.

I think the spec cases relevant here are:

  1. Conversions: string -> []byte, string -> []rune, []byte -> string, []rune -> string
  2. copy([]byte, string)
  3. append([]byte, string...)

My 2c is that the spec should allow this compilation unit:

package p

type (
	String string
	Byte   byte
	Bytes  []Byte
	Rune   rune
	Runes  []Rune
)

var (
	s  String
	bs Bytes
	rs Runes
)

func f() {
	bs = Bytes(s)
	rs = Runes(s)

	s = String(bs)
	s = String(rs)

	copy(bs, s)
	bs = append(bs, s...)
}

That is, I think we should liberally interpret "slice of bytes" and "slice of runes" to mean "slice type with element type of underlying type byte/rune". I.e., defined slice types and (user-)defined element type should be allowed.

gccgo accepts this.

cmd/compile and go/types currently reject the copy call, claiming that Bytes's element type (Byte) is different from byte.

go/types currently additionally rejects the append call too.

(Also notably, cmd/compile's typechecker allows String(Bytes(nil)) and String(Runes(nil)), but then walk.go fails to generate the appropriate conversions when calling runtime.slicebytetostring and runtime.slicerunetostring, so compilation fails. It has a similar problem with the append call too.)

@mdempsky
Copy link
Member

mdempsky commented Sep 27, 2019

/cc @griesemer @ianlancetaylor

@griesemer griesemer added the NeedsDecision label Oct 22, 2019
@gopherbot gopherbot removed the NeedsFix label Oct 22, 2019
@griesemer griesemer modified the milestones: Unplanned, Backlog Oct 22, 2019
@griesemer
Copy link
Contributor Author

griesemer commented Oct 22, 2019

@rsc, @ianlancetaylor, @robpike Do you have any opinions here?

We're inconsistent at the moment and different implementations (cmd/compile, gccgo, go/types) differ for operations where this matters.

@bradfitz
Copy link
Contributor

bradfitz commented Oct 23, 2019

Personally, if there's any ambiguity that needs to be specified, I'd prefer to be a permissive as possible, letting copies/appends/conversions work as expected if the underlying representation is the same.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Oct 31, 2019

I'm inclined to agree with @mdempsky 's comment #23814 (comment) . I don't see any particular reason to restrict operations on a "slice of bytes" to only the type []byte.

@josharian
Copy link
Contributor

josharian commented Feb 2, 2020

I just rediscovered this in #36965.

Is there sufficient consensus here that I can fix cmd/compile’s implementation to use a maximally liberal interpretation of “byte slices?

Other things to check on once the desired direction is clear: is the spec clear enough? Do go/types, package reflect, cmd/compile, and gccgo agree?

@josharian
Copy link
Contributor

josharian commented Feb 2, 2020

(The reason I’d like to fix cmd/compile now/soon is that I’m touching this code anyway for #36890.)

@griesemer
Copy link
Contributor Author

griesemer commented Feb 3, 2020

@josharian The reason for this issue is that not all implementations agree. @mdempsky 's comment explains how they disagree.

I think we should first get the spec to be crystal clear before making changes. It sounds like we're leaning towards making the interpretation more liberal, so this should not break future code, but enable more. This is not urgent, either.

@griesemer
Copy link
Contributor Author

griesemer commented Feb 3, 2020

Marking for Go 1.15.

@griesemer griesemer modified the milestones: Backlog, Go1.15 Feb 3, 2020
@ianlancetaylor ianlancetaylor modified the milestones: Go1.15, Go1.16 May 19, 2020
@odeke-em odeke-em modified the milestones: Go1.16, Go1.17 Feb 5, 2021
@griesemer
Copy link
Contributor Author

griesemer commented Apr 29, 2021

It looks like the consensus here is that "slice of bytes" should be maximally liberal, which would mean any slice of elements whose underlying type is bytes.

@griesemer griesemer modified the milestones: Go1.17, Go1.18 Apr 29, 2021
@griesemer griesemer removed the NeedsDecision label Apr 29, 2021
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jan 28, 2022

@griesemer This issue has milestone 1.18; shall we move to 1.19 or backlog?

@griesemer
Copy link
Contributor Author

griesemer commented Jan 28, 2022

This is documentation that we might be able to address. Prefer to leaving open for now.

@griesemer
Copy link
Contributor Author

griesemer commented Mar 10, 2022

Moving to 1.19.

See also: test cases.

@griesemer griesemer modified the milestones: Go1.18, Go1.19 Mar 10, 2022
@gopherbot
Copy link

gopherbot commented Jun 14, 2022

Change https://go.dev/cl/412094 mentions this issue: spec: clarify "slice of bytes" and "slice of runes" through examples

@griesemer
Copy link
Contributor Author

griesemer commented Jun 14, 2022

The above CL adjusts the spec to interpret slice of bytes/runes in the most liberal way (underlying type of the slices' element types must be byte/rune) for conversions. This matches the current implementation (and closely matches 1.17 - see the CL description).

The behavior for copy/append is (and always was) more strict; also, also no explicit conversion happens in those cases, so the more strict behavior can be justified.

Since this issue is about "slice of bytes/runes", terminology that is only used in the Conversions section of the spec, the CL addresses only that aspect.

If we want to change the behavior for append/copy, we probably need to file a mini-proposal.

@gopherbot
Copy link

gopherbot commented Jun 14, 2022

Change https://go.dev/cl/412095 mentions this issue: test: add tests for string/[]byte/[]rune conversions

gopherbot pushed a commit that referenced this issue Jun 15, 2022
Matches examples in spec section on string conversions.

For #23814.

Change-Id: I08099c27bfdb98735868266f5a42901321b97b56
Reviewed-on: https://go-review.googlesource.com/c/go/+/412095
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Robert Griesemer <gri@google.com>
@go101
Copy link

go101 commented Jun 15, 2022

Maybe, the reflect implementation should also be made consistent with this change: #23536 (comment)

@griesemer
Copy link
Contributor Author

griesemer commented Jun 23, 2022

@go101 Filed #53523.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants