Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: read-only escape analysis and avoiding string <-> []byte copies #2205

Open
bradfitz opened this issue Aug 29, 2011 · 28 comments
Open

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Aug 29, 2011

Many functions take a []byte but only read from it.

If the escape analysis code could flag parameters of a function as read-only, then code
which passes in a []byte(string) conversion could be cheaper.

bytes.IndexByte is one example.

For example, http/sniff.go does:

// -------------------
  // Index of the first non-whitespace byte in data.                                                                                               
  firstNonWS := 0
  for ; firstNonWS < len(data) && isWS(data[firstNonWS]); firstNonWS++ {
  }

func isWS(b byte) bool {
        return bytes.IndexByte([]byte("\t\n\x0C\n "), b) != -1
}

// -------------------

But it's faster to avoid re-creating the []byte:

func BenchmarkIndexByteLiteral(b *testing.B) {
        for i := 0; i < b.N; i++ {
        IndexByte([]byte("\t\n\x0C\n "), 'x')
        }
}

func BenchmarkIndexByteVariable(b *testing.B) {
        var whitespace = []byte("\t\n\x0C\n ")
        for i := 0; i < b.N; i++ {
                IndexByte(whitespace, 'x')
        }
}

bytes_test.BenchmarkIndexByteLiteral    20000000           125 ns/op
bytes_test.BenchmarkIndexByteVariable   100000000           25.4 ns/op

Related is issue #2204.
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 30, 2011

Comment 1 by jp@webmaster.ms:

It seem to be much easier to add the possibility of zerocopish taking string slices from
byte slices/arrays.
As far I understand, string slices are actually readonly and cap()'less byte slices:
http://research.swtch.com/2009/11/go-data-structures.html

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Aug 30, 2011

Comment 2:

zerocopish?

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Aug 30, 2011

Comment 3:

Can you be more specific?
Examples of programs that you think should be
handled specially would be a good way to do that.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Sep 15, 2011

Comment 4:

Owner changed to @rsc.

Status changed to Accepted.

Loading

@lvdlvd
Copy link

@lvdlvd lvdlvd commented Nov 7, 2011

Comment 5:

Labels changed: added compilerbug, performance.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Dec 9, 2011

Comment 6:

Labels changed: added priority-later.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Sep 12, 2012

Comment 8:

Labels changed: added go1.1maybe.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Oct 25, 2012

Comment 9:

I run into this often, but I should start listing examples.
In goprotobuf, text.go calls writeString with both a string and a []byte converted to a
string:
// writeAny writes an arbitrary field.
func writeAny(w *textWriter, v reflect.Value, props *Properties) {
        v = reflect.Indirect(v)
        // We don't attempt to serialise every possible value type; only those
        // that can occur in protocol buffers, plus a few extra that were easy.
        switch v.Kind() {
        case reflect.Slice:
                // Should only be a []byte; repeated fields are handled in writeStruct.
                writeString(w, string(v.Interface().([]byte)))
        case reflect.String:
                writeString(w, v.String())
Note that the function writeString reallly just wants a read-only slice of bytes:
func writeString(w *textWriter, s string) {
        w.WriteByte('"')
        // Loop over the bytes, not the runes.
        for i := 0; i < len(s); i++ {
                // Divergence from C++: we don't escape apostrophes.
                // There's no need to escape them, and the C++ parser
                // copes with a naked apostrophe.
                switch c := s[i]; c {
                case '\n':
                        w.Write([]byte{'\\', 'n'})
                case '\r':
                        w.Write([]byte{'\\', 'r'})
                case '\t':
                        w.Write([]byte{'\\', 't'})
                case '"':
                        w.Write([]byte{'\\', '"'})
                case '\\':
                        w.Write([]byte{'\\', '\\'})
                default:
                        if isprint(c) {
                                w.WriteByte(c)
                        } else {
                                fmt.Fprintf(w, "\\%03o", c)
                        }
                }
        }
        w.WriteByte('"')
}
It doesn't matter that it's frozen (like a string), nor writable (like a []byte).  But
Go lacks that type, so if instead it'd be nice to write writeAny with a []byte parameter
and invert the switch above to be like:
        switch v.Kind() {
        case reflect.Slice:
                // Should only be a []byte; repeated fields are handled in writeStruct.
                writeString(w, v.Interface().([]byte))
        case reflect.String:
                writeString(w, []byte(v.String())) // no copy!
Where the []byte(v.String()) just makes a slice header pointing in to the string's
memory, since the compiler can verify that writeAny never mutates its slice.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Nov 17, 2012

Comment 10:

See patch and CL description at http://golang.org/cl/6850067 for the opposite
but very related case: strconv.ParseUint, ParseBool, etc take a string but calling code
has a []byte.

Loading

@robpike
Copy link
Contributor

@robpike robpike commented Mar 7, 2013

Comment 11:

Labels changed: removed go1.1maybe.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Mar 12, 2013

Comment 12:

[The time for maybe has passed.]

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Mar 31, 2013

Comment 13:

People who like this bug also like issue #3512 (cmd/gc: optimized map[string] lookup from
[]byte key)

Loading

@dvyukov
Copy link
Member

@dvyukov dvyukov commented Mar 31, 2013

Comment 14:

FWIW
var m map[string]int
var b []byte
_ = m[string(b)]
case must be radically simpler to implement than general read-only analysis.
It's peephole optimization, when the compiler sees such code it can generate hacky
string object using the []byte pointer.
Another example would be len(string(b)), but this seems useless.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Mar 31, 2013

Comment 15:

Yes. That's why they're separate bugs.
I want issue #3512 first, because it's easy.

Loading

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 26, 2013

Comment 16:

Labels changed: added garbage.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Jul 30, 2013

Comment 18:

Labels changed: added priority-someday, removed priority-later.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 19:

Labels changed: added repo-main.

Loading

@rsc
Copy link
Contributor

@rsc rsc commented Mar 3, 2014

Comment 20:

Adding Release=None to all Priority=Someday bugs.

Labels changed: added release-none.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 30, 2018

Change https://golang.org/cl/146018 mentions this issue: strings: declare Index as noescape

Loading

gopherbot pushed a commit that referenced this issue Oct 30, 2018
This lets []byte->string conversions which are used as arguments to
strings.IndexByte and friends have their backing store allocated on
the stack.

It only prevents allocation when the string is small enough (32
bytes), so it isn't perfect. But reusing the []byte backing store
directly requires a bunch more compiler analysis (see #2205 and
related issues).

Fixes #25864.

Change-Id: Ie52430422196e3c91e5529d6e56a8435ced1fc4c
Reviewed-on: https://go-review.googlesource.com/c/146018
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@josharian
Copy link
Contributor

@josharian josharian commented Jan 18, 2019

I run into this often, but I should start listing examples.

Good idea. From #29802/#29810: encoding/hex.Decode.

Loading

@cespare
Copy link
Contributor

@cespare cespare commented Jan 18, 2019

This comes up with hash functions (example: sha256.Sum256([]byte(s))).

For github.com/cespare/xxhash in addition to Sum64 I added Sum64String which does the unsafe string-to-slice conversion:

https://github.com/cespare/xxhash/blob/3767db7a7e183c0ad3395680d460d0b61534ae7b/xxhash_unsafe.go#L28-L35

but I'd rather the caller just be able to use Sum64([]byte(s)) and have the compiler optimize it.

Loading

@go101
Copy link

@go101 go101 commented Apr 18, 2019

If read-only byte slice is supported, we can set the cap of read-only byte slices as a negative value to indicate the underlying bytes are immutable. []byte(aString) results a read-only byte slice without duplicating the underlying bytes of aString. Convertingthe result byte slice, if its cap is a negative value, back again to a string also needs not to duplicate the underlying bytes.

Without the read-only slice feature, is it possible to lazy duplicate the underlying bytes for coversion []byte(aString) on demand?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 13, 2020

What I think might be useful:

  • For every function/method that has a parameter of type []byte, record whether the slice is modified.
    • Record this in the export data so that the information is correct interprocedurally.
  • When calling a function with an argument of the form []byte(s) for some s of type string, if the function does not modify the []byte, don't copy the string.
  • For an interface with a method M that has a parameter of []byte, add an additional hidden method MString which has a parameter of type string.
    • Perhaps only for methods with a single []byte parameter.
    • Perhaps wait until we have profile guided optimization and use that to see whether this is useful.
  • When converting a type to this interface, if the type's M method does not modify the []byte, have MString forward to M.
  • If the type's M method does modify the []byte, have MString copy to []byte and call M.
  • Given an value of the interface type, when calling M with []byte(s), call MString instead.

Loading

@go101
Copy link

@go101 go101 commented Nov 13, 2020

For every function/method that has a parameter of type []byte, record whether the slice is modified.

This might be hard to implement overall, mainly caused by the cases in which the []byte parameter is passed to an interface method within the function body. We could think such functions will modify the slice parameters, but this will discount the usefulness of the idea much.

Loading

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 20, 2020

When converting a type to this interface, if the type's M method does not modify the []byte, have MString forward to M.

Many io.Writer implementations wrap one or more other io.Writer implementations invoked somewhere in the method body. How would this forwarding logic work for those implementations?

Loading

@mdempsky mdempsky self-assigned this Jan 22, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 23, 2021

Change https://golang.org/cl/285681 mentions this issue: [dev.regabi] cmd/compile: optimize read-only []byte conversions

Loading

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Jan 23, 2021

Made some initial progress towards this in CL 285681. It needs polish and more tests, but it works well enough for Brad's initial test case.

One known issue at the moment is that it optimizes []byte("") to []byte(nil) instead of []byte{}, so it could misbehave for programs that are sensitive to the difference. (I'll of course fix this before sending the CL for review.) But otherwise, I think it should work, and folks could test it out to see if there are notable cases where it helps or (even more critically) causes problems.

It currently finds 188 instances of string->[]byte conversions within GOROOT that can be converted without copying:

188 zero-copy byte-slice conversions from CL 285681
cmd/cgo/main.go:383:25: zero-copy byte-slice conversion
cmd/cgo/out.go:129:41: zero-copy byte-slice conversion
cmd/compile/internal/ssa/html.go:1186:38: zero-copy byte-slice conversion
cmd/compile/internal/syntax/scanner.go:424:19: zero-copy byte-slice conversion
cmd/cover/cover.go:170:13: zero-copy byte-slice conversion
cmd/dist/test.go:1646:26: zero-copy byte-slice conversion
cmd/dist/util.go:241:15: zero-copy byte-slice conversion
cmd/go/internal/bug/bug.go:176:15: zero-copy byte-slice conversion
cmd/go/internal/bug/bug.go:208:63: zero-copy byte-slice conversion
cmd/go/internal/cache/cache.go:375:24: zero-copy byte-slice conversion
cmd/go/internal/cache/default.go:51:52: zero-copy byte-slice conversion
cmd/go/internal/envcmd/env.go:507:15: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:269:19: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:269:19: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:283:36: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:283:36: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:289:19: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:289:19: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:323:36: zero-copy byte-slice conversion
cmd/go/internal/generate/generate.go:323:86: zero-copy byte-slice conversion
cmd/go/internal/modcmd/download.go:98:26: zero-copy byte-slice conversion
cmd/go/internal/modcmd/vendor.go:116:25: zero-copy byte-slice conversion
cmd/go/internal/modcmd/vendor.go:121:26: zero-copy byte-slice conversion
cmd/go/internal/modcmd/vendor.go:149:26: zero-copy byte-slice conversion
cmd/go/internal/modfetch/codehost/codehost.go:213:44: zero-copy byte-slice conversion
cmd/go/internal/modfetch/codehost/git.go:181:41: zero-copy byte-slice conversion
cmd/go/internal/modfetch/codehost/git.go:828:51: zero-copy byte-slice conversion
cmd/go/internal/modfetch/codehost/git.go:869:31: zero-copy byte-slice conversion
cmd/go/internal/modfetch/codehost/git.go:870:26: zero-copy byte-slice conversion
cmd/go/internal/test/test.go:1437:38: zero-copy byte-slice conversion
cmd/go/internal/test/test.go:1473:50: zero-copy byte-slice conversion
cmd/go/internal/trace/trace.go:147:26: zero-copy byte-slice conversion
cmd/go/internal/trace/trace.go:156:32: zero-copy byte-slice conversion
cmd/go/internal/version/exe.go:41:33: zero-copy byte-slice conversion
cmd/go/internal/version/exe.go:49:33: zero-copy byte-slice conversion
cmd/go/internal/version/exe.go:57:33: zero-copy byte-slice conversion
cmd/go/internal/version/exe.go:57:86: zero-copy byte-slice conversion
cmd/go/internal/web/api.go:224:55: zero-copy byte-slice conversion
cmd/go/internal/work/buildid.go:338:55: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:1471:38: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2332:35: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2372:52: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2374:34: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2392:67: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2589:42: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2590:30: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2591:30: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2592:30: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2874:34: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2878:47: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2884:41: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2895:52: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2899:50: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:2908:34: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:3098:35: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:3177:33: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:3177:78: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:640:37: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:640:77: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:641:35: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:641:75: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:642:35: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:642:76: zero-copy byte-slice conversion
cmd/go/internal/work/exec.go:96:39: zero-copy byte-slice conversion
cmd/internal/archive/archive.go:420:27: zero-copy byte-slice conversion
cmd/internal/buildid/rewrite.go:29:19: zero-copy byte-slice conversion
cmd/internal/codesign/codesign.go:245:26: zero-copy byte-slice conversion
cmd/internal/dwarf/dwarf.go:1628:33: zero-copy byte-slice conversion
cmd/internal/dwarf/dwarf.go:1635:36: zero-copy byte-slice conversion
cmd/internal/dwarf/dwarf.go:1636:40: zero-copy byte-slice conversion
cmd/internal/pkgpath/pkgpath.go:43:47: zero-copy byte-slice conversion
cmd/internal/pkgpath/pkgpath.go:56:31: zero-copy byte-slice conversion
cmd/internal/pkgpath/pkgpath.go:58:38: zero-copy byte-slice conversion
cmd/internal/pkgpath/pkgpath.go:60:38: zero-copy byte-slice conversion
cmd/link/internal/ld/data.go:2628:18: zero-copy byte-slice conversion
cmd/link/internal/ld/elf.go:1442:63: zero-copy byte-slice conversion
cmd/link/internal/ld/elf.go:1446:69: zero-copy byte-slice conversion
cmd/link/internal/ld/elf.go:593:18: zero-copy byte-slice conversion
cmd/link/internal/ld/elf.go:695:18: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1177:38: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1405:35: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1418:34: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1569:33: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1603:51: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1605:33: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1615:33: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1676:41: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1744:50: zero-copy byte-slice conversion
cmd/link/internal/ld/lib.go:1744:98: zero-copy byte-slice conversion
cmd/link/internal/ld/xcoff.go:1800:19: zero-copy byte-slice conversion
cmd/link/internal/loader/loader.go:2060:60: zero-copy byte-slice conversion
cmd/link/internal/loadpe/ldpe.go:366:84: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/fetch.go:524:22: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/options.go:63:23: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/options.go:91:15: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/webui.go:336:42: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/webui.go:336:56: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/internal/driver/webui.go:339:35: zero-copy byte-slice conversion
cmd/vendor/github.com/google/pprof/profile/legacy_java_profile.go:69:34: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:233:36: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:318:9: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:323:9: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:380:11: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:527:10: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:542:11: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:559:10: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:560:10: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:574:53: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:614:10: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:663:20: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:668:21: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:813:20: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:818:21: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/crypto/ssh/terminal/terminal.go:824:9: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/modfile/read.go:524:58: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:413:36: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:486:33: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:486:47: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:488:51: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:488:51: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:489:51: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/client.go:489:51: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/test.go:56:25: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/tlog/note.go:126:29: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/mod/sumdb/tlog/note.go:53:67: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:976:26: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/sys/unix/syscall_linux.go:979:26: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/sys/unix/syscall_unix.go:355:29: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/tools/go/analysis/passes/buildtag/buildtag.go:112:34: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/tools/go/cfg/cfg.go:149:49: zero-copy byte-slice conversion
cmd/vendor/golang.org/x/tools/go/cfg/cfg.go:149:63: zero-copy byte-slice conversion
crypto/tls/handshake_messages.go:131:25: zero-copy byte-slice conversion
crypto/tls/handshake_messages.go:210:26: zero-copy byte-slice conversion
crypto/tls/handshake_messages.go:658:25: zero-copy byte-slice conversion
crypto/tls/handshake_messages.go:861:25: zero-copy byte-slice conversion
crypto/tls/key_schedule.go:39:20: zero-copy byte-slice conversion
crypto/tls/key_schedule.go:40:20: zero-copy byte-slice conversion
crypto/x509/x509.go:1809:24: zero-copy byte-slice conversion
crypto/x509/x509.go:1829:24: zero-copy byte-slice conversion
crypto/x509/x509.go:1841:24: zero-copy byte-slice conversion
encoding/base64/base64.go:385:35: zero-copy byte-slice conversion
encoding/xml/marshal.go:345:35: zero-copy byte-slice conversion
encoding/xml/xml.go:1091:23: zero-copy byte-slice conversion
go/internal/gccgoimporter/ar.go:85:66: zero-copy byte-slice conversion
go/internal/gccgoimporter/ar.go:95:83: zero-copy byte-slice conversion
index/suffixarray/suffixarray.go:294:15: zero-copy byte-slice conversion
internal/trace/parser.go:308:35: zero-copy byte-slice conversion
internal/trace/writer.go:16:16: zero-copy byte-slice conversion
net/dnsclient_unix.go:241:30: zero-copy byte-slice conversion
net/dnsclient_unix.go:241:30: zero-copy byte-slice conversion
net/http/client.go:243:53: zero-copy byte-slice conversion
net/http/client.go:419:49: zero-copy byte-slice conversion
net/http/fcgi/child.go:304:51: zero-copy byte-slice conversion
net/http/h2_bundle.go:4053:38: zero-copy byte-slice conversion
net/http/httputil/dump.go:163:35: zero-copy byte-slice conversion
net/http/request.go:975:50: zero-copy byte-slice conversion
net/http/transport.go:845:30: zero-copy byte-slice conversion
os/file.go:249:23: zero-copy byte-slice conversion
reflect/type.go:2370:30: zero-copy byte-slice conversion
reflect/type.go:2411:27: zero-copy byte-slice conversion
reflect/type.go:2560:28: zero-copy byte-slice conversion
regexp/regexp.go:712:26: zero-copy byte-slice conversion
strconv/atoi.go:120:26: zero-copy byte-slice conversion
strconv/atoi.go:240:28: zero-copy byte-slice conversion
syscall/exec_linux.go:169:16: zero-copy byte-slice conversion
syscall/exec_linux.go:174:22: zero-copy byte-slice conversion
syscall/exec_linux.go:175:16: zero-copy byte-slice conversion
syscall/exec_linux.go:178:22: zero-copy byte-slice conversion
syscall/exec_linux.go:180:22: zero-copy byte-slice conversion
syscall/exec_linux.go:576:29: zero-copy byte-slice conversion
syscall/exec_linux.go:613:16: zero-copy byte-slice conversion
syscall/exec_linux.go:615:16: zero-copy byte-slice conversion
syscall/lsf_linux.go:57:26: zero-copy byte-slice conversion
syscall/syscall_unix.go:333:29: zero-copy byte-slice conversion
vendor/golang.org/x/crypto/cryptobyte/asn1.go:116:20: zero-copy byte-slice conversion
vendor/golang.org/x/crypto/cryptobyte/asn1.go:116:20: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1847:15: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1847:15: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1851:24: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1851:24: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1857:19: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1857:19: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1857:19: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:1857:19: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:2379:31: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:2379:31: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:2381:35: zero-copy byte-slice conversion
vendor/golang.org/x/net/dns/dnsmessage/message.go:2381:35: zero-copy byte-slice conversion

Loading

@mdempsky
Copy link
Member

@mdempsky mdempsky commented Jan 23, 2021

Hm, CL 285681 tries to zero-copy this conversion, which would be bad:

package p

import "syscall"

func f() {
	syscall.Read(0, []byte("bad"))
}

I thought I was already conservatively handling when pointers are passed as uintptr to assembly functions, but apparently not. I'll have to dig into what's going on there.

Though it also means we'll probably need a way to distinguish syscalls that may mutate memory (e.g., Read) vs syscalls that won't (e.g., Write).

Looking through the 188 instances, I notice there are also some cases of range []byte(s), which we already recognize and optimize. Though it's nice to see this optimization generalizes it.

Loading

fatih added a commit to planetscale/cli that referenced this issue May 7, 2021
It looks like we don't need to rely on the common package. The `[]byte`
to `string` conversions shouldn't be an issue for us. The Go compiler is
smart enough to make some optimizations already and there are some work
on this area: golang/go#2205
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet