strings: Replace return wrong result on some machines

[galric]

I used strings.Replace in my application to replace prefix '/cloud/app' with '/cloud/data', and got wrong result sometime.

expect: "/cloud/data/abc/abc/abc/123"
actual: some times "/cloud/app/abc/abc/abc/123" , some times "/cloud/data/abc/abc/abc/123"

Debuged 'strings.Replace' with gdb i found strings.indexShortStr return bad result, which is written with assembly language and i am not able to debug it.

summary of gdb message

strings.Replace (stirngs.go: 696)  if m := Count(s, old); m == 0 { 
stirngs.Count (strings.go: 85)  i := Index(s[offset:], sep)
strings.Index (strings_amd64.go:42) return indexShortStr(s, sep)  
strings.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1743

strings.Index("/cloud/app/abc/abc/abc", "/cloud/app") return an unexpected -1 
cause strings.go:87 return 

full gdb message

Thread 1 hit Breakpoint 1, main.main () at /home/admin/work/tj16/go/src/my_application/main/test.go:22 warning: Source file is more recent than executable.
22          fmt.Printf("replace: %s\n", strings.Replace(dir, "/cloud/app", "/cloud/data", 1)) 

(gdb) s
strings.Replace (s="/cloud/app/abc/abc#/abc", old="/cloud/app", new="/cloud/data", n=1, ~r4="") at /home/admin/work/tj16/go/go/src/strings/strings.go:690
warning: Source file is more recent than executable.
690     func Replace(s, old, new1 string, n int) string {

(gdb) n
691         if old == new1 || n == 0 {

(gdb) n
696         if m := Count(s, old); m == 0 {

(gdb) s
strings.Count (s="/cloud/app/abc/abc#/abc", sep="/cloud/app", ~r2=842350510104)
    at /home/admin/work/tj16/go/go/src/strings/strings.go:77
77      func Count(s, sep string) int {

(gdb) n
80          if len(sep) == 0 {

(gdb) n
77      func Count(s, sep string) int {

(gdb) n
85              i := Index(s[offset:], sep)

(gdb) p s
$1 = "/cloud/app/abc/abc#/abc"

(gdb) p sep
$2 = "/cloud/app"

(gdb) s
89              n++

(gdb) n
85              i := Index(s[offset:], sep)

(gdb) s
strings.Index (s="/cloud/app/abc/abc#/abc", sep="/cloud/app", ~r2=0)
    at /home/admin/work/tj16/go/go/src/strings/strings_amd64.go:25
25      func Index(s, sep string) int {

(gdb) n
28              case n == 0:
(gdb) n
30              case n == 1:
(gdb) n
32              case n == len(s):
(gdb) n
37              case n > len(s):
(gdb) n
39              case n <= shortStringLen:
(gdb) n
41                      if len(s) <= 64 {
(gdb) n
42                              return indexShortStr(s, sep)
(gdb) s
strings.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1743
1743            MOVQ s+0(FP), DI
(gdb) p s
No symbol "s" in current context.
(gdb) n
1745            MOVQ s_len+8(FP), DX
(gdb) n
1746            MOVQ c+16(FP), BP
(gdb) n
1747            MOVQ c_len+24(FP), AX
(gdb) n
1748            MOVQ DI, R10
(gdb) n
1749            LEAQ ret+32(FP), R11
(gdb) n
1750            JMP  runtime·indexShortStr(SB)
(gdb) n
runtime.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1767
1767            CMPQ AX, DX
(gdb) n
1768            JA fail
(gdb) n
1769            CMPQ DX, $16
(gdb) n
1770            JAE sse42
(gdb) n
1970            MOVL runtime·cpuid_ecx(SB), CX
(gdb) n
1971            ANDL $0x100000, CX
(gdb) n
1972            JZ no_sse42
(gdb) n
1973            CMPQ AX, $12
(gdb) n
1978            JAE _9_or_more
(gdb) n
1979            LEAQ 16(BP), SI
(gdb) n
1980            TESTW $0xff0, SI
(gdb) n
1981            JEQ no_sse42
(gdb) n
1982            MOVOU (BP), X1
(gdb) n
1983            LEAQ -15(DI)(DX*1), SI
(gdb) n
1984            MOVQ $16, R9
(gdb) n
1985            SUBQ AX, R9 // We advance by 16-len(sep) each iteration, so precalculate it into R9
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) n
1996            JBE sse42_success
(gdb) n
1997            ADDQ R9, DI
(gdb) n
1998            CMPQ DI, SI
(gdb) n
1999            JB loop_sse42
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) n
1996            JBE sse42_success
(gdb) n
1997            ADDQ R9, DI
(gdb) n
1998            CMPQ DI, SI
(gdb) n
1999            JB loop_sse42
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) n
1996            JBE sse42_success
(gdb) n
1997            ADDQ R9, DI
(gdb) n
1998            CMPQ DI, SI
(gdb) n
1999            JB loop_sse42
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) n
1996            JBE sse42_success
(gdb) n
1997            ADDQ R9, DI
(gdb) n
1998            CMPQ DI, SI
(gdb) n
1999            JB loop_sse42
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) n
1996            JBE sse42_success
(gdb) n
1997            ADDQ R9, DI
(gdb) n
1998            CMPQ DI, SI
(gdb) n
1999            JB loop_sse42
(gdb) n
2000            PCMPESTRI $0x0c, -1(SI), X1
(gdb) n
2001            CMPQ CX, R9
(gdb) n
2002            JA fail
(gdb) n
1964            MOVQ $-1, (R11)
(gdb) n
runtime.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1965
1965            RET
(gdb) n
strings.Count (s="/cloud/app/abc/abc#/abc", sep="/cloud/app", ~r2=842350510104)
    at /home/admin/work/tj16/go/go/src/strings/strings.go:86
86              if i == -1 {
(gdb) n
87                  return n
(gdb) p s
$3 = "/cloud/app/abc/abc#/abc"
(gdb) p sep
$4 = "/cloud/app"
(gdb) l
82          }
83          offset := 0
84          for {
85              i := Index(s[offset:], sep)
86              if i == -1 {
87                  return n
88              }
89              n++
90              offset += i + len(sep)
91          }

What version of Go are you using (go version)?

go1.8

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)

cpu
model name : Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
address sizes : 46 bits physical, 48 bits virtual

What did you do?

call: strings.Replace("/cloud/app/abc/abc/abc/123", "/cloud/app", "/cloud/data", 1)

What did you expect to see?

"/cloud/data/abc/abc/abc/123"

What did you see instead?

some times "/cloud/app/abc/abc/abc/123" , some times "/cloud/data/abc/abc/abc/123"

[dcheney-atlassian]

@galric thank you for your detailed report. Can you please include a runnable example, perhaps from the Go playground, that shows the issue. Thank you.

[galric]

complete code:

package main

import (
    "flag"
    "fmt"
    "path/filepath"
    "strings"
)

func main() {
    downloadDir := flag.String("p", "", "")
    flag.Parse()

    fmt.Printf("downloadDir1: %s\n", *downloadDir)

    dir := strings.TrimRight(*downloadDir, "/")
    fmt.Printf("downloadDir2: %s\n", dir)

    dir = filepath.Dir(dir)
    fmt.Printf("dir: %s\n", dir)

    fmt.Printf("replace: %s\n", strings.Replace(dir, "/cloud/app", "/cloud/data", 1))

}

This problem happened on an physical machine in a intranet, and difficult to tell which system env has responsibility to it . It is really difficult for you to check this problem.

most of machines are good

can you list infos wanted

[mvdan]

This works as expected: https://play.golang.org/p/oeV3Bl17iy

Without a reliable way to reproduce the issue, it's going to be near impossible to figure out what the issue is or if there even is one in the standard library.

[galric]

yes, it only appears in some machines not all, and I do understand it is difficult to debug a problem unable to reproduce.

it seems I should learn 'how this assembly language works in runtime/asm_amd64.s' to debug this problem.

[mvdan]

In that case, try to rule out as many causes as you can. Have you tried 1.9rc2? What is the go env for the machine this happens on? Does it happen in multiple machines, or just that one?

[davecheney]

Have you tried building your program with the -race flag.

…

On 24 Aug 2017, at 21:34, Daniel Martí ***@***.***> wrote: In that case, try to rule out as many causes as you can. Have you tried 1.9rc2? What is the go env for the machine this happens on? Does it happen in multiple machines, or just that one? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[galric]

@mvdan I build binary on one machine and run it on others, only one machine has this problem. Shall I install go on this machine to see go env?

with the code below and try many times, i found this problem disappear if the origin string is short

binary build with this cmd also reproduce this problem @davecheney

go build -race test3.go && mv test3 test3-race

[admin@e38a07468 /home/admin/gix]
$./test3 -f /cloud/app/123 -o /cloud/app -n /cloud/data
FullString: /cloud/app/123
OldSub: /cloud/app
NewSub: /cloud/data
replace: /cloud/data/123

[admin@e38a07468 /home/admin/gix]
$./test3 -f /cloud/app/1234 -o /cloud/app -n /cloud/data
FullString: /cloud/app/1234
OldSub: /cloud/app
NewSub: /cloud/data
replace: /cloud/app/1234

[admin@e38a07468 /home/admin/gix]
$./test3-race -f /cloud/app/1234567890 -o /cloud/app -n /cloud/data
FullString: /cloud/app/1234567890
OldSub: /cloud/app
NewSub: /cloud/data
replace: /cloud/app/1234567890

code:

package main

import (
    "flag"
    "fmt"
    "strings"
)

func main() {
    fullString := flag.String("f", "", "") 
    oldSub := flag.String("o", "", "") 
    newSub := flag.String("n", "", "") 
    flag.Parse()

    fmt.Printf("FullString: %s\n", *fullString)
    fmt.Printf("OldSub: %s\n", *oldSub)
    fmt.Printf("NewSub: %s\n", *newSub)

    fmt.Printf("replace: %s\n", strings.Replace(*fullString, *oldSub, *newSub, 1)) 

}

go env on the build machine

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/admin/work/tj16/go/ext:/home/admin/work/tj16/go"
GORACE=""
GOROOT="/home/admin/work/tj16/go/go"
GOTOOLDIR="/home/admin/work/tj16/go/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-I /home/admin/work/tj16/go/ext/include -I /home/admin/work/tj16/cpp/ext/amd64/include -I /home/admin/work/tj16/cpp/src"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-L /home/admin/work/tj16/go/ext/lib -L /home/admin/work/tj16/cpp/ext/amd64/lib -L /home/admin/work/tj16/build/amd64/lib"

[TocarIP]

Could you execute
i r xmm1 , i r and x/16x $rdi just after first occurrence of 1991 PCMPESTRI $0x0c, (DI), X1 in gdb?
This will show content of needle (in xmm1 register), memory (rdi derefernce) and content of all registers.
First comparison should be successful, but it's not

[galric]

gdb message of reproduce problem

(gdb) r -f /cloud/app/1234567890##########12345678901234567890 -o /cloud/app -n /cloud/data
Starting program: /home/admin/gix/test3 -f /cloud/app/1234567890##########12345678901234567890 -o /cloud/app -n /cloud
/data
[New LWP 1464]
[New LWP 1465]
[New LWP 1466]
[New LWP 1467]
FullString: /cloud/app/1234567890##########12345678901234567890
OldSub: /cloud/app
NewSub: /cloud/data
Thread 1 hit Breakpoint 1, strings.Index (s="/cloud/app/1234567890##########12345678901234567890", sep="/cloud/app", 
    ~r2=0) at /home/admin/work/tj16/go/go/src/strings/strings_amd64.go:42
42                              return indexShortStr(s, sep)
(gdb) s
strings.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1743
1743            MOVQ s+0(FP), DI
(gdb) n
1745            MOVQ s_len+8(FP), DX
(gdb) n
1746            MOVQ c+16(FP), BP
(gdb) n
1747            MOVQ c_len+24(FP), AX
(gdb) n
1748            MOVQ DI, R10
(gdb) n
1749            LEAQ ret+32(FP), R11
(gdb) n
1750            JMP  runtime·indexShortStr(SB)
(gdb) n
runtime.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1767
1767            CMPQ AX, DX
(gdb) n
1768            JA fail
(gdb) n
1769            CMPQ DX, $16
(gdb) n
1770            JAE sse42
(gdb) n
1970            MOVL runtime·cpuid_ecx(SB), CX
(gdb) n
1971            ANDL $0x100000, CX
(gdb) n
1972            JZ no_sse42
(gdb) n
1973            CMPQ AX, $12
(gdb) n
1978            JAE _9_or_more
(gdb) n
1979            LEAQ 16(BP), SI
(gdb) n
1980            TESTW $0xff0, SI
(gdb) n
1981            JEQ no_sse42
(gdb) n
1982            MOVOU (BP), X1
(gdb) n
1983            LEAQ -15(DI)(DX*1), SI
(gdb) n
1984            MOVQ $16, R9
(gdb) n
1985            SUBQ AX, R9 // We advance by 16-len(sep) each iteration, so precalculate it into R9
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) i r xmm1
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {
    0x2f, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x61, 0x70, 0x70, 0x0, 0x2d, 0x6e, 0x0, 0x2f, 0x63}, v8_int16 = {
    0x632f, 0x6f6c, 0x6475, 0x612f, 0x7070, 0x2d00, 0x6e, 0x632f}, v4_int32 = {0x6f6c632f, 0x612f6475, 0x2d007070, 
    0x632f006e}, v2_int64 = {0x612f64756f6c632f, 0x632f006e2d007070}, uint128 = 0x632f006e2d007070612f64756f6c632f}
(gdb) i r
rax            0xa      10
rbx            0x7fffffffeb2a   140737488349994
rcx            0x10     16
rdx            0x33     51
rsi            0x7fffffffeb17   140737488349975
rdi            0x7fffffffeaf3   140737488349939
rbp            0x7fffffffeb2a   0x7fffffffeb2a
rsp            0xc42005dd90     0xc42005dd90
r8             0x7fffffffeaf3   140737488349939
r9             0x6      6
r10            0x7fffffffeaf3   140737488349939
r11            0xc42005ddb8     842350845368
r12            0x40     64
r13            0x30     48
r14            0x2f0    752
r15            0x200    512
rip            0x44dc45 0x44dc45 <runtime.indexShortStr+725>
eflags         0x282    [ SF IF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x63     99
gs             0x0      0
(gdb) x/16x $rdi
0x7fffffffeaf3: 0x6f6c632f      0x612f6475      0x312f7070      0x35343332
0x7fffffffeb03: 0x39383736      0x23232330      0x23232323      0x31232323
0x7fffffffeb13: 0x35343332      0x39383736      0x33323130      0x37363534
0x7fffffffeb23: 0x00303938      0x2f006f2d      0x756f6c63      0x70612f64
(gdb) c
Continuing.
replace: /cloud/app/1234567890##########12345678901234567890

gdb message of not reproduce message

(gdb) r -f /cloud/app/1234567890##########12345678901234567890 -o /cloud/app -n /cloud/data
Starting program: /home/admin/gix/test3 -f /cloud/app/1234567890##########12345678901234567890 -o /cloud/app -n /cloud
/data
[New LWP 24455]
[New LWP 24456]
[New LWP 24457]
[New LWP 24458]
FullString: /cloud/app/1234567890##########12345678901234567890
OldSub: /cloud/app
NewSub: /cloud/data
Thread 1 hit Breakpoint 1, strings.Index (s="/cloud/app/1234567890##########12345678901234567890", sep="/cloud/app", 
    ~r2=0) at /home/admin/work/tj16/go/go/src/strings/strings_amd64.go:42
42                              return indexShortStr(s, sep)
(gdb) s
strings.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1743
1743            MOVQ s+0(FP), DI
(gdb) n
1745            MOVQ s_len+8(FP), DX
(gdb) n
1746            MOVQ c+16(FP), BP
(gdb) n
1747            MOVQ c_len+24(FP), AX
(gdb) n
1748            MOVQ DI, R10
(gdb) n
1749            LEAQ ret+32(FP), R11
(gdb) n
1750            JMP  runtime·indexShortStr(SB)
(gdb) n
runtime.indexShortStr () at /home/admin/work/tj16/go/go/src/runtime/asm_amd64.s:1767
1767            CMPQ AX, DX
(gdb) n
1768            JA fail
(gdb) n
1769            CMPQ DX, $16
(gdb) n
1770            JAE sse42
(gdb) n
1970            MOVL runtime·cpuid_ecx(SB), CX
(gdb) n
1971            ANDL $0x100000, CX
(gdb) n
1972            JZ no_sse42
(gdb) n
1973            CMPQ AX, $12
(gdb) n
1978            JAE _9_or_more
(gdb) n
1979            LEAQ 16(BP), SI
(gdb) n
1980            TESTW $0xff0, SI
(gdb) n
1981            JEQ no_sse42
(gdb) n
1982            MOVOU (BP), X1
(gdb) n
1983            LEAQ -15(DI)(DX*1), SI
(gdb) n
1984            MOVQ $16, R9
(gdb) n
1985            SUBQ AX, R9 // We advance by 16-len(sep) each iteration, so precalculate it into R9
(gdb) n
1991            PCMPESTRI $0x0c, (DI), X1
(gdb) n
1995            CMPQ CX, R9
(gdb) i r xmm1
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {
    0x2f, 0x63, 0x6c, 0x6f, 0x75, 0x64, 0x2f, 0x61, 0x70, 0x70, 0x0, 0x2d, 0x6e, 0x0, 0x2f, 0x63}, v8_int16 = {
    0x632f, 0x6f6c, 0x6475, 0x612f, 0x7070, 0x2d00, 0x6e, 0x632f}, v4_int32 = {0x6f6c632f, 0x612f6475, 0x2d007070, 
    0x632f006e}, v2_int64 = {0x612f64756f6c632f, 0x632f006e2d007070}, uint128 = 0x632f006e2d007070612f64756f6c632f}
(gdb) i r
rax            0xa      10
rbx            0x7fffffffeb2a   140737488349994
rcx            0x0      0
rdx            0x33     51
rsi            0x7fffffffeb17   140737488349975
rdi            0x7fffffffeaf3   140737488349939
rbp            0x7fffffffeb2a   0x7fffffffeb2a
rsp            0xc42005dd90     0xc42005dd90
r8             0x7fffffffeaf3   140737488349939
r9             0x6      6
r10            0x7fffffffeaf3   140737488349939
r11            0xc42005ddb8     842350845368
r12            0x40     64
r13            0x30     48
r14            0x2f0    752
r15            0x200    512
rip            0x44dc45 0x44dc45 <runtime.indexShortStr+725>
eflags         0xa83    [ CF SF IF OF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x63     99
gs             0x0      0
(gdb) x/16x $rdi
0x7fffffffeaf3: 0x6f6c632f      0x612f6475      0x312f7070      0x35343332
0x7fffffffeb03: 0x39383736      0x23232330      0x23232323      0x31232323
0x7fffffffeb13: 0x35343332      0x39383736      0x33323130      0x37363534
0x7fffffffeb23: 0x00303938      0x2f006f2d      0x756f6c63      0x70612f64
(gdb) c
Continuing.
Thread 1 hit Breakpoint 1, strings.Index (s="/1234567890##########12345678901234567890", sep="/cloud/app", ~r2=0)
    at /home/admin/work/tj16/go/go/src/strings/strings_amd64.go:42
42                              return indexShortStr(s, sep)
(gdb) c
Continuing.
Thread 1 hit Breakpoint 1, strings.Index (s="/cloud/app/1234567890##########12345678901234567890", sep="/cloud/app", 
    ~r2=52) at /home/admin/work/tj16/go/go/src/strings/strings_amd64.go:42
42                              return indexShortStr(s, sep)
(gdb) c
Continuing.
replace: /cloud/data/1234567890##########12345678901234567890

diff

rcx            0x10     16
eflags         0x282    [ SF IF ]

rcx            0x0      0 
eflags         0xa83    [ CF SF IF OF ] 

[TocarIP]

This is really strange. All inputs to PCMPESTRI (xmm1,memory,rax,rdx,rdi) are the same, yet result is different. Just to rule out crazy idea, can you use disassemble 'runtime.indexShortStr' in gdb to verify that code is really identical in both cases , i suspect that binary was somehow corrupted?

[galric]

I find the issue only appear on cpu9 of this machine, using this blew code.

$sudo mkdir /cgroup_cpuset
$sudo mount -t cgroup cpuset cpuset /cgroup_cpuset
$sudo mkdir  /cgroup_cpuset/test
$sudo sh -c "echo 9 >> /cgroup_cpuset/test/cpuset.cpus"
$sudo sh -c "echo 0 >> /cgroup_cpuset/test/cpuset.mems"
$cat wrapper 
echo $$
sudo sh -c "echo $$ >> /cgroup_cpuset/test/tasks"
cat /proc/$$/cgroup
cat /cgroup_cpuset/test/cpuset.cpus
./test3 -f /cloud/app/12345678/123 -o /cloud/app -n /cloud/data

I tried many times, and all results were the same

$sh wrapper 
743
2:cpuset:/test
1:memory:/
9
FullString: /cloud/app/12345678/123
OldSub: /cloud/app
NewSub: /cloud/data
replace: /cloud/app/12345678/123

Other cpus are good.

Dose any other c(c++) function also use this 'PCMPESTRI', so that i can prove it is the matter of cpu9? I tried 'string.find' and it works well on this cpu