os/exec: Cmd adds random slashes to paths on run

[dogancelik]

What version of Go are you using (go version)?

go version go1.6 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\dogan\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GO15VENDOREXPERIMENT=1
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1

What did you do?

Run 7za (7-zip) and extract files

package main

import (
    "fmt"
    "os/exec"
    "strings"
)

func main() {
    zipcmd := `7za x "%s" -aoa -r -o"%s" *.xml`
    zippath := "C:\\test.zip"
    outputpath := "C:\\testfolder\\"

    cmdStr := fmt.Sprintf(zipcmd, zippath, outputpath)
    cmdSlice := strings.Fields(cmdStr)
    fmt.Println("command:", cmdStr)

    cmd := exec.Command(cmdSlice[0], cmdSlice[1:]...)
    fmt.Println("cmd.Args:", cmd.Args)

    cmdoutput, err := cmd.CombinedOutput()
    fmt.Println("output:", string(cmdoutput), err)
}

What did you expect to see?

dogan@DOGAN-PC E:\Projects\Go\test
$ 7za x "C:\test.zip" -aoa -r -o"C:\testfolder\" *.xml

7-Zip (a) [32] 15.07 beta : Copyright (c) 1999-2015 Igor Pavlov : 2015-09-17

Scanning the drive for archives:
1 file, 6225 bytes (7 KiB)

Extracting archive: C:\test.zip
--
Path = C:\test.zip
Type = zip
Physical Size = 6225

Everything is Ok

Files: 4
Size:       9014
Compressed: 6225

What did you see instead?

dogan@DOGAN-PC E:\Projects\Go\test
$ go run test.go
cmdStr: 7za x "C:\test.zip" -aoa -r -o"C:\testfolder\" *.xml
cmd.Args: [7za x "C:\test.zip" -aoa -r -o"C:\testfolder\" *.xml]
output:
7-Zip (a) [32] 15.07 beta : Copyright (c) 1999-2015 Igor Pavlov : 2015-09-17

Scanning the drive for archives:

ERROR: The filename, directory name, or volume label syntax is incorrect.
\C:\test.zip



System ERROR:
The filename, directory name, or volume label syntax is incorrect.
 exit status 2   

cmd.CombinedOutput(), cmd.Output(), and cmd.Run() adds / prefixes \ to C:\test.zip

Here's example test.zip file: test.zip

I can't do put space between -o and "C:\testfolder" because 7-Zip doesn't work that way.

Second test, same error, this time, it puts \ at -o argument:

cmd := exec.Command("7za", "x", "C:\\test.zip", "-aoa", "-r", `-o"C:\testfolder\"`, "*.xml")
fmt.Println("cmd.Args:", cmd.Args)

cmdoutput, err := cmd.CombinedOutput()
fmt.Println("output:", string(cmdoutput), err)

dogan@DOGAN-PC E:\Projects\Go\test
$ go run test.go
cmd.Args: [7za x C:\test.zip -aoa -r -o"C:\testfolder\" *.xml]
output:
7-Zip (a) [32] 15.07 beta : Copyright (c) 1999-2015 Igor Pavlov : 2015-09-17

Scanning the drive for archives:
1 file, 3725 bytes (4 KiB)

Extracting archive: C:\test.zip
--
Path = C:\test.zip
Type = zip
Physical Size = 3725



ERROR:
Can not create output directory: \C:\testfolder\\\


System ERROR:
The filename, directory name, or volume label syntax is incorrect.
 exit status 2

[alexbrainman]

exec.Cmd does inserts a lot of back slashes (and other characters) when it builds command line on windows. See https://github.com/golang/go/blob/master/src/syscall/exec_windows.go#L84 for details. But I don't think these back slashes are random. That is what Microsoft recomends we do (see a link in the code).

I think your case fails because exec.Cmd tries hard to incorporate " into paths and other parameters, and 7za is confused by these. I think you're trying too hard, tyr this (not tested):

package main

import (
    "fmt"
    "os/exec"
)

func main() {
    zippath := "C:\\test.zip"
    outputpath := "C:\\testfolder\\"

    cmd := exec.Command("7za", "x", zippath, "-aoa", "-r", "-o"+outputpath, "*.xml")
    fmt.Printf("cmd.Args: %q\n", cmd.Args)

    cmdoutput, err := cmd.CombinedOutput()
    fmt.Println("output:", string(cmdoutput), err)
}

Alex

[dogancelik]

That works, fortunately 7-Zip supports "-oC:\Path with spaces\", otherwise this would be impossible to do if a program didn't support it but only supported -o"...". Another workaround would be using Batch or Bash files but that's dependent on the operating system.

[alexbrainman]

... fortunately 7-Zip supports "-oC:\Path with spaces", otherwise this would be impossible to do if a program didn't support it but only supported -o"...".

I don't understand. Are you saying that my suggestion above will not work if output path contains space? But I think it will just work too.

Try this

package main

import (
    "fmt"
    "os/exec"
)

func main() {
    zippath := "C:\\test.zip"
    outputpath := "C:\\Path with spaces\\"

    cmd := exec.Command("7za", "x", zippath, "-aoa", "-r", "-o"+outputpath, "*.xml")
    fmt.Printf("cmd.Args: %q\n", cmd.Args)

    cmdoutput, err := cmd.CombinedOutput()
    fmt.Println("output:", string(cmdoutput), err)
}

Alex

[dogancelik]

Your example works fine, I'm talking about a program that does not support "-o..." but only supports -o"..."

[alexbrainman]

I'm talking about a program that does not support "-o..." but only supports -o"..."

I still don't understand what you're after. But, perhaps, that is not important. Should we close this issue as "exec.Command is working as intended"?

Alex

[dogancelik]

Let's assume there is a program that accepts a parameter -o, it only accepts this parameter if -o is in this format:

someprogram -o"………"

It does not accept:

someprogram "-o………"

So someprogram user can't do this in shell: someprogram "-oC:\Folder with space\"
But can do this: someprogram -o"C:\Folder with space\"

_This is how this program behaves, it is coded this way_, it does not allow quotes dash o path quotes because the programmer doesn't use a good parsing method. Programmer coded their program so only it accepts values if parameter meets this format: dash o quotes path quotes

Windows doesn't have any problem and executes this command someprogram -o"C:\Folder with space\ without any errors. When you execute the exact command in Go, it escapes things inaccurately and adds \ in front of C:\Folder with space\ thus command fails.

someprogram has to use -o"..." format to work, because someprogram -o"C:\Folder with space\" works in Command Prompt, it should work in Go too:

cmd := exec.Command("someprogram", `-o"C:\Folder with space\"`)
cmd.Run() // doesn't work, path becomes \C:\Folder with space\, program fails

[alexbrainman]

Thank you for explaining. Unfortunately everything you say is just you guessing about how system works. Why don't you provide a real example of a problem?

Alex

[bradfitz]

You can't use fmt.Sprintf followed by strings.Fields and pass the space-separated split to exec.Command and expect it to work when you have arguments containing spaces. That's just very broken, since that doesn't match how your shell (cmd.exe) is splitting your arguments.

Let's move this to the mailing list since this isn't a bug as far as I can see.

[alexbrainman]

Since we are talking about cmd.exe, http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx explains how you pass parameters to it. Not for fainthearted.

Alex