os/exec: removes trailing slash in args required for rsync --exclude

[NuLL3rr0r]

What version of Go are you using (go version)?

$ go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mamadou/.cache/go-build"
GOENV="/home/mamadou/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mamadou/dev/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mamadou/dev/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD="/home/mamadou/dev/sgum-packager/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3272390416=/tmp/go-build -gno-record-gcc-switches"

What did you do?

OK, here is the issue. Rsync is a very sensitive program with lots of gotchas. If I try the following, it copies all the files except the ones that I excluded:

/usr/bin/rsync \
    --checksum \
    --delete-after \
    --ignore-errors \
    --ignore-times \
    --links \
    --perms \
    --recursive \
    --verbose \
    --exclude="/.git" \
    --exclude="/.clang-format" \
    --exclude="/.editorconfig" \
    --exclude="/.gitattributes" \
    --exclude="/.gitignore" \
    --exclude="/LICENSE-THIRD-PARTY.md" \
    --exclude="/LICENSE.md" \
    --exclude="/README.md" \
    /tmp/Plugin-Unreal/Repo/5.1/ \
    /tmp/Plugin-Unreal/Stage/5.1/

Now, if I remove the trailing slashes from the last two parameters, it copies everything and ignores all the exclusion list:

/usr/bin/rsync \
    --checksum \
    --delete-after \
    --ignore-errors \
    --ignore-times \
    --links \
    --perms \
    --recursive \
    --verbose \
    --exclude="/.git" \
    --exclude="/.clang-format" \
    --exclude="/.editorconfig" \
    --exclude="/.gitattributes" \
    --exclude="/.gitignore" \
    --exclude="/LICENSE-THIRD-PARTY.md" \
    --exclude="/LICENSE.md" \
    --exclude="/README.md" \
    /tmp/Plugin-Unreal/Repo/5.1 \
    /tmp/Plugin-Unreal/Stage/5.1

So, in such a use case that trailing slash is required, otherwise, the command is the same as a simple cp command. Now, I try to reproduce the first command in go using the following program:

package main

import (
	"bytes"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
)

func run(command string, arguments []string) (int, error) {
	cmd := exec.Command(command, arguments...)

	var stdoutBuffer bytes.Buffer
	var stderrBuffer bytes.Buffer

	cmd.Stdout = io.MultiWriter(os.Stdout, &stdoutBuffer)
	cmd.Stderr = io.MultiWriter(os.Stderr, &stderrBuffer)
	exitCode := 0

	err := cmd.Run()
	if err != nil {
		exitError, ok := err.(*exec.ExitError)
		if ok {
			exitCode = exitError.ExitCode()
		}

		fmt.Println(err)
		return exitCode, errors.New(fmt.Sprintf("Failed to run '%s'!", command))
	}

	exitError, ok := err.(*exec.ExitError)
	if ok {
		exitCode = exitError.ExitCode()
	}

	return exitCode, nil
}

func main() {
	run("/usr/bin/rsync",
		[]string{
			"--checksum",
			"--delete-after",
			"--ignore-errors",
			"--ignore-times",
			"--links",
                        "--perms",
			"--recursive",
			"--verbose",
			"--exclude=\"/.git\"",
			"--exclude=\"/.clang-format\"",
			"--exclude=\"/.editorconfig\"",
			"--exclude=\"/.gitattributes\"",
			"--exclude=\"/.gitignore\"",
			"--exclude=\"/LICENSE-THIRD-PARTY.md\"",
			"--exclude=\"/LICENSE.md\"",
			"--exclude=\"/README.md\"",
			"/tmp/Plugin-Unreal/Repo/5.1/",
			"/tmp/Plugin-Unreal/Stage/5.1/",
		})
}

But, it behaves like the second command which ignores the exclusion list. I realized os.exec() tries to be smart and removes those trailing slashes. This is a terrible idea! How do I know? I change these two lines from:

			"/tmp/Plugin-Unreal/Repo/5.1/",
			"/tmp/Plugin-Unreal/Stage/5.1/",

To

			"/tmp/Plugin-Unreal/Repo/5.1\\/",
			"/tmp/Plugin-Unreal/Stage/5.1\\/",

And when I run the program it fails with the following error from Rsync, which proves go removes those trailing slashes:

rsync: [sender] change_dir "/tmp/Plugin-Unreal/Repo/5.1\" failed: No such file or directory (2)

Wondering how can I keep those trailing slashes?

What did you expect to see?

The rsync command should work just like the command line.

What did you see instead?

Go removes those trailing slashes in the arg list.

[ianlancetaylor]

Thanks for including the test case. I ran your program, but I changed "/usr/bin/rsync" to "/usr/bin/echo". The output was

--checksum --delete-after --ignore-errors --ignore-times --links --perms --recursive --verbose --exclude="/.git" --exclude="/.clang-format" --exclude="/.editorconfig" --exclude="/.gitattributes" --exclude="/.gitignore" --exclude="/LICENSE-THIRD-PARTY.md" --exclude="/LICENSE.md" --exclude="/README.md" /tmp/Plugin-Unreal/Repo/5.1/ /tmp/Plugin-Unreal/Stage/5.1/

The os/exec package is not removing trailing slashes. Why would it? I don't know what is wrong with rsync, but it's not due to the os/exec package.

[rittneje]

@NuLL3rr0r I'm not sure if this is the issue, but putting quotes around parameters is a shell thing. Since you are directly invoking rsync you should not include them. Otherwise rsync itself will see them, and may be interpreting it as a path starting with " (which doesn't exist). Try changing it to:

		[]string{
			"--checksum",
			"--delete-after",
			"--ignore-errors",
			"--ignore-times",
			"--links",
                        "--perms",
			"--recursive",
			"--verbose",
			"--exclude=/.git",
			"--exclude=/.clang-format",
			"--exclude=/.editorconfig",
			"--exclude=/.gitattributes",
			"--exclude=/.gitignore",
			"--exclude=/LICENSE-THIRD-PARTY.md",
			"--exclude=/LICENSE.md",
			"--exclude=/README.md",
			"/tmp/Plugin-Unreal/Repo/5.1/",
			"/tmp/Plugin-Unreal/Stage/5.1/",
		}

[NuLL3rr0r]

@rittneje thank you very much! Your suggestion worked! I jumped to the conclusion too early.