x/text/encoding/charmap: windows1252 encoding not decoding properly

[gdm85]

What version of Go are you using (go version)?

go version go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

Did not test with 1.11; I think this is not relevant because the charmap package is external?

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build857292294=/tmp/go-build -gno-record-gcc-switches"

Standard linux amd64 env.

What did you do?

Conversion of a Windows1252 text to UTF-8 using golang.org/x/text/encoding/charmap gives an incorrect result; iconv (both the Linux CLI and the library used via CGO) work as expected.

I have added a test here to reproduce this problem: https://github.com/gdm85/go-iconv/blob/charmap-test/charmap_test.go

You need to initialise submodules on that repo after clone, so:

git clone -b charmap-test https://github.com/gdm85/go-iconv.git
cd go-iconv
git submodule update --init --recursive
go test -v .

What did you expect to see?

Conversion happening same as iconv does.

What did you see instead?

=== RUN   TestCharmapWindows1252Conversion
--- FAIL: TestCharmapWindows1252Conversion (0.00s)
	charmap_test.go:21: expected "Jaime De Armiñán y Héctor Alterio" but got "Jaime De Armiñán y Héctor Alterio" instead
=== RUN   TestIconvWindows1252Conversion
--- PASS: TestIconvWindows1252Conversion (0.00s)
=== RUN   TestIconv
--- PASS: TestIconv (0.00s)
=== RUN   TestIconvReverse
--- PASS: TestIconvReverse (0.00s)
=== RUN   TestError
--- PASS: TestError (0.00s)
FAIL
FAIL	github.com/gdm85/go-iconv	0.002s

[agnivade]

/cc @mpvl

[mpvl]

The test you provided converts UTF-8 to Win 1252, instead of the inverse. So replacing NewDecoder with NewEncoder in the test case you provided to reproduce it fixes it.

It is indeed confusing, as your input test string looks like some non-UTF-8 encoding, but it is. It is probably the UTF-8 encoding of a string that looks like a Win 1252 encoding.