encoding/csv: BOM presents in fields read with Reader

[tp]

I was checking the header fields of an external CSV file and noticed, that the file BOM is part of the first field when reading with csv.Reader.

package main

import (
    "bytes"
    "encoding/csv"
    "fmt"
)

func main() {
    csvData := []byte("\uFEFFa,b")

    r := bytes.NewReader(csvData)

    csvR := csv.NewReader(r)

    header, err := csvR.Read()

    if err != nil {
        fmt.Println(err.Error())
        return
    }

    fmt.Printf("%q", header[0]) // prints "\ufeffa" where I expected "a"
}

snippet on playground

Related: Since U+FEFF is called a "[...] space", I was expected string.TrimSpace to remove it, which it did not. (Which would have been my preferable work-around to remove and spaces around fields). I would guess this is also the reason why csvR.TrimLeadingSpace = true does not remove the BOM.

[ianlancetaylor]

The BOM is a bizarre idea in general, and it makes absolutely no sense when using UTF-8. It's not appropriate for encoding/csv to do anything special with a BOM. If you have to deal with it, deal it with before passing your reader to encoding/csv. If you have a file that is not UTF-8, you will to use a translating reader anyhow, as encoding/csv, like all Go code, expects UTF-8.

While it's true that U+FEFF is a space, the UTF-8 representation of U+FEFF is not the literal bytes FEFF.