Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/encoding/unicode: add examples #14133

TomOnTime opened this issue Jan 28, 2016 · 1 comment


None yet
3 participants
Copy link

commented Jan 28, 2016's documentation is a bit unclear to the newbie like myself that just wants to read files that Windows folks send them without having to become an expert in UTF-16. I think the docs make sense to someone that already understands transforms, unicode, etc, but not everyone should need to know such plumbing. All I wanted was something like ioutil.ReadFile() that automagically read MS-Windows UTF-16 files and gave me UTF-8.

An example like this would have helped:

func NewReader(rd io.Reader) io.Reader {
    // Make an tranformer that decodes MS-Windows (16LE) UTF files:
    winutf := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
    // Make a transformer that is like winutf, but abides by BOM if found:
    decoder := winutf.NewDecoder()
    // Make a Reader that uses decoder:
    return transform.NewReader(rd, unicode.BOMOverride(decoder))

fd, _ := os.Open(filename)
r := NewReader(fd)  // Read from "r" to get UTF-8.
utf8, _ := ioutil.ReadAll(r)
text := string(utf8)

It would be awesome if that example (or one like it) was added to the documentation. I'd be glad to submit a PR.

Furthermore, it would be useful to have a library similar to ioutil but automagically detects UTF-16 is found. I've made an example here:

@ianlancetaylor ianlancetaylor changed the title Add examples to's x/text/encoding/unicode: add examples Jan 28, 2016

@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jan 28, 2016


This comment has been minimized.

Copy link
Contributor Author

commented Jan 29, 2016

P.S. I think the example might be clearer if the func was inlined.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.