New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/text/encoding/unicode: add examples #14133

Open
TomOnTime opened this Issue Jan 28, 2016 · 1 comment

Comments

Projects
None yet
3 participants
@TomOnTime
Copy link
Contributor

TomOnTime commented Jan 28, 2016

golang.org/x/text/encoding/unicode's documentation is a bit unclear to the newbie like myself that just wants to read files that Windows folks send them without having to become an expert in UTF-16. I think the docs make sense to someone that already understands transforms, unicode, etc, but not everyone should need to know such plumbing. All I wanted was something like ioutil.ReadFile() that automagically read MS-Windows UTF-16 files and gave me UTF-8.

An example like this would have helped:

func NewReader(rd io.Reader) io.Reader {
    // Make an tranformer that decodes MS-Windows (16LE) UTF files:
    winutf := unicode.UTF16(unicode.LittleEndian, unicode.IgnoreBOM)
    // Make a transformer that is like winutf, but abides by BOM if found:
    decoder := winutf.NewDecoder()
    // Make a Reader that uses decoder:
    return transform.NewReader(rd, unicode.BOMOverride(decoder))
}

fd, _ := os.Open(filename)
r := NewReader(fd)  // Read from "r" to get UTF-8.
utf8, _ := ioutil.ReadAll(r)
text := string(utf8)

It would be awesome if that example (or one like it) was added to the documentation. I'd be glad to submit a PR.

Furthermore, it would be useful to have a library similar to ioutil but automagically detects UTF-16 is found. I've made an example here: https://github.com/TomOnTime/utfutil

@ianlancetaylor ianlancetaylor changed the title Add examples to golang.org/x/text/encoding/unicode's x/text/encoding/unicode: add examples Jan 28, 2016

@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jan 28, 2016

@TomOnTime

This comment has been minimized.

Copy link
Contributor

TomOnTime commented Jan 29, 2016

P.S. I think the example might be clearer if the func was inlined.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment