Skip to content
Utilities to make it easier to use golang.org/x/text/encoding/unicode
Branch: master
Clone or download
malexdev and TomOnTime Added the ability to close file handles (#1)
* Updated to be a readcloser while reading.
Still need to implement ScanCloser and uncomment NewScanner().
* Update readme.
* Add test case
* Implement scancloser, tests soon
* Improved docs and test for scanner
* All changes should be done, updating readme
* Update to address review feedback:
Latest commit 09c4100 May 11, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
catutf Added the ability to close file handles (#1) May 11, 2018
testdata First draft. Jan 28, 2016
.gitignore .gitignore binary Jan 30, 2016
LICENSE Initial commit Jan 28, 2016
README.md Improve comments. Nov 3, 2017
utfutil.go Added the ability to close file handles (#1) May 11, 2018
utfutil_test.go Added the ability to close file handles (#1) May 11, 2018

README.md

utfutil

Utilities to make it easier to read text encoded as UTF-16.

Dealing with UTF-16 files you receive from Windows.

Have you encountered this situation? Code that has worked for years suddenly breaks. It turns out someone tried to use it with a file that came from a MS-Windows system. Now this perfectly good code stops working. Looking at a hex dump you realize every other byte is \0. WTF? No, UTF. More specifically UTF-16LE with an optional BOM.

What does all that mean? Well, first you should read "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky.

Now you understand what the problem is, but how do you fix it? Well, you can spend a week trying to figure out how to use golang.org/x/text/encoding/unicode and you'll be able to decode UTF-16LE files. (No offense to the authors of that module. It is a fantastic module but if you aren't already an expert in Unicode encoding, it is pretty difficult to use.)

If you don't have a week, you can just use this module. Take the easy way out! Just change ioutil.ReadFile() to utfutil.ReadFile(). Everything will just work.

The goal of utfutl is to provide replacement functions that magically do the right thing. There is a demo program that shows how to use it called catutf.

utfutil.ReadFile() is the equivalent of ioutil.ReadFile()

OLD: Works with UTF8 and ASCII files:

		data, err := ioutil.ReadFile(filename)

NEW: Works if someone gives you a Windows UTF-16LE file occasionally but normally you are processing UTF8 files:

		data, err := utfutil.ReadFile(filename, utfutil.UTF8)

utfutil.OpenFile() is the equivalent of os.Open().

OLD: Works with UTF8 and ASCII files:

		data, err := os.Open(filename)

NEW: Works if someone gives you a file with a BOM:

		data, err := utfutil.OpenFile(filename, utfutil.HTML5)

utfutil.NewScanner() is for reading files line-by-line

It works like os.Open():

		s, err := utfutil.NewScanner(filename, utfutil.HTML5)

Encoding hints:

What's that second argument all about? utfutil.UTF8? utfutil.HTML5?

If a file has no BOM, it is impossible to guess the file encoding with 100% accuracy. Therefore, the 2nd parameter is an "EncodingHint" that specifies what to assume for BOM-less files.

UTF8        No BOM?  Assume UTF-8
UTF16LE     No BOM?  Assume UTF 16 Little Endian
UTF16BE     No BOM?  Assume UTF 16 Big Endian
WINDOWS = UTF16LE   (i.e. a reasonable guess if file is from MS-Windows)
POSIX   = UTF8      (i.e. a reasonable guess if file is from Unix or Unix-like systems)
HTML5   = UTF8      (i.e. a reasonable guess if file is from the web)

Future Directions

If someone writes a golang equivalent of uchatdet, I'll add a hint called "AUTO" which uses it. That would be awesome. Volunteers?

You can’t perform that action at this time.