Transform is a Go package that provides a simple pattern for performing chainable data transformations on streams of bytes. It conforms to the io.Reader interface and is useful for operations such as converting data formats, audio/video resampling, image transforms, log filters, regex line matching, etc.
The transutil package provides few examples that work with JSON such as JSONToMsgPack
, MsgPackToJSON
, JSONToPrettyJSON
, JSONToUglyJSON
, JSONToProtoBuf
, and ProtoBufToJSON
. It also includes a handy Gzipper
and Gunzipper
.
To start using Transform, install Go and run go get
:
$ go get -u github.com/tidwall/transform
Below are a few very simple examples of custom transformers.
Convert a string to uppper case. Unicode aware. In this example we only process one rune at a time.
func ToUpper(r io.Reader) io.Reader {
br := bufio.NewReader(r)
return transform.NewTransformer(func() ([]byte, error) {
c, _, err := br.ReadRune()
if err != nil {
return nil, err
}
return []byte(strings.ToUpper(string([]rune{c}))), nil
})
}
msg := "Hello World"
data, err := ioutil.ReadAll(ToUpper(bytes.NewBufferString(msg)))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
Output:
HELLO WORLD
The Rot13 cipher.
func Rot13(r io.Reader) io.Reader {
buf := make([]byte, 256)
return transform.NewTransformer(func() ([]byte, error) {
n, err := r.Read(buf)
if err != nil {
return nil, err
}
for i := 0; i < n; i++ {
if buf[i] >= 'a' && buf[i] <= 'z' {
buf[i] = ((buf[i] - 'a' + 13) % 26) + 'a'
} else if buf[i] >= 'A' && buf[i] <= 'Z' {
buf[i] = ((buf[i] - 'A' + 13) % 26) + 'A'
}
}
return buf[:n], nil
})
}
msg := "Hello World"
data, err := ioutil.ReadAll(Rot13(bytes.NewBufferString(msg)))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
Output:
Uryyb Jbeyq
A line reader that filters lines that match on a RegExp pattern.
func LineMatch(r io.Reader, pattern string) io.Reader {
br := bufio.NewReader(r)
return NewTransformer(func() ([]byte, error) {
for {
line, err := br.ReadBytes('\n')
matched, _ := regexp.Match(pattern, line)
if matched {
return line, err
}
if err != nil {
return nil, err
}
}
})
}
logs := `
23 Apr 17:32:23.604 [INFO] DB loaded in 0.551 seconds
23 Apr 17:32:23.605 [WARN] Disk space is low
23 Apr 17:32:23.054 [INFO] Server started on port 7812
23 Apr 17:32:23.141 [INFO] Ready for connections
`
data, err := ioutil.ReadAll(LineMatch(bytes.NewBufferString(logs), "WARN"))
if err != nil {
log.Fatal(err)
}
fmt.Println(string(data))
Output:
23 Apr 17:32:23.605 [WARN] Disk space is low
A line reader that trims the spaces from all lines.
func LineTrimSpace(r io.Reader, pattern string) io.Reader {
br := bufio.NewReader(r)
return transform.NewTransformer(func() ([]byte, error) {
for {
line, err := br.ReadBytes('\n')
if len(line) > 0 {
line = append(bytes.TrimSpace(line), '\n')
}
return line, err
}
})
}
phrases := " lacy timber \n"
phrases += "\t\thybrid gossiping\t\n"
phrases += " coy radioactivity\n"
phrases += "rocky arrow \n"
out, err := ioutil.ReadAll(LineTrimSpace(bytes.NewBufferString(phrases)))
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
Output:
lacy timber
hybrid gossiping
coy radioactivity
rocky arrow
A reader that matches lines on the letter 'o', trims the space from the lines, and transforms everything to upper case.
phrases := " lacy timber \n"
phrases += "\t\thybrid gossiping\t\n"
phrases += " coy radioactivity\n"
phrases += "rocky arrow \n"
r := ToUpper(LineTrimSpace(LineMatch(bytes.NewBufferString(phrases), "o")))
// Pass the string though the transformer.
out, err := ioutil.ReadAll(r)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
Output:
HYBRID GOSSIPING
COY RADIOACTIVITY
ROCKY ARROW
The github.com/tidwall/transform/transutil
package includes additional examples.
func Gunzipper(r io.Reader) io.Reader
func Gzipper(r io.Reader) io.Reader
func JSONToMsgPack(r io.Reader) io.Reader
func JSONToPrettyJSON(r io.Reader) io.Reader
func JSONToProtoBuf(r io.Reader, pb proto.Message, multimessage bool) io.Reader
func JSONToUglyJSON(r io.Reader) io.Reader
func MsgPackToJSON(r io.Reader) io.Reader
func ProtoBufToJSON(r io.Reader, pb proto.Message, multimessage bool) io.Reader
Josh Baker @tidwall
Transform source code is available under the ISC License.