Custom Go text token scanner implementation

Package scanner is a custom text scanner implementation. It has the same idiomatic Go scanner programming interface, and it lets the client to freely navigate the buffer. The scanner is also capable of peeking ahead of the cursor. Read runes are rendered as tokens with additional information on their position in the buffer. Consult the package documentation or see Usage to see how to use it.

Installation

Use the following command to add the package to an existing project.

go get github.com/mdm-code/scanner

Usage

Here is a snippet showing the basic usage of the scanner to read text as a stream of tokens using the public API of the scanner package.

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"

    "github.com/mdm-code/scanner"
)

func main() {
    r := bufio.NewReader(os.Stdin)
    s, err := scanner.New(r)
    if err != nil {
        log.Fatalln(err)
    }
    var ts []scanner.Token
    for s.Scan() {
        t := s.Token()
        ts = append(ts, t)
    }
    fmt.Println(ts)
}

Development

Consult Makefile to see how to format, examine code with go vet, run unit test, run code linter with golint in order to get test coverage and check if the package builds all right.

Remember to install golint before you try to run tests and test the build:

go install golang.org/x/lint/golint@latest

License

Copyright (c) 2023 Michał Adamczyk.

This project is licensed under the MIT license. See LICENSE for more details.