Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

A "tail -f" like file watcher for log indexing in Go

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 example
Octocat-spinner-32 .gitignore
Octocat-spinner-32 LICENSE
Octocat-spinner-32 README.md
Octocat-spinner-32 err.go
Octocat-spinner-32 indexer.go
Octocat-spinner-32 log_file.go
Octocat-spinner-32 watcher.go
Octocat-spinner-32 windex.go
README.md

Windex: Watch and Index

Windex is a library for concurrently watching log files and indexing their data.

A Watcher provides cross-platform file system notifications via github.com/howeyc/fsnotify. These events help keep track of changes to the size of a file over time. This information is stored, and data from the tail of the file is sent via a channel to an external source, known as anIndexer.

The default Indexer flushes to stdout, effectively making windex a programatically accessible tail -f in its default state. However it can be much more powerful than that.

Indexer is an interface that contains two methods:

type Indexer interface {
    Parse(chan []byte) ([]byte, error)
    Flush([]byte) error
}

The stdout indexer does nothing with Parse(chan []byte), and simply prints with Flush([]byte). Since Parse has access to the data, it will be a powerful step toward implementing Indexers which can parse and flush data to any number of a variety of external sources: Redis, Postgres, Statsd, etc.

Here's an example naive implementation of a Redis Indexer:

type RedisIndexer struct {
    address string
}

func (i *RedisIndexer) Parse(log_data chan []byte) (parsed_log_data []byte, err error) {
    parsed_log_data = <-log_data
    return parsed_log_data, nil
}

func (i *RedisIndexer) Flush(parsed_log_data []byte) (err error) {
    redis, err := redis.Dial("tcp", i.address)

    if err != nil {
        log.Fatal(err)
    }

    redis.Do("lpush", "errorz", parsed_log_data)

    log.Print("Data Pushed: ", len(parsed_log_data), " bytes")

    return
}

func NewRedisIndexer(address string) (stdout *RedisIndexer) {
    return &RedisIndexer{
        address: address,
    }
}

Usage

// Use the windex generator to get a windex instance
windex, err = windex.New("logfile01.log")

// Launch the goroutine to watch our file
go windex.Watch()

// Launch the goroutine to index our data. The default indexer
// flushes data to stdout.
go windex.Index()

// Call select on our exit channel. This will let us keep
// everything moving and in the future may be of more use.
select {
case exit := <-windex.Exit:
    if exit {
        return
    }
}

Examples

The above is paraphrased from the example program in example/stdoutindexer/example.go. There is an example that uses the above Redis Indexer in example/redisindexer/example.go

TODO

  • Bring back the tests
  • Add wcld like Indexer

Credits

windex is (c) Michael R. Bernstein, 2012

License

windex is distributed under the MIT License, see LICENSE file for details.

Something went wrong with that request. Please try again.