Skip to content

logrange/range

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

91 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Go Report Card GoDoc

range

Range is persistent storage of streams of records. It is a horizontally-scalable, highly-available, blazinbly fast aggregation system of streams of records.

Range is designed to store unlimited number of sequentual records from thousands of sources. Due to its simplicity, range is very cost effective solution what makes it to be an effective tool for storing logging data, dozens of millions records per second. Range is used by logrange - distributed and extremely fast log aggregation system.

Range's highlights:

  • Aggregation system of streams of records
  • Sstand-alone and Distributed configurations are available
  • Streams size doesn't impact system performance
  • Range is able to work with hundreds of thousands of streams
  • Created for and support Security of various systems
  • Optimized for storing dozens of millions records per second

Embedding Range as stand-alone stream storage

Range could be used as a library. In this configuration the streams are persisted on local file-system. Records are available via journals - persisted streams of records. The following piece of code illustrates, how it works:

package main

import (
	"log"
	"context"
	"os"

	"github.com/logrange/range/embed"
	"github.com/logrange/range/pkg/records"
)

func main() {
	// Creating a new journal controller in os.Arg[1] directory
	jc, err := embed.NewJCtrlr(embed.JCtrlrConfig{JournalsDir: os.Arg[1]})
	if err != nil {
		log.Fatal("Could not create new controller, err=", err)
	}
	defer jc.Close()

	// Obtaining the journal by its name "teststream"
	j, err := jc.GetOrCreate(context.Background(), "teststream")
	if err != nil {
		log.Fatal("could not get new journal err=", err)
	}

	// Writes couple records "aaaa" and "bbbb" into the journal
	_, _, err = j.Write(context.Background(), records.SrtingsIterator("aaaa", "bbbb"))
	if err != nil {
		log.Fatal("Write err=", err)
	}

	// telling the journal to call fsync(), to be able to iterate over the fresh data right now
	j.Sync()

	// obtaining iterator to read the journal
	it := j.Iterator()
	r, _ := it.Get(context.Background())
	log.Println("First record ", string(r))

	it.Next(context.Background())
	r, _ = it.Get(context.Background())
	log.Println("Second record ", string(r))

	// jump to the first record again
	pos := it.Pos()
	pos.Idx = 0
	it.SetPos(pos)
	r, _ = it.Get(context.Background())
	log.Println("First recor ", string(r))

	// release resources
	it.Close()
}

Building range server

Connecting to the range server

License

This project is licensed under the Apache Version 2.0 License - see the LICENSE file for details

Acknowledgments

  • GoLand IDE by JetBrains is used for the code development

About

Persistent storage of streams of records

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages