File Storage Engine

File Storage Engine (FSEngine) is a write-optimized object storage library designed mainly for storing large, continuous streams on commodity hardware and Disk-Drives. This project includes APIs and primitives for storing raw data and fetching data from the storage. Use-cases of this library include storing live video streams, logs, metadata, or any write-optimized workload.

This library might be used with ArchiverMedia wrapper library for easily transforming media objects into consumable binary packets for FSEngine.

This library uses a pseudo log storage model for storing multiple data streams into a single file. It also supports a circular storage model for a fixed-sized store.

Currently, FSEngine has been offered as a Golang only library, and all APIs are usable within any Golang 1.12+ project.

Features

• Write-optimized storage
• Local storage
• Circular storage
• Multi-instance storage
• Spare storage with ECC

Installing

1. To install FSEngine library into and existing project, having Go installed, use the bellow Go command to install FSEngine.

$ go get github.com/fanap-infra/fsEngine

2. Import it to your library

import "github.com/fanap-infra/fsEngine"

How to use

First you need to establish storage location on local system and initialize storage engine by maximum size of storage used by the engine:

package main
import (
	"github.com/fanap-infra/fsEngine"
	"github.com/fanap-infra/log"
)
type EventsTrigger struct {
	fileID uint32
}
func main() {
	// define storage location and size
	const storagePath = "/var/fsEngine/volume1"
	const storageSize = 1 << 32 // 4GB volume
	fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
		fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
	if err != nil {
		log.Fatal(err)
	}
	
	return
}
// VirtualFileDeleted 
// Implement event listener.
func (e EventsTrigger) VirtualFileDeleted(fileID uint32, message string) {
	log.Warnf("File with ID:%v, deleted: %s", fileID, message)
	return
}

This is the barebone definition for using FSEngine, however, to store objects you have to create a virtual identity for the object which here is called a virtualFile.

	fileSystem, err := fsEngine.CreateFileSystem(storagePath, storageSize,
		fsEngine.BLOCKSIZE, &EventsTrigger{}, log.GetScope("Example"))
	if err != nil {
		log.Fatal(err)
	}
    const vfID = 1
    const vfName = "Hello"
    virtualFile, err := fileSystem.NewVirtualFile(vfID, vfName)
    if err != nil {
        log.Fatal(err)
    }
    _, err = virtualFile.Write([]byte("HelloStorage"))
    if err != nil {
        log.Fatal(err)
    }
    err = virtualFile.Close()
	if err != nil {
        log.Fatal(err)
    }