A simple and fast tool to parse the AOF file of redis

Document ：中文 / English

Content

• Content
• 1、Introduction
  • (1) Features
  • (2) Architecture
• 2、Build
• 3、Usage
  • (1) The input file
  • (2) The output file
  • (3) Example
• 4、Performance
  • (1) Testing

1、Introduction

A simple and fast tool to parse the AOF file of redis

(1) Features

• Code is clean, simple and easy to customize
• Speed up parsing through multiple goroutines
• A list of commands will be generated after parsing for log querying

(2) Architecture

2、Build

git clone https://github.com/WGrape/parseAOF
cd parseAOF
go mod download
make build

3、Usage

Run the binary under bin dir parseAOF_<os>_<arch> with the path of the aof file

./bin/parseAOF_macos_arm64 -i ~/Download/appendonly.aof -r 8
./bin/parseAOF_macos_arm64 -h
parse redis aof to readable

Usage:
  parseAOF [flags]

Flags:
  -h, --help            help for parseAOF
  -i, --input string    input AOF file path
  -o, --output string   output dir path
  -r, --routines int    max goroutines (default 8)

(1) The input file

Here's an example input file ./data/appendonly.aof for you to test

Before running, pass the path of the aof file to the start.sh script, the content is as follows

*2
$6
SELECT
$1
0
... ...

(2) The output file

Here's an example output file ./data/aof.merged for you to test

After the parsing is complete, the file aof.merged will be generated in the directory of data, the content is as follows

--------------------parseAOF | version=0.5.0--------------------
SELECT 0 
set key1 1 
set key2 2 
set key3 3 
sadd key4 1 2 3 4 
lpush key5 1 2 3 4 5 
zadd key6 1 2 3 4 5 6 

(3) Example

4、Performance

• The average speed to parse is 50000 lines/s
• The maximum size of the aof is 1GB

(1) Testing

Id	Lines	Size	Cost	CPU
1	1,2301,117	39MB	3m50s	<=65%
2	3,435,263	13MB	1m12s	<=65%
3	357,850	8.6MB	3.47s	<=113%