Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

A simple Redis RDB file parser for Java

Build Status

Overview

A simple Java library for parsing Redis RDB files.

This library does the minimal amount of work to read entries (e.g. a new DB selector, or a key/value pair with an expiry) from an RDB file, mostly limiting itself to returning byte arrays or lists of byte arrays for keys and values. The caller is responsible for application-level decisions such as how to interpret the contents of the returned byte arrays or what types of objects to instantiate from them.

For example, sorted sets and hashes are parsed as a flat list of value/score pairs and key/value pairs, respectively. Simple Redis values are parsed as a singleton. As expected, Redis lists and sets are parsed as lists of values.

Furthermore, this library performs lazy decoding of the packed encodings (ZipMap, ZipList, Hashmap as ZipList, Sorted Set as ZipList, Intset, and QuickList) such that those are only decoded when needed. This allows the caller to efficiently skip over these entries or defer their decoding to a worker thread.

RDB files created by all versions of Redis through 5.0.x are supported (i.e., RDB versions 1 through 9). Some features, however, are not supported:

  • Modules, introduced in RDB version 8
  • Streams, introduced in RDB version 9.

If you need these, please open an issue or a pull request.

To use this library, including the following dependency in your pom.xml.

<dependency>
    <groupId>net.whitbeck</groupId>
    <artifactId>rdb-parser</artifactId>
    <version>1.5.1</version>
</dependency>

Javadocs are available at javadoc.io/doc/net.whitbeck/rdb-parser/.

Example usage

Let's begin by creating a new Redis RDB dump file.

Start a server in the background, connect a client to it, and flush all existing data.

$ redis-server &
$ redis-cli
127.0.0.1:6379> flushall

Now let's create some data structures. Let's start with a simple key/value pair with an expiry.

127.0.0.1:6379> set foo bar
127.0.0.1:6379> expire foo 3600

Then let's create a small hash and a sorted set.

127.0.0.1:6379> hset myhash field1 val1
127.0.0.1:6379> hset myhash field2 val2
127.0.0.1:6379> zadd myset 1 one 2 two 2.5 two-point-five

Finally, let's save the dump to disk. This will create a dump.rdb file in the current directory.

127.0.0.1:6379> save
127.0.0.1:6379> exit
$ killall redis-server

Now let's see how to parse the dump.rdb file from Java.

import java.io.File;
import net.whitbeck.rdbparser.*;

public class RdbFilePrinter {

  public static void printRdbFile(File file) throws Exception {
    try (RdbParser parser = new RdbParser(file)) {
      Entry e;
      while ((e = parser.readNext()) != null) {
        switch (e.getType()) {

        case DB_SELECT:
          System.out.println("Processing DB: " + ((DbSelect)e).getId());
          System.out.println("------------");
          break;

        case EOF:
          System.out.print("End of file. Checksum: ");
          for (byte b : ((Eof)e).getChecksum()) {
            System.out.print(String.format("%02x", b & 0xff));
          }
          System.out.println();
          System.out.println("------------");
          break;

        case KEY_VALUE_PAIR:
          System.out.println("Key value pair");
          KeyValuePair kvp = (KeyValuePair)e;
          System.out.println("Key: " + new String(kvp.getKey(), "ASCII"));
          if (kvp.hasExpiry()) {
            System.out.println("Expiry (ms): " + kvp.getExpiryMillis());
          }
          System.out.println("Value type: " + kvp.getValueType());
          System.out.print("Values: ");
          for (byte[] val : kvp.getValues()) {
            System.out.print(new String(val, "ASCII") + " ");
          }
          System.out.println();
          System.out.println("------------");
          break;
        }
      }
    }
  }
}

Call this function on the dump.rdb file. The output will look like:

Processing DB: 0
------------
Key value pair
Key: myset
Value type: SORTED_SET_AS_ZIPLIST
Values: one 1 two 2 two-point-five 2.5
------------
Key value pair
Key: myhash
Value type: HASHMAP_AS_ZIPLIST
Values: field1 val1 field2 val2
------------
Key value pair
Key: foo
Expiry (ms): 1451518660934
Value type: VALUE
Values: bar
------------
End of file. Checksum: 157e40ad49ef13f6
------------

References

As of October 2019, the most recent RDB format version is 9. The source of truth is the rdb.h file in the Redis repo. The following resources provide a good overview of the RDB format up to version 7.

About

A simple Redis RDB file parser for Java

Resources

License

Packages

No packages published