Estoraje

Key Value distributed database

Estoraje is the simplest distributed system for key-value storage. It is temporary consistent -but quite close to hard consistency-, high available, lightweight, scalable and gives a good performance.

You just need a load balancer on the top and your estoraje's nodes. No external service mesh coordination (like Consul or Zookeeper) is needed. It uses a consistent hashing algorithm to distribute and replicate the data among the nodes and an embed etcd server for coordination.

Quick start

There is a docker compose configuration and a Makefile for development.

• Running a cluster

make start

Take a look at other options running make.

Now you should be able to use estoraje. Using it is quite simple:

# Add a key/value
curl -X POST -d "{value}" http://localhost:8000/{key}

# Read a value
curl http://localhost:8000/{key}

# Delete key/value
curl -X DELETE http://localhost:8000/{key}

Benchmarking

For a production-like cluster (keep in mind estoraje is not ready for production at all):

You can run a three nodes server this way

1. Install estoraje on each node as any other go app.

2. Run estoraje

# Node 1
estoraje -name=node_1 \
	-initialCluster=node_1=https://n1.satur.io:2380,node_2=https://n2.satur.io:2380,node_3=https://n3.satur.io:2380 \
	-host=n1.satur.io \
	-port=8001 \
	-dataPath=data
	
# Node 2
estoraje -name=node_2 \
	-initialCluster=node_1=https://n1.satur.io:2380,node_2=https://n2.satur.io:2380,node_3=https://n3.satur.io:2380 \
	-host=n2.satur.io \
	-port=8001 \
	-dataPath=data
	
# Node 3
estoraje -name=node_3 \
	-initialCluster=node_1=https://n1.satur.io:2380,node_2=https://n2.satur.io:2380,node_3=https://n3.satur.io:2380 \
	-host=n3.satur.io \
	-port=8001 \
	-dataPath=data

You should also have a load balancer. You could install caddy and just run as reverse proxy load balancer

caddy reverse-proxy --from estoraje.satur.io --to n1.satur.io --to n2.satur.io --to n3.satur.io

To add a new node

# Node 4
estorage -name=node_4 \
	--add
	-initialCluster=node_1=https://n1.satur.io:2380,node_2=https://n2.satur.io:2380,node_3=https://n3.satur.io:2380,node_4=https://n4.satur.io:2380
	-host=n3.satur.io
	-port=8001
	-dataPath=data

Remember to update also the load balancer adding the new node or use the /_nodes_discovery or the /_nodes_discovery_plain endpoint for dinamic config.

Description

This project is developed for self training purposes. My main goal is building a simple although working system from scratch trying to avoid as much as possible using third parties packages. It is inspired by some other existing and more complex products and follows some well-known approaches for building data intensive applications. For more info in this topic I encourage recommend Martin Kleppmann's Designing Data-Intensive Applications book. Most of the code is only in two files: One for the consistent hashing algorithm and other one with less than 1000 lines that contains the whole logic.

Architecture

We actually need two different features to make our system functional: On one side, we need to decide on an approach to distribute the data among the nodes, and on the other side, we must coordinate these nodes. This could be done in some different ways: master-slave, consensus algorithm... To make a decision is essential to know what are we expecting from our system: High availability? Large storage? Real-time? Consistency? So, to simplify, we are assuming some outlines our use-case:

• All nodes should have the same responsibility and be, as far as possible, identical. Just one source code for each piece.
• We expect more reading than writing. Also, we expect that a handful of keys are requested more times than the other ones.
• It should be fast. Reading faster than Writing faster than Deleting.
• Temporary consistency is enough.
• We want to add and removes nodes with no downtime.
• Avoid using third-parties software.
• Most important, it should be as simple as possible.

Taking in mind the acceptance criteria, this is the approach:

• Use consistent hashing to distribute the data among the nodes.
• Use a hard consistency system to coordinate the nodes. In our case, we have an embed etcd server as a sidecar on each node.

Also, keeping simplicity in mind, we need to decide how to store the data in the node. We resolved this in the simplest way: we create a new file for each key. By doing this, we delegate all indexing and caching work to the file system, which should function adequately in modern operating systems. This approach has some clear weaknesses, but it also has a main strength: it is clearly the simplest.

Why all logic in one file?

Almost all the code is in one file, main.go. Why?

This is a way to force myself to keep the system simple! If you can use only one file -and you don't want to go crazy- all non-essential code will be removed, and you won't develop unwanted features.

Finally, we have less than 800 lines and no plans to make many changes: the main goal -learning- was reached. The code is also more accessible this way, at least you can understand how are implemented most of the core concepts just taking a look for some minutes at the one-file.

My learning goals:

• Distributed systems and data intensive applications
• gRPC
• Go basics

References:

• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems
• Etcd
• Minikeyvalue
• Consistent hashing
• Profiling Go