Odilon is an Open Source Object Storage that runs on standard hardware (Odilon project website).
It was designed as a redundant and secure file storage for applications that need to manage medium to large size objects (like pdfs, photos, audio, video).
It has a simple single-level folder structure similar to the Bucket / Object model of Amazon S3. It is small and easy to integrate, offers encryption, data protection and fault tolerance (software RAID and Erasure Codes) and detection of silent data degradation. Odilon also supports version control and master - standby replication over the Internet.
- Scalable Object Storage on commodity disks
- Single binary, does not need Database or other external service
- Developed in Java, the server requires Java 17+ (uses Spring Boot, OkHttp, Jackson, Caffeine, Metrics, among others)
- Runs on Linux and Windows
- SDK Java 11+ for client applications
- HTTP/S for client server communication
- License Open Source Apache 2. It can be used for Open Source and commercial projects
- Encryption at rest (AES 256)
- Integration with Key Management Server Hashicorp Vault
- Simple operation. Adding new disks requires one line in the config file, and an async process sets up disks and replicata data in background
- Data replication using Erasure Coding and software RAID
- Data immutability. Odilon supports two storage modes that protect data from deletion, whether accidental or intentional: Read Only and WORM (Write Once Read Many)
- Master - Standby architecture with async replication over the web, for disaster recovery, high availability, archival, ransomware recovery
- Version Control
- Tolerates full disk failures
- Disk monitoring for silent and slow data degradation detection (bit rot detection)
Odilon keeps objects encrypted (Encryption at Rest) using modern algorithms such as AES-256. Each object has a unique encryption key. In turn, the encryption key of the object can be generated by Odilon or, which is recommended for greater security, by a Key Management Server (KMS)
A KMS is software for generating, distributing, and managing cryptographic keys. It includes back-end functionality for key generation, distribution, and replacement. Moving key management to KMS prevents application reverse engineering attacks, simplifies operational maintenance, and compliance with security policies and regulations.
Odilon integrates with the KMS Open Source Hashicorp Vault.
Odilon can be configured to use software RAID for data replication. The supported configurations are
- RAID 0. Two or more disks are combined to form a volume, which appears as a single virtual drive.
It is not a configuration with data replication, its function is to provide greater storage and performance by allowing access to the disks in parallel.
- RAID 1.For each object, 1 or more exact copies (or mirrors) are created on two or more disks. This provides redundancy in case of disk failure. At least 2 disks are required, Odilon also supports 3 or more for greater redundancy.
- RAID 6 / Erasure Coding.
It is a method of encoding data into blocks that can be distributed across multiple disks or nodes and then reconstructed from a subset of those blocks. It has great flexibility since you can adjust the number and size of the blocks and the minimum required for recovery. It uses less disk space than RAID 1 and can withstand multiple full disk failures. Odilon implements this architecture using Reed Solomon error-correction codes. The configurations are:
3 disks (2 data 1 parity, supports 1 full disk failure),
6 disks (4 data 2 parity, supports up to 2 full disks failures)
12 disks (8 data 4 parity, supports up to 4 full disks failures)
Odilon supports Master - Standby Architecture for disaster recovery, high availability, archival, and anti-ransomware protection. Data replication is done asynchronously using HTTP/S over the local network or the Internet. Setting up a standby server is simple, just add the URL and credentials to the master configuration file and restart.
Odilon will propagate each operation to the standby server. It will also run a replication process in background for data existing before connecting the standby server.
- Odilon is not a Distributed Storage like Cassandra, Hadoop etc.
- Odilon is not a File System like GlusterFS, Ceph, ext4, etc.
It uses the underlying file system to stores objects as encrypted files, or in some configurations to break objects into chunks.
- Odilon is not a NoSQL database like MongoDB, CouchDB, etc.
It does not use a database engine, Odilon uses its own journaling agent for Transaction Management and only supports very simple queries, ie. to retrieve an object and to list the objects of a bucket filtered by objectname's prefix.
- Odilon API is not fully S3 compatible
Odilon API is simpler than S3. The only thing it has in common with AWS S3 it that uses the bucket/object methafor to organize the object space.
- Odilon is not optimized for a very large number of small files
Odilon does not have optimization for lots of small files. The files are simply stored encrypted and compressed to local disks. Plus the extra meta file and shards for erasure coding.
Odilon supports master-standby architecture for archival, backup and data protection, but it is not a Distributed Storage and it does not support active-active replication.
A Java client program that interacts with the Odilon server must include the Odilon SDK jar in the classpath. A typical architecture for a Web Application is
Example to upload 2 pdf files:
String endpoint = "http://localhost";
/** default port */
int port = 9234;
/** default credentials */
String accessKey = "odilon";
String secretKey = "odilon";
String bucketName = "demo_bucket";
String objectName1 = "demo_object1";
String objectName2 = "demo_object2";
File file1 = new File("test1.pdf");
File file2 = new File("test2.pdf");
/* put two objects in the bucket,
the bucket must exist before sending the object,
and object names must be unique for that bucket */
OdilonClient client = new ODClient(endpoint, port, accessKey, secretKey);
client.putObject(bucketName, objectName1, file1);
client.putObject(bucketName, objectName2, file2);
More info in Odilon's website
Java Application Development with Odilon
- Installation, Configuration and Operation on Linux
- Installation, Configuration and Operation on Windows
- Advanced configuration (encryption, master-standby, version control)
- Java Application Development with Odilon
- Odilon SDK Javadoc
- odilon-client GitHub project
- odilon-model GitHub project