RoseDB - v1.2.0: Shana, The Star.

RoseDB: Shana, The Star.

The second major release of RoseDB after v1.1.0 which only lasted for 5 days. This version implements a lot of critical and important features for the application.

What's new?

1. RoseDB is now fully utilizing its own query format which we call Queria, examples are found on #11 and also on a section below here.
2. Authorization is now stronger than ever with implementation of both Layer 1 (Authorization header x Hashed Authorization) and also Layer 3 which is SSL SUPPORT
3. Fixes issue with a condition where items are being revived after having been dropped (this is due to < v1.1.0, dropping collections/database won't actually remove anything that was on enqueued to be added).
4. Configuration file is now beautified. #8
5. Fixes write issues with paths (specifically making directories).
6. Fixes issue with versioning where versions weren't being dumped properly (they were being dumped as literal folders... yes).
7. Development snapshot versions will no longer check for updates (since the API considers dev-snapshots as old versions).
8. Added internal locking which allows threads to communicate with one another properly over who is writing and who is reading.
9. Moved from io.javalin to TooTallNate's websocket server which SHOCKINGLY reduced the file size up to 50%.
10. Almost all methods on the source code are now documented, hope it will help future contributors.
11. Major improvements on write and read methods.
12. A major code organization was done.
13. Fixes issue with no value Queria requests not being accepted by server.
14. All additions to the database from now (and modifications) will be written down on a journal, and if the write to the journal fails then the transaction WILL FAILL and the client who sent the transaction will receive an error with the kode field at 0.
15. There is now a rate limit of 3 connections per 10 seconds for each client, this helps lower CPU usage from each connection since each connection will have to hash the Authorization field of the client entering and compare it with the hash of our Authorization.
16. The reads and writes buffer size has been reduced from 65536 to 8192, this change is still experimental and may change again in the future.
17. Adds to the journal has a timeout of 30 seconds, if the data is still not added to the journal by then, the transaction will be considered a failure and the future will be canceled.
18. Writes and reads have a retry limit of 500 attempts with a 50 millisecond delay between them, this retry limit involves when a Thread locks onto the file and also when a Thread writes onto a file, the timeout is at 25 seconds.
19. The journal is read every startup to ensure that all transactions were written properly, any transactions left inside the journal will be added before being removed from the records.
20. Writes are now synchronous for journals but for writes that are heading to the queue (and not the journal), they are done asynchronously.
21. Fixes issue with unique field being appended to value.
22. DELETE, DROP transactions by the time when sent to the client has already been executed (may cause performance degradation but durability > performance).
23. Additional fixes to ListenerManager and some listeners that didn't guarantee completion of the request transaction.
24. pom.xml will now build RoseDB.jar instead of RoseDB-v{version}-jar-with-dependencies.jar.

How to use SSL?

To use SSL, there needs to be several conditions.

1. You must have a domain registered (for example, mihou.pw).
2. You must have an LetsEncrypt SSL certificate.

If you have all of them, please proceed to the next step which is to create an ssl.json file which looks like this:

{
  "pathToPEM": "/path/to/pem",
  "keyPassword": "SSL KEY PASSWORD"
}

After that, restart your RoseDB server and it should attempt to use SSL for the server.

What is Internal Locking?

Internal Locking is, simply, a fancy way of saying I AM WRITING THIS FILE, NO ONE SHOULD READ THIS FILE YET! or I AM READING THIS FILE, NO ONE SHOULD WRITE YET! which allows the threads to coordinate with one another over when to write and when to read, avoiding threads where the results are broken.

This does not prevent external processes from causing mayhem though, FileLocks is needed for that but the current implementation of write and read proves to be a challenge to make FileLocks for me.

Queria

What is Queria and how does it look like?

• Queria is the official query format of RoseDB which is extremely similar to MongoDB's query format in many ways which allows both users from RoseDB to be familiar with queries of MongoDB and also MongoDB users to be already familiar with the queries of RoseDB.

How does Queria look like?

database.collection.get(identifier, {"unique":"Unique value"})

A Queria request that does not need a unique field will look like this:

database.collection.get(identifier)

A Queria multiple field update/delete request will look like this:

database.collection.update(identifier, {"key":["key1","key2"],"value":["value1","value2"]})

A Queria add request will look like this:

database.collection.add(identifier, {"someKey":"someValue"})

In a sense, the format is database.collection.method(identifier, value), please note that value can also contain unique field which is a protected field that will be removed from the value during write to file and addition to cache. This unique field will be used for callbacks to the client.

Important notes

1. Queria will NOT AND NEVER SUPPORT DATABASES/DIRECTORIES THAT HAS . or , or ( or ) ON THEIR NAMES.
2. The reason for the change above is because we are using databases and collections with . as a protected RoseDB directory, for example, storing the hash token.
3. We will be dropping support for JSON requests on v1.3.0.

Notes about Queria

Queria is still experimental and will undergo several changes along the way but the basic concept is the same, all the value will always be in JSON since RoseDB will still store data as JSON files.

How to install?

You can quickly install RoseDB by downloading the jar file below and placing it on its own folder.

v1.2.0 DEVELOPMENT SNAPSHOT

What's NEW?

1. RoseDB is now utilizing its own query which is called Queria, examples are found on: #11
2. A little bit stronger authorization (layer 1), more information can be found on: #10
3. Fixes issue with a condition where items are being revived after having been dropped (because < v1.1.0, dropping collections/database won't actually remove anything that is on queue to be written).
4. Configuration file is now beautified: #8
5. Fixes some write issues with paths (especially with making directories).
6. Fixes issue with versioning where the versions aren't actually being dumped properly (they are being dumped as folders, yes literal folders).
7. Development snapshot version will no longer update check (since the update check API considers dev-snapshots as old versions).
8. All additions to the database from now (and modifications) will be written down on a journal, and if the write to the journal fails then the transaction WILL FAILL and the client who sent the transaction will receive an error with the kode field at 0.
9. There is now a rate limit of 3 connections per 10 seconds for each client, this helps lower CPU usage from each connection since each connection will have to hash the Authorization field of the client entering and compare it with the hash of our Authorization.
10. The reads and writes buffer size has been reduced from 65536 to 8192, this change is still experimental and may change again in the future.
11. Adds to the journal has a timeout of 30 seconds, if the data is still not added to the journal by then, the transaction will be considered a failure and the future will be canceled.
12. Writes and reads have a retry limit of 500 attempts with a 50 millisecond delay between them, this retry limit involves when a Thread locks onto the file and also when a Thread writes onto a file, the timeout is at 25 seconds.
13. The journal is read every startup to ensure that all transactions were written properly, any transactions left inside the journal will be added before being removed from the records.
14. Writes are now synchronous for journals but for writes that are heading to the queue (and not the journal), they are done asynchronously.
15. More to be added.

Important notes

1. Queria will NOT AND NEVER SUPPORT DATABASES/DIRECTORIES THAT HAS . or , or ( or ) ON THEIR NAMES.
2. The reason for the change above is because we are using databases and collections with . as a protected RoseDB directory, for example, storing the hash token.
3. We will be dropping support for JSON requests on v1.3.0.

How does Queria look like?

A Queria request that has a unique field (which is used for callbacks) will look like this:

database.collection.get(identifier, {"unique":"Unique value"})

A Queria request that does not need a unique field will look like this:

database.collection.get(identifier)

A Queria multiple field update/delete request will look like this:

database.collection.update(identifier, {"key":["key1","key2"],"value":["value1","value2"]})

A Queria add request will look like this:

database.collection.update(identifier, {"someKey":"someValue"})

Notes about Queria

Queria is still experimental and will undergo several changes along the way but the basic concept is the same, all the value will always be in JSON since RoseDB will still store data as JSON files.

Stage 2 of v1.2.0: What's New?

• Added Internal Locking, this way threads won't mess up with one another.
• Moved to TooTallNate's Java Websocket which SHOCKINGLY reduced the file size up to 50%.
• Almost all methods and files should now be documented to help anyone understand the code better.
• Added potential SSL support (please test it out first, instructions below).
• Improved write and read methods.
• A more organized code.
• Fixed issue with no value Queria queries not being received by server.

How to use SSL?

To use SSL, there needs to be several conditions.

1. You must have a domain registered (for example, mihou.pw).
2. You must have an LetsEncrypt SSL certificate.

If you have all of them, please proceed to the next step which is to create an ssl.json file which looks like this:

{
  "pathToPEM": "/path/to/pem.pem",
  "keyPassword": "SSL KEY PASSWORD"
}

After that, restart your RoseDB server and it should attempt to use SSL for the server.

How does Internal Locking work?

Internal Locking is simply a fancy way of saying 'I am writing this file, NO ONE SHOULD READ YET!' or 'I am reading this file, NO ONE SHOULD WRITE YET!' which basically allows threads to communicate when to write and when to read, this way the threads wouldn't randomly read a broken file.

This does not prevent external processes from causing mayhem though, FileLocks is needed for that but the current implementation of write and read proves to be a challenge to make FileLocks for me.

Is there any critical changes that I have to take note?

None, there are no breaking changes applied to this patch and everything should work as intended.

What to expect next?

1. POSSIBLE Adding support for instant write to disk.
2. WatchService which allows the application to detect any external changes to our files.
3. And a few more that will come to mind.

How to install the development snapshot version?

Please note that you will be installing a possibly bug-filled version of RoseDB, but Iwon't discourage you from installing this new version as I also want other testers to help find issues with the application.

To install, simply replace your RoseDB.jar file with the development snapshot jar.

Roses and Jewels

v1.1.0: Roses and Jewels

This is a major version change of RoseDB and will introduce major breaking changes (to the drivers), please read.

What's new

• File writing is now handled by schedulers (called every 5 seconds), will add an option to have it write immediately but take note that the code will naturally be slower.
• Add support for authorization code to be located on headers, will also add SSL support for websockets later.
• Moved from ConcurrentHashMap to Caffeine Cache since according to benchmarks it is faster and most likely handles thread-safety better than I do.
• Logging is now done through Logback.
• Colorful console.
• Revert feature is now available with the limitations specified on README.md
• Malfunctioning write is now fixed.
• BREAKING Authorization header is now enforced.
• Fixes the messy logs from Jetty (it should be cleaner now).

Limitations of Revert

• The versions are saved on application-level cache and is dumped when the application is closed normally and not abruptly.
• Each item is limited to one version and will be overriden each time an ADD or UPDATE request is sent that will override the item.

Revert Request

You can revert an add or update request by simply sending a revert request to the server which looks like:

{
  "method": "revert",
  "database": "rose_db",
  "collection": "mana",
  "idenitifer": "revert_test",
  "unique": "5"
}

The response of this request will be the last version of the file.

{
  "response": "{\"test\":1}",
  "kode": 1,
  "replyTo": "Unique ID here."
}

You can also disable this feature via the config for maybe a little performance boost but I doubt it would add much, disabling the feature
would then give out this reply to any attempt to revert:

{
  "response": "This method is unsupported by the server.",
  "kode": 0,
  "replyTo": "Unique ID here"
}

Another to note:

• The application will now start logging files directly onto a log folder [logs/] which will help in debugging issues.
• Also, errors from main thread will print out information such as: JDK Version, JDK Vendor, Memory Usage of RoseDB, Memory allocated to JVM, OS Architecture, OS Version, OS.
• Please create an issue for any bugs that you find with this release.

v1.1.0-DEV-SNAPSHOT

NOTE

This is a pre-release but the build file will be available since it needs some testing.

This build adds/improves the following.

• File writing is now handled by schedulers (called every 5 seconds), will add an option to have it write immediately but take note that the code will naturally be slower.
• Add support for authorization code to be located on headers, will also add SSL support for websockets later.
• Moved from ConcurrentHashMap to Caffeine Cache since according to benchmarks it is faster and most likely handles thread-safety better than I do.
• Logging is now done through Logback.
• Colorful console.
• Revert feature is now available with the limitations specified on README.md
• Malfunctioning write is now fixed.
• Authorization header is now enforced.
• Fixes the messy logs from Jetty (it should be cleaner now).

Another to note:

• The application will now start logging files directly onto a log folder [logs/] which will help in debugging issues.
• Also, errors from main thread will print out information such as: JDK Version, JDK Vendor, Memory Usage of RoseDB, Memory allocated to JVM, OS Architecture, OS Version, OS.

Petal 107 (v1.0.7)

Adds the following

• Ability to perform requests through the terminal.
• Compress all files with GZIP from now on.
• Fixes several bugs involving writes.
• Fixes issue with UPDATE only accepting STRING datatype.

Petal 105 (v1.0.5)

Patch notes

• Automatic update check (it won't download though, simply checks whether there is an update and notifies) every specific time set on config.
• Aggregation methods, for example, aggregating an entire collection or database.
• Focus on thread-safe reads and writes since current implementation does nothing about it.
• Pre-loading of all databases and collections.
• Configurable maximum message buffer & size.
• Create a thread before shutdown to finish writing everything.
• Customizable heartbeat interval.

Petal 103 (v1.0.3)

We recommend updating to this version immediately especially for Linux users and users who wants to send large amounts of data

Patch notes:

• Utilizes cross-platform slashes.
• Removes RoseEntity in favor of raw Strings.
• Uses NIO writes and reads.
• Configurable Core Count on config.json.
• Moved from loops to streams instead.
• Reads only one line of files (since JSONObject encodes them in one line, it'd be too much of an overhead; please do not change the files, and if ever you need to change, please don't forget to minify them into one line otherwise it would break).
• Utilizes .putIfAbsent() instead of the previous if blocks.
• Increased maximum message size and message buffer size to (1024 * 5000) or 5,120,000 bytes, if you want it to be increased, please write an Issue.
• Disabled websocket logging of connection, close and message for logging levels below debug.
• And a few more changes inside.

RoseDB - v1.0.1

This mini-patch fixes the following:

• ERROR Message that says File specified does not exist (during ADD and UPDATE requests).
• Fixes issue with RoseDB having 'null' as directory.
• Fixes issue with database and collection being mixed up.
• Better handling of Invalid JSON Requests (we don't want to cut them off from the websocket immediately).
• Allows deletion of multiple keys.
• Allows a more advanced update (addition of new keys and values also removal).

RoseDB - Basic

v1.0.0

Add README