[FEAT] Rotating Write Ahead Logs

[AlecDivito]

Overview

WALs at some point need to be rotated. Rotation can happen from the following events.

• The WAL reaches a maximum size limit (ex. 5mb)
• The WAL reaches a maximum of rows written (ex. 1000 rows)
• A Schema is closed (Flushed)

When one of these events happen, we are required to rotate the WAL file

Rotation

When I say rotation, I mean we need to archive the WAL file and create a new one where all of our new writes will go. When doing this action, we also need to record it inside of our MANIFEST file system.

Deletion

Although a WAL is archived, it does not mean it's deleted. A WAL can only be deleted once all Schemas that have written into that WAL have been FLUSHED to disk.

For example, if you have Schema A, B and C in a WAL, and then you flush A, we would do the following steps:

1. Create a new file that would act as the new WAL
2. Mark the current WAL as "archived" and the new WAL as the WAL. Save this to the MANIFEST system
3. Flush the Schema A Memtable to disk as a SST file
4. Continue writing to the new WAL

This archived WAL will stay active in the system until B and C are also flushed, at which point, the WAL can be marked for deletion and cleaned up at some point.

All these actions must be recorded to the manifest file system.

TODO

1. Configure each scenario and test them our using a virtual file system
2. Test file rotation, make sure