MD5Dumper is a tool that:
- Can generate a canonical representation of every file under a directory. (the Fsmap format)
- Can minify Fsmap representations to reduce interchanged data size.
- Can summarize Fsmap to reduce interchanged data size for initial checksum.
- Compare minified Fsmap and local directory Fsmap, and detect missing/corrupted/extra files
MD5Dumper aims to automatise file system synchronization between distinct ends. Intended specially for auto-patch servers.
Due to the way auto-patch servers interact with clients, data traffic on each check is really important to optimize. MD5Dumper attempts to minimize the size of data interchanged between those checks.
An Fsmap is a canonical representation of a directory. It is generated by recursively traversing a directory in a sorted way and creating MD5 hashes of every file's content + file's relative path.
declare type Fsmap = FsmapEntry[];
declare type FsmapEntry = {
index: number; // Index to ensure traverse order
path?: string; // Relative file system path
md5: string; // h(relativePath + content)
};
- Once development is done, store built artifacts under a directory. (Let's call it
./target
for this example) - Generate fsmap of
./target
dir and store it under./autopatcher/build.fsmap
. This file will be used to deserialize on the server side, client won't need this one. - Minify fsmap of
./target
and store it under./autopatcher/build.min.fsmap
- Summarize fsmap of
./target
and store it under./autopatcher/summary.md5
- Serve those last 2 files for clients to fetch;
build.min.fsmap
andsummary.md5
. (They will need them to perform checksum :p) - Fsmap maps Indices to relative file paths. Serve files and their relative paths by indices using the fsmap as a lookup table! Easy peasy!
- Generate fsmap of the local directory
- Fetch
summary.md5
from the server and compare with local fsmap's summary. - If they match, then the client is up to date! Else, fetch
build.min.fsmap
from the server. - Check for the differences between
build.min.fsmap
and local fsmap. The diff functionality of MD5Dumper will yield "invalid local files" and "missing file indices". - Delete invalid local files.
- Request every missing file by their indices. Server shall know which file is represented by the indice! Server will respond you with the file blob and it's relative path!
- The client should be up to date after all the steps! Easy peasy!