Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Goal of PR
When the scope is specified, file writes are not permitted to overwrite existing files. A file rename is performed using a historical naming scheme:
Note: this is using the popular and fast nanoid lib to generate a 10 char mixed-case-alphanumeric string rather than an actual uuid/guid.
File deletes simply perform the same rename and nothing more. This results in the original file 404ing on request like a regular delete.
This historical naming scheme differs from the current collections proposal in that it uses unix timestamps (milliseconds since epoch) rather than a contiguous incrementing version integer.
This has the benefit of not depending on an index file and the additional read->write overhead, and does not require additional locks or expensive index file conflict checking & resolution.
However, this does present the usual edge-cases with using timestamps:
If a timestamp cannot be used then I have a couple alternative proposals that also avoid bucket-wide index file race conditions and conflict resolution:
Per-file version tracking
Use a per-file historical version tracker rather than a bucket-wide index file. Something like:
The storage overhead of an additional small file seems equal to or better than the storage overhead of an index file containing a map of file paths and version integers.
Rename with GUID
Use a randomly generated ID rather than an incrementing integer or timestamp. Future file history restoration tooling could use the cloud driver provided
Requires crafting a
Full test coverage of new logic.
@@ Coverage Diff @@ ## develop #248 +/- ## ========================================== + Coverage 80.08% 80.68% +0.6% ========================================== Files 21 21 Lines 1486 1553 +67 Branches 264 271 +7 ========================================== + Hits 1190 1253 +63 - Misses 221 225 +4 Partials 75 75
Speaking to this topic, you can avoid the need for a dedicated index file while simultaneously avoiding any clock-drift problems by using both a timestamp and a GUID in the
I'm in favor of this approach paired with a nonce (or a random integer):
As unlikely as an overlap would be with a random integer, a local incrementing nonce would eliminate the possibility of overwrite entirely (and could be transient -- it doesn't need to live past a ms). I'm not sure how much that matters though -- if we used a random integer, the possibility of overwrite is very low.
* feature/file-mutex: Update driver model README with a notice about path mutex. Add docs for performRead to driver model definition in README Add docs for performRename and performStat to driver model definition in README Update blockstack.js dep out of beta version Simplify mutx with Set<string> Implement decodeTokenForPayload helper function Update to blockstack.js beta package to fix typing collisions with bitcoinjs-lib v5 Add tests for schema validation warnings Add additional config param schema descriptions, cleanup sample configs Config schema validation that logs console warnings # Conflicts: # hub/src/server/authentication.ts
* develop: Add `list-files` http endpoint documentation to readme Update DriverModel in readme Minor version bump with changelog entries for recent changes. Update with file rename function changes (merge conflict #2) Add docs for listFilesStat to driver model definition in README Integration tests for listFiles with stat metadata Option to return file stat info in listFiles # Conflicts: # hub/src/server/server.ts
The Gaia hub shouldn't be making that many back-end requests on list-files. Instead, it should return a sentinel value for files that shouldn't be listed, and blockstack.js should filter these sentinel values out in