-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tstore backend and plugin architecture. #1180
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukebp
changed the title
[WIP] tlogbe: Add tlogbe implementation.
[WIP] backend: Add tlogbe implementation.
Jul 31, 2020
Politeia's routes which are currently used in decrediton:
|
lukebp
changed the title
[WIP] backend: Add tlogbe implementation.
[WIP] multi: Add tlog implementation.
Sep 16, 2020
lukebp
changed the title
[WIP] multi: Add tlog implementation.
Add tstore backend and plugin architecture.
Mar 26, 2021
marcopeereboom
approved these changes
Mar 28, 2021
This was referenced Jul 9, 2021
This was referenced Sep 12, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This diff rewrites much of the politeia codebase. A summary of the major
changes are listed below.
politeiad tstore backend
A new backend was added to politeiad that is referred to as the tstore backend.
It combines a trillian log, commonly referred to as a tlog, and a key-value
store. The tstore backend differs from the git backend in three main ways.
Scalability
The git backend relied on git repos saved to the file system to store data.
This approach cannot be scaled out since it is constrained to the file system
of a single politeiad instances. The tstore backend moves the data into an
actual database, allowing for much greater scalability.
Separate the timestamps and the data blobs
The git backend would timestamp the git commit hashes onto the decred
blockchain. These commit hashes are merkle roots of all the files in the
commit. The timestamp and the data blobs in this setup are interconnected. If
you delete a data blob from the commit history, such as if you need to censor
a public comment, you destroy the ability to recreate the merkle path for
other data blobs included in that commit merkle root, thus destroying you're
ability to prove inclusion of data in a decred timestamp. Your only option is
to remove the data from the git repo in a new commit, but the censored data
remains in the git history. The tstore backend seperates out the timestamps
from the data blobs. A merkle tree of the data is stored in a tlog instance
while the data itself is stored in a key-value database. This means we can
delete data from the database, such as if you need to censor a public
comment, without impacting any of the timestamps or our ability to prove
the inclusion of a piece of data in a timestamp.
Ability to retrieve timestamp inclusion proofs
The tstore backend gives us the ability to retrieve an inclusion proof for
any piece of data in the backend, providing cryptographic proof that the data
blob was included in a dcr timestamp. There was no easy way to do this in the
git backend.
politeiad v2 API
A politeiad v2 API was added. This API is a simplified version of the politeiad
v1 API. The tstore backend relies solely on the politeiad v2 API.
politeiad plugin architecture
A formalized plugin architecture was added to the tstore backend.
A politeiad plugin extends a record with additional functionality. Some
examples of this include adding comments to a record, adding the ability to run
a dcr ticket vote on a record, and adding record validation that is specific to
the decred proposal system.
This plugin architecture allows politeiad to become a configurable system. If a
user wants to runs a politeiad instance that allows for specific file types and
record comments, they can do so by simply adjusting the politeiad
configuration. No new code is required unless they want to add in more
granular, application specific validation.
The end goal is to eventually allow plugin code to exist outside of the
politeia repository that can be dynamically loaded at runtime. While this is
not possible yet, this diff lays the foundation for doing this in the future.
politeiawww APIs
The politeiawww APIs have been rewritten to be generic, application agnostic
APIs. These APIs were formally tied to specific politeia applications such as
the decred proposal system and the decred contractor management system. You
would use a proposal route to submit new proposals and a separate invoice
route to submit cms invoices. These application specific routes have been
replaced with more generic APIs. For example, there now exists a single set
of routes for submitting all records, regardless of whether that record is a
proposal or an invoice. The application specific validation has been pushed
into the politeiad plugins. The validation that is specific to proposals is
now performed in a proposal system politeiad plugin.
The politeiawww APIs now map to the politeiad plugins. For example, the
politeiad comments plugin has a corresponding politeiawww API that can be used
for submitting comments on any record, regardless of whether its a proposal
comment or invoice comment.
These application agnostic APIs allow politeia to become a configurable system.
You configure the politeiad plugins for your specific use case and you
automatically get the corresponding politeiawww API routes.