Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tstore backend and plugin architecture. #1180

Merged
merged 496 commits into from
Mar 29, 2021
Merged

Add tstore backend and plugin architecture. #1180

merged 496 commits into from
Mar 29, 2021

Conversation

lukebp
Copy link
Member

@lukebp lukebp commented Apr 30, 2020

This diff rewrites much of the politeia codebase. A summary of the major
changes are listed below.

politeiad tstore backend

A new backend was added to politeiad that is referred to as the tstore backend.
It combines a trillian log, commonly referred to as a tlog, and a key-value
store. The tstore backend differs from the git backend in three main ways.

  1. Scalability

    The git backend relied on git repos saved to the file system to store data.
    This approach cannot be scaled out since it is constrained to the file system
    of a single politeiad instances. The tstore backend moves the data into an
    actual database, allowing for much greater scalability.

  2. Separate the timestamps and the data blobs

    The git backend would timestamp the git commit hashes onto the decred
    blockchain. These commit hashes are merkle roots of all the files in the
    commit. The timestamp and the data blobs in this setup are interconnected. If
    you delete a data blob from the commit history, such as if you need to censor
    a public comment, you destroy the ability to recreate the merkle path for
    other data blobs included in that commit merkle root, thus destroying you're
    ability to prove inclusion of data in a decred timestamp. Your only option is
    to remove the data from the git repo in a new commit, but the censored data
    remains in the git history. The tstore backend seperates out the timestamps
    from the data blobs. A merkle tree of the data is stored in a tlog instance
    while the data itself is stored in a key-value database. This means we can
    delete data from the database, such as if you need to censor a public
    comment, without impacting any of the timestamps or our ability to prove
    the inclusion of a piece of data in a timestamp.

  3. Ability to retrieve timestamp inclusion proofs

    The tstore backend gives us the ability to retrieve an inclusion proof for
    any piece of data in the backend, providing cryptographic proof that the data
    blob was included in a dcr timestamp. There was no easy way to do this in the
    git backend.

politeiad v2 API

A politeiad v2 API was added. This API is a simplified version of the politeiad
v1 API. The tstore backend relies solely on the politeiad v2 API.

politeiad plugin architecture

A formalized plugin architecture was added to the tstore backend.

A politeiad plugin extends a record with additional functionality. Some
examples of this include adding comments to a record, adding the ability to run
a dcr ticket vote on a record, and adding record validation that is specific to
the decred proposal system.

This plugin architecture allows politeiad to become a configurable system. If a
user wants to runs a politeiad instance that allows for specific file types and
record comments, they can do so by simply adjusting the politeiad
configuration. No new code is required unless they want to add in more
granular, application specific validation.

The end goal is to eventually allow plugin code to exist outside of the
politeia repository that can be dynamically loaded at runtime. While this is
not possible yet, this diff lays the foundation for doing this in the future.

politeiawww APIs

The politeiawww APIs have been rewritten to be generic, application agnostic
APIs. These APIs were formally tied to specific politeia applications such as
the decred proposal system and the decred contractor management system. You
would use a proposal route to submit new proposals and a separate invoice
route to submit cms invoices. These application specific routes have been
replaced with more generic APIs. For example, there now exists a single set
of routes for submitting all records, regardless of whether that record is a
proposal or an invoice. The application specific validation has been pushed
into the politeiad plugins. The validation that is specific to proposals is
now performed in a proposal system politeiad plugin.

The politeiawww APIs now map to the politeiad plugins. For example, the
politeiad comments plugin has a corresponding politeiawww API that can be used
for submitting comments on any record, regardless of whether its a proposal
comment or invoice comment.

These application agnostic APIs allow politeia to become a configurable system.
You configure the politeiad plugins for your specific use case and you
automatically get the corresponding politeiawww API routes.

@lukebp lukebp changed the title [WIP] tlogbe: Add tlogbe implementation. [WIP] backend: Add tlogbe implementation. Jul 31, 2020
@amass01
Copy link
Member

amass01 commented Sep 14, 2020

Politeia's routes which are currently used in decrediton:

/v1/proposals/tokeninventory
/v1/proposals/batch
/v1/proposals/batchvotesummary
/v1/proposals/[token]
/v1/proposals/[token]/votes
/v1/proposals/castvotes

@lukebp lukebp changed the title [WIP] backend: Add tlogbe implementation. [WIP] multi: Add tlog implementation. Sep 16, 2020
@lukebp lukebp changed the title [WIP] multi: Add tlog implementation. Add tstore backend and plugin architecture. Mar 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants