Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[politeia_verify] no longer verifies file contents #718

Closed
s-ben opened this issue Mar 8, 2019 · 7 comments

Comments

@s-ben
Copy link
Contributor

commented Mar 8, 2019

In the process of creating a page in dcrdocs on how to verify censorship, I attempted to verify a sample proposal. I was able to successfully verify using the -json method:

sben$ politeia_verify -v -jsonin /Users/sben/Documents/coins/decred/test_files/e1df7e22336d1984eda646fd2602c1a8fd96f91563635d70ab84a1f42d5bc3ef.json
Record successfully verified

However, when I use the other method (directly passing in filenames along with censorship variables), it fails to verify:

sben$ politeia_verify -v -k 0c141d185ba18d7ab82f041431f8875dd0c4f8e59b4ab230de4fd07b3542d6cd -t e1df7e22336d1984eda646fd2602c1a8fd96f91563635d70ab84a1f42d5bc3ef -s 559c381c1e87d3fdfe03842b93c6c4dc19e3780776ccb26b375ba6946e37268d0a9b50c6574ca1e952cf34ef294477bbdcce7c8958f21e94e4f5568e73bd1407 /Users/sben/Documents/coins/decred/test_files/XRP_Integrate.md
Record failed verification. Please ensure the public key and merkle are correct.
  Merkle: 1d5f5a78402151cdc01377236a9d3a7342af07aa2bf81477f9104eb66b7e7b22

I have tried this using sample proposals uploaded using politeiagui (running locally) as well as politeiawwwcli, to the same result.

This is problematic, as passing in files appears to be the only feasible way to verify the content of a censored proposal. A spurned censored proposal owner can't show the community a string of random characters in a json payload as proof of censorship.

It appears that for politeiagui (the main way users will upload proposals), this is because the way we encode a proposal has diverged from the way politeia_verify encodes it. The UI takes in data through fields in a web form (title, body in Markdown, attachments), and creates a proposal file that is then used to calculate the Merkle root. The user can then download the 'Proposal Bundle', but this is just a json with a payload of random characters.

@lukebp lukebp added the bug label Mar 9, 2019

@xaur

This comment has been minimized.

Copy link

commented Mar 11, 2019

What's in the JSON file that passes verification?

@s-ben

This comment has been minimized.

Copy link
Contributor Author

commented Mar 11, 2019

Here's an example JSON:

{
  "name": "XRP prop",
  "state": 1,
  "status": 2,
  "timestamp": 1552030203,
  "userid": "4a0f6404-ebd1-4530-92a0-773ae8430be2",
  "username": "s_boon",
  "publickey": "b74f66fed058679b3aebebb51b9e3b8a50653d0a56ed9f45c20ca046f077a3f8",
  "signature": "02d8cb2783991cf04afe72a357e02d8d332af232b29b298f314b5958856595f4277a12e6427c259c014c1485e5e0e935cfd0662524aeb2989acf59d0889a3c01",
  "files": [
    {
      "name": "index.md",
      "mime": "text/plain; charset=utf-8",
      "digest": "6528a69cdc674b25485a38829190026891a0e8a55e46cfd07aa41f9554ee64f7",
      "payload": "WFJQIHByb3AKIyBJbnRlZ3JhdGUgWFJQCgpTcGVuZCAxMDAsMDAwIERDUiB0byBJbnRlZ3JhdGUgUmlwcGxlJ3MgWFJQIHRva2VuIGludG8gdGhlIERlY3JlZCBibG9ja2NoYWluIHRvIHNvbHZlIERlY3JlZCdzIHNjYWxhYmlsaXR5IGlzc3VlcyBhbmQgYWxsb3cgaXQgdG8gYmVjb21lIHRoZSBuZXcgc3RhbmRhcmQgZm9yIGRpZ2l0YWwgcGF5bWVudHMh"
    }
  ],
  "numcomments": 0,
  "version": "1",
  "censorshiprecord": {
    "token": "e1df7e22336d1984eda646fd2602c1a8fd96f91563635d70ab84a1f42d5bc3ef",
    "merkle": "6528a69cdc674b25485a38829190026891a0e8a55e46cfd07aa41f9554ee64f7",
    "signature": "559c381c1e87d3fdfe03842b93c6c4dc19e3780776ccb26b375ba6946e37268d0a9b50c6574ca1e952cf34ef294477bbdcce7c8958f21e94e4f5568e73bd1407"
  },
  "serverPubkey": "0c141d185ba18d7ab82f041431f8875dd0c4f8e59b4ab230de4fd07b3542d6cd"
}

So it has the censorship record and the digest and payload (which are derived from the raw proposal data/files I believe, but afaik, not reversible back to the raw data).

@xaur

This comment has been minimized.

Copy link

commented Mar 12, 2019

Digest is clearly not easily reversible, but the payload probably is. Looks like the payload is derived from the single index.md file and each file is encoded separately. Perhaps it's some baseXX encoding? What was in the XRP_Integrate.md file?

Side note:

"digest": "6528a69cdc674b25485a38829190026891a0e8a55e46cfd07aa41f9554ee64f7"

This field doens't explicitly specify the type of hash used (e.g. `"sha256": "blabla"). Weird.

@s-ben

This comment has been minimized.

Copy link
Contributor Author

commented Mar 13, 2019

So it appears the payload is just a base64 encoded markdown file. If I take the payload from the above JSON and copy/paste it into this online base64 decoder, I get this:

XRP prop
# Integrate XRP

Spend 100,000 DCR to Integrate Ripple's XRP token into the Decred blockchain to solve Decred's scalability issues and allow it to become the new standard for digital payments!

Which is the title I gave my proposal (XRP prop) when I uploaded it, followed by the markdown which I copy/pasted into the UI. Below is the markdown from my original test file, which matches.

# Integrate XRP

Spend 100,000 DCR to Integrate Ripple's XRP token into the Decred blockchain to solve Decred's scalability issues and allow it to become the new standard for digital payments!

I haven't tested it with attachments (e.g. an image), but it looks like the function that creates the proposal would just appended attachments after the markdown...

So...I suppose someone looking to prove censorship could base64 decode their payload from the JSON. Then run politeia_verify on that same json using the -jsonin parameter. However, that isn't really proving censorship though, because the -jsonin method uses only the censorship token, signature, and server public key. It would be possible to just swap out the payload of the JSON and it would still verify.

It looks like we could get politeia_verify working again if we base64 encode the files before hashing them, as we do in politeiagui. The only problem there is that the payload doesn't just consist of uploaded files anymore. The user now inputs the title and markdown into the UI, and politeiagui creates the payload on the backend. Additionally, there's an open PR to add a summary field when you're creating a proposal, adding more data through the UI. Presumably it's a better user experience creating the proposal through the UI. But I think we now need to figure out some way for the user to input all that data back into politeia_verify.

@xaur

This comment has been minimized.

Copy link

commented Mar 13, 2019

However, that isn't really proving censorship though, because the -jsonin method uses only the censorship token, signature, and server public key. It would be possible to just swap out the payload of the JSON and it would still verify.

Wouldn't swapping the JSON payload change the censorship token? Can the user generate the token himself from the payload?

Side question: did the now-broken verification flow ever work?

@s-ben

This comment has been minimized.

Copy link
Contributor Author

commented Mar 14, 2019

Wouldn't swapping the JSON payload change the censorship token? Can the user generate the token himself from the payload?

The token is just a random number generated on the backend. The merkle field is the "ordered merkle root of all files in the record" (hashes of raw files concatenated together). The signature field is just the cryptographically signed (by the pi server key I believe) 'merkle+token' (concatenated string of merkle and token). So this signature is really what proves that this unique (due to the randomness of censorship token) profile was uploaded at a specific time (time proven by the fact that this censorship record was recorded at a specific block on the Decred blockchain using dcrtime). Phew!

So the payload cannot be used to generate the token. As @lukebp has pointed out, 'censorship token' is kind of a misnomer, as it's the entire 'censorship record' that is needed to prove censorhip. I updated the Politeia docs recently actually to replace 'censorship token' with 'censorship recored' btw.

  "censorshiprecord": {
    "token": "e1df7e22336d1984eda646fd2602c1a8fd96f91563635d70ab84a1f42d5bc3ef",
    "merkle": "6528a69cdc674b25485a38829190026891a0e8a55e46cfd07aa41f9554ee64f7",
    "signature": "559c381c1e87d3fdfe03842b93c6c4dc19e3780776ccb26b375ba6946e37268d0a9b50c6574ca1e952cf34ef294477bbdcce7c8958f21e94e4f5568e73bd1407"
  }

The way the user proves censorship really, is generating the merkle root. They can then sign the 'merkle'+'token' (which they have from the JSON), and get the signature (which is what is stored on the blockchain?). Anyway, this is actually what politea_verify does under the hood.

Side question: did the now-broken verification flow ever work?

I'm sure it did at one point. But the last code commit was 9 months ago. What happened in that time it appears, is that politeia and politeiagui continued evolving, and diverged from politeia_verify (so far the focus has been on launching and iterating features, not verification). The problem is that politeiagui now encodes proposals and calculates the merkle root in a way that's different. This relates back to the convo you and @lukebp are having this Issue about metadata. If the user inputs metadata through the UI, which isn't put into a file, we no longer have a file that contains all the proposal data. And our current mechanism of verifying censorship breaks down.

Going back to "first principles", what we need to accomplish is: a user needs a way to cryptographically verify censorship in a way that doesn't present an unreasonable technical burden.

The problem with our current strategy is that users are not uploading raw source files anymore. They're doing it through the GUI. The way I see it, there are two basic solutions:

  • Encode the user-input data from the UI into a file format politeia_verify can parse. E.g. all proposal "metadata" (title, summary, etc.) are inserted into a single markdown file. That, plus separate attachments, gives us a set of source files that could be input into politeia_verify and work. This would require though that the 'Proposal Bundle' that users download contain not just the JSON, but the JSON and raw files used to create the proposal on the backend (markdown+image+whatever).
  • Go backwards from the payload. We can just feed politeia_verify the payload (perhaps with slight tweaking) and it will create a matching merkle root that verifies the submission. But then we'd presumably need an easy way for the user to decode that verified payload (a long string of random characters) to show its contents. Like I did with my XRP example above. Then you just have the problem of decoding a wonky, self-created file format. If that format is simple (e.g. 'markdown file'+'attachment1'+'attachment2'+'mime types of all files?), then maybe politeia_voter just spits out the verified proposal's raw files?
@lukebp

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

I'm not able to reproduce this. Closing this issue. If other people are able to reproduce it I will reopen it.

@lukebp lukebp closed this Apr 25, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.