Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose block filters over REST. #17631

Open
wants to merge 3 commits into
base: master
from

Conversation

@TheBlueMatt
Copy link
Contributor

TheBlueMatt commented Nov 29, 2019

This adds a new rest endpoint:
/rest/blockfilter/filtertype/requesttype/blockhash (eg
/rest/blockfilter/basic/header/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f.hex)
which exposes either the filter "header" or the filter data itself.
Most of the code is cribbed from the equivalent RPC.

You can test it at http://cloudflare.deanonymizingseed.com/rest//blockfilter/basic/header/000000005b7a58a939b2636f61fa4ddd62258c5fed57667a35d23f2334c4f86d.hex

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2019-11-filter-rest branch from a2abac0 to 661f03f Nov 29, 2019
@promag

This comment has been minimized.

Copy link
Member

promag commented Nov 29, 2019

A test would be nice.

@practicalswift

This comment has been minimized.

Copy link
Member

practicalswift commented Nov 29, 2019

After reading doc/REST-interface.md I'm not entirely clear about the assumed trust boundaries.

What recommendations do we give to our users regarding exposing the REST endpoints publicly? Do the recommendations differ from our recommendations with regards to exposing the JSON-RPC endpoints publicly?

As I've understood it we regard the JSON-RPC interface as as an internal control plane only to be accessible by trusted clients. The assumption we're making from a trust boundary perspective seems to be that we assume that an untrusted clients will never be able to connect to the port serving the JSON-RPC interface (which is the same port as the REST interface).

@laanwj

This comment has been minimized.

Copy link
Member

laanwj commented Nov 29, 2019

Concept ACK.

After reading doc/REST-interface.md I'm not entirely clear about the assumed trust boundaries.

That's a fair question (FWIW the limit has always been: only public data, no complex queries, do not parse JSON as input), but I'd suggest opening a new issue for it. Please keep this one for review of the code changes.

@practicalswift

This comment has been minimized.

Copy link
Member

practicalswift commented Nov 29, 2019

@laanwj Without knowing if consumers are trusted or not it is pretty hard to review it from a security perspective :)

@laanwj

This comment has been minimized.

Copy link
Member

laanwj commented Nov 29, 2019

The REST interface is a lightweight interface for querying public data. Consumers are trusted but less so than on RPC (as they don't authenticate). I still wouldn't recommend exposing it directly to the internet. But maybe it's OK to open it "publicly" inside some LAN or VPN that your applications run in.

This is my last general comment on this, please open a new issue if you want to continue this discussion.

doc/REST-interface.md

Speaking of which, please update the documentation to mention this new call.

src/rest.cpp Show resolved Hide resolved
@jnewbery

This comment has been minimized.

Copy link
Member

jnewbery commented Nov 29, 2019

Concept ACK

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2019-11-filter-rest branch from 661f03f to 6f2e02f Nov 29, 2019
@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

TheBlueMatt commented Nov 29, 2019

Added a basic sanity test, redid the way headers work to make it easy to get many of them just like the /headers/ request.

src/rest.cpp Outdated Show resolved Hide resolved
src/rest.cpp Outdated Show resolved Hide resolved
return true;
}
default: {
return RESTERR(req, HTTP_NOT_FOUND, "output format not found (available: .bin, .hex)");

This comment has been minimized.

Copy link
@paymog

paymog Dec 1, 2019

If I'm reading the code right it looks like json is a valid output format too along with binary and hex.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Dec 2, 2019

Author Contributor

Hmm, good catch. Looks like headers is busted too, I fixed both.

}

BlockFilterIndex* index = GetBlockFilterIndex(filtertype);
if (!index) {

This comment has been minimized.

Copy link
@paymog

paymog Dec 1, 2019

the code for extracting filtertype, index and blockhash is shared between these two functions. Should this extraction code be pulled out into their own functions?

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Dec 2, 2019

Author Contributor

Lets leave DRY'ing up rest.cpp for a separate commit. I played with it a bit and there isn't an obvious solution here that doesn't end up adding more lines, but the whole of REST probably could get DRY'd up a lot especially in the results-providing section.

LOCK(cs_main);
block_index = LookupBlockIndex(block_hash);
if (!block_index) {
return RESTERR(req, HTTP_NOT_FOUND, uriParts[1] + " not found");

This comment has been minimized.

Copy link
@paymog

paymog Dec 1, 2019

nit: can the error message be changed to "Block " + uriParts[1] + " not found"?

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Dec 2, 2019

Author Contributor

Hmm, its coped from the block code, so to keep it the same everywhere I'll leave it.

@jonasschnelli

This comment has been minimized.

Copy link
Member

jonasschnelli commented Dec 1, 2019

Concept ACK

@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2019-11-filter-rest branch from c2e8ffb to 8c33533 Dec 2, 2019
TheBlueMatt added 3 commits Nov 29, 2019
This adds a new rest endpoint:
/rest/blockfilter/filtertype/requesttype/blockhash (eg
/rest/blockfilter/basic/header/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f.hex)
which exposes either the filter "header" or the filter data itself.
Most of the code is cribbed from the equivalent RPC.
@TheBlueMatt TheBlueMatt force-pushed the TheBlueMatt:2019-11-filter-rest branch from 8c33533 to 3ab6abc Dec 3, 2019
@DrahtBot

This comment has been minimized.

Copy link
Contributor

DrahtBot commented Dec 5, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #17675 (tests: Enable tests which are incorrectly skipped when running test_runner.py --usecli by practicalswift)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@DrahtBot

This comment has been minimized.

Copy link
Contributor

DrahtBot commented Dec 9, 2019

Needs rebase
if (count < 1 || count > 2000)
return RESTERR(req, HTTP_BAD_REQUEST, "Header count out of range: " + path[0]);
if (count < 1 || count > MAX_HEADERS_RESULTS)
return RESTERR(req, HTTP_BAD_REQUEST, "Header count out of acceptable range (1-2000): " + path[0]);

This comment has been minimized.

Copy link
@luke-jr

luke-jr Jan 3, 2020

Member

Use %u here instead of 2000?

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Jan 5, 2020
This adds a new rest endpoint:
/rest/blockfilter/filtertype/requesttype/blockhash (eg
/rest/blockfilter/basic/header/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f.hex)
which exposes either the filter "header" or the filter data itself.
Most of the code is cribbed from the equivalent RPC.

Github-Pull: bitcoin#17631
Rebased-From: aeca9ab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants
You can’t perform that action at this time.