Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Block Observatory: a catalog of all blocks seen and when we saw them #1922

Open
zookozcash opened this issue Dec 5, 2016 · 15 comments

Comments

Projects
8 participants
@zookozcash
Copy link

commented Dec 5, 2016

Importing this from https://github.com/Electric-Coin-Company/infrastructure/issues/44

Add a feature to zcashd that stores all blocks, along with a timestamp of precisely when it received that block, and from which peer. The point of this is it records all blocks, including blocks that are invalid for any reason, including invalid because they have invalid transactions, or invalid because they are inconsistent from the main chain (when they arrive) or they get invalidated (orphaned) by a subsequent re-org after they arrived. Store all of them.

@zookozcash

This comment has been minimized.

Copy link
Author

commented Dec 5, 2016

This is a necessary step to implement #1924

@nathan-at-least nathan-at-least added this to the 1.0.5 milestone Dec 5, 2016

@zookozcash zookozcash changed the title network observatory: a catalog of all blocks seen and when we saw them The Block Observatory: a catalog of all blocks seen and when we saw them Dec 5, 2016

@zookozcash

This comment has been minimized.

Copy link
Author

commented Dec 5, 2016

This would be a great basis with which to implement #1925, although it sounds like @str4d might be implementing #1925 right now which a quicker and dirtier approach than this.

@ebfull ebfull modified the milestones: 1.0.5, 1.0.6 Jan 19, 2017

@bitcartel bitcartel modified the milestones: 1.0.7, 1.0.6 Feb 6, 2017

@str4d str4d modified the milestones: 1.0.7, 1.0.8 Mar 10, 2017

@abrkn

This comment has been minimized.

Copy link

commented Mar 19, 2017

Has any progress been made? I'm looking to do this coin agnostically for another project, but not sure how to approach the problem.

@daira

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2017

We are going to implement this via AMQP messaging rather than by adding a specific feature for it in zcashd, so this should be moved to an infrastructure ticket.

@zookozcash

This comment has been minimized.

Copy link
Author

commented Mar 23, 2017

@abrkn: I'm wondering about progress, too (for Zcash, and branches thereof). Judging from the fact that it is put into the 1.0.8 Milestone, I predict it will be deployed live on 2017-03-27, because that's the due date for that Milestone: https://github.com/zcash/zcash/milestone/52

Zcash team: if I'm wrong and I can't expect to see this in production in 5 days, please let me down gently ASAP and change the status of this ticket.

@ebfull

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2017

We need to store everything about the block, btw. The hex, all the transactions, everything the remote client sent to us regarding the block.

@nathan-at-least

This comment has been minimized.

Copy link
Contributor

commented Mar 23, 2017

I sense some diverging expectations about what this ticket provides and who does what. In terms of the scope, to me it sounds like Zooko wants:

  • a website he can visit (or other easy query mechanism) from which he can learn "relevant information".
  • an obvservatory system that tracks low-level protocol details such as invalid blocks. (Note, I call this 'low-level' because AFAIK, this information is not provided for in the RPC interface.)

There are at least three engineering projects here that are significant work:

  • setting up the website or front-end.
  • 'observatory backend' logic and deployment which gathers that information from one or more nodes.
  • altering zcashd to report low-level information.

I think each of these deserves at least one separate ticket, but probably more.

The first two are 'infrastructure' tasks, except the second may also involve significant software engineering. In that case it deserves more than one ticket, since one might be 'deploy the observatory code and monitoring nodes' and another might be 'write the observatory code', or these might both fall into an infrastructure config task, depending on how the observatory backend implemented.

The third is a zcashd feature implementation task. AMQP support itself is just supporting code component, but doesn't directly address the need. For the third task, we need multiple tickets, each with specific information the observatory needs. If some of these are already trivial because they map closely to RPC calls, great, just make sure they're provided on AMQP.

Here are some thoughts on information that should be provided and how trivial or significant they are:

  • chaintip info - trivial (and redundant with the existing chaintips detector, so don't break what's already working)
  • peer info - trivial (same as above)
  • 'firehose' block producer - report on all blocks received, even if not valid - significant, based on my guess that this layer of the code isn't close to the AMQP producer interface.
  • block arrival time - easy (unlike another ticket [FIXME link] this is not persistent and can hopefully just be included information in an AMQP producer code for blocks)

So we need to break these into multiple tickets; I have to run right now. Zooko, could you be more specific about what you need?

@ageis

This comment has been minimized.

Copy link
Contributor

commented Mar 24, 2017

What we need in order to deliver on this ticket is pretty clear to me. It's also clear to me that it does require a significant engineering effort, including a new codebase/repository with multiple people contributing and reviewing the code, and we definitely won't be able to accomplish it by the milestone. I also agree that it needs to be divided into pieces.

But what it looks like in my general sense is this:

  • A Python service running alongside zcashd on detector nodes and which communicates to it using AMQP (which is still awaiting merge in #2189 ) and is able to then constitute an array or set of information about each incoming block (call it a JSON object) as it is witnessed, and then ships that information to a listening central server, via an authenticated HTTP POST.
  • Another service running on that server which ingests the data coming from in from the detectors' POST requests, and stores it in some type of persistent database.
  • A web-hosted user-interface to display all of this data about each and every block publicly. Obviously requires some front-end development and design chops so that it all looks and feels nice, you can explore it over time, filter or query it, and so on.

We began an earlier effort in December which was focused on porting python-bitcoinlib to be able to access the protocol data structures, I am not sure why that did not pan out.

There's also apparently an RPC test for hard fork detection which I'm just catching up on: #1009 Is that of any value to us at this point?

Also, what's the overarching vision and justification for why this is important to have? This is definitely a cool-to-have, but is it need-to-have? We have a policy that hard forks will be inevitable and should be embraced and nodes will definitely have to upgrade, so we want to manage the network and ensure everybody remains on the correct chain. What are our potential responses to this new array of information we'll have?

@zookozcash

This comment has been minimized.

Copy link
Author

commented Mar 25, 2017

Here are examples of the sort of effects that an observatory might be able to detect: https://btc-hijack.ethz.ch

@daira daira modified the milestones: 1.0.9, 1.0.8 Mar 26, 2017

@zookozcash

This comment has been minimized.

Copy link
Author

commented Mar 26, 2017

"Block Observatory User Story Number One":

"""
Alice heard a rumor on IRC that an invalid block was seen on the Zcash p2p network. She loads https://observatory.z.cash and it shows a bunch of information about the blockchain and the p2p network. She scans through the events shown there, or runs "search in text", or types in a query into a search box, and finds the most recent case that an invalid block was received by one of the nodes that contributes to the observatory. She sees which node received the block, from which peer, at what time, and what message the node emitted when deciding that the block was invalid. She then clicks, and the full binary content of that invalid block is downloaded to her computer so she can analyze it.
"""

How's that?

@nathan-at-least

This comment has been minimized.

Copy link
Contributor

commented Apr 1, 2017

That story encompasses a vast scope. "a bunch of information"? Running queries? Happening to find exactly the kind of data the user wants to learn about?

Let's pick a single kind of data with a well defined measurement, restrict "queries" to selecting which sources a measurement comes from and date ranges, and then we might have a ticket we can close. After that, we can begin adding more measurements or query options.

@zookozcash

This comment has been minimized.

Copy link
Author

commented Apr 15, 2017

"Block Observatory User Story Number One, Second Version":

"""
Alice heard a rumor on IRC that an invalid block was seen on the Zcash p2p network. She loads https://observatory.z.cash and it shows a list of blocks that have been seen, when they were seen by which nodes, and which ones are currently present in the main chain. She scans through the blocks shown there, or runs "search in text", and finds the most recent case that an invalid block was received by one of the nodes that contributes to the observatory. She sees which node received the block, from which peer, at what time, and what message the node emitted when deciding that the block was invalid. She then clicks, and the full binary content of that invalid block is downloaded to her computer so she can analyze it.
"""

How's that?

@nathan-at-least nathan-at-least removed this from Work Queue in Continuous Improvement May 10, 2017

@nathan-at-least nathan-at-least removed this from the 1.0.9 milestone May 10, 2017

@nathan-at-least nathan-at-least moved this from In Progress to Nominated for Release in Monitoring, Metrics, and Analysis Jul 3, 2017

@nathan-at-least

This comment has been minimized.

Copy link
Contributor

commented Jul 13, 2017

"Block Observatory User Story Number One, Second Version" is more specific about the information available.

I think we should approach this in a "vertical MVP" fashion: make the full stack work, then add features (which require changes to multiple stack layers). Proposed roadmap:

  1. https://github.com/bitpay/insight-ui modified to integrate with the latest zcashd release (ex: 1.0.11)
  2. augment zcashd to report timestamps-when-seen and from which peer, then augment insight-ui to display that. (rationale: I guess this is easier than reporting invalid blocks.)
  3. augment zcashd to report invalid blocks, then augment insight-ui to display that.
  4. augment insight-ui to consume information from multiple peers (perhaps over AMQP), then to display peer-source information for relevant info (ex: this block was reported by A and B but not C).

Edit: Made this comment more concise.

@abrkn

This comment has been minimized.

Copy link

commented Jul 15, 2017

Ever set up insight? That code is not great.

@nathan-at-least

This comment has been minimized.

Copy link
Contributor

commented Sep 21, 2017

Our latest thinking is that it'd be much better to not modify zcashd to have more complexity to support this, and instead use a simpler peer-network level tool, such as https://github.com/str4d/pynode/tree/rebased-often (which depends on https://github.com/str4d/python-bitcoinlib/tree/rebased-often), a fork of (recent) bitcore, or some other tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.