Skip to content
This repository has been archived by the owner. It is now read-only.

Next Consensus Architecture discussion #1631

Open
vukolic opened this issue May 27, 2016 · 70 comments

Comments

@vukolic
Copy link

commented May 27, 2016

Use the comment field to discuss the next consensus architecture proposal.

@gregmisiorek

This comment has been minimized.

Copy link

commented May 28, 2016

for section 2.1, we should try to use a UTC timestamp of possible: tsUTC.

@gregmisiorek

This comment has been minimized.

Copy link

commented May 28, 2016

for section 2.2, 1st alternative design, HASH(txPayload) seems preferable as we should try to avoid clear text wherever we can without losing the clarity of the specification.

@adecaro

This comment has been minimized.

Copy link
Contributor

commented May 29, 2016

Just some clarifications:
In Section 1.2.i, shall 'Clients create and thereby invoke transactions.' be '' Clients create and thereby invoke chaincodes'?
In Section 2.1, shall 'To invoke a transaction' be 'To invoke a chaincode'?

@binhn

This comment has been minimized.

Copy link
Contributor

commented May 31, 2016

@gregmisiorek yes, hash would be preferred, especially in the case of confidential transactions.
@adecaro at this level, we should use transaction instead of chaincode to be more generic.

@bmandler

This comment has been minimized.

Copy link

commented May 31, 2016

A couple of comments / thoughts

  1. The struggle to keep a blurry distinction between the different kinds of peers seems to add unnecessary confusion without having a real need (examples of how this complicates the text can be seen in the last paragraph of section 1.2: "Nodes other than committing ..., and section 2.3 first bullet, second paragraph). For simplicity I'd unify these entities and state that there is an entity called peer and it has 3 distinct capabilities, namely endorser, submitter, and committer. To my understanding all peers can play all roles and their specific role is determined per chaincode (so a peer can be endorser for one chaincode, and a submitter for another, but I think that all need to be committers). Since I think that all peers need to maintain the state and all peers should be able to communicate with clients (see 1.2.i: the client may connect to any peer of its choice), I don't see a real value in the separation.
    • I believe an endorsing peer has to maintain the state as well, first because it may very well be needed for endorsements, and second because it's required for simulating the transaction execution (and is required to respond to a query request).
    • In section 1.2.ii.a it's stated that a committing peer may give clients read access to the blockchain and state. Once again. It means that all committing peers are submitting peers; and in addition if read access to the state means responding to chaincode queries, than only the endorsers can respond (and potentially the submitting peer), since they are the only ones that have the actual chaincode in question.
  2. The flow of deploying a transaction is not detailed, I assume the deployment takes place only on the endorsing peers specified by the endorsement policy and on the submitting peer. What happens if a following transaction invocation for the same chaincode comes through a different submitting peer? That peer may not have the chaincode deployed and thus won't be able to simulate the execution of the transaction.
    • It is stated that deploy transactions are controlled by the system chaincode rather than the endorsement policy of the specific chaincode being installed. Why is that?
  3. The flow of a query request is not detailed. Is it serviced by the submitting peer? Does the submitting peer involve the endorsers? Does it go through the consenters at all?
  4. Section 2.3 second bullet b (STALE_VERSION): if the submitting peer receives back such an answer it can drop the transaction or restart the process from the beginning rather than send this to the consenters even if the endorsement criteria was obtained, since it will fail at commit time. Naturally this gets complicated if taking into account that an endorser can be Byzantine.
  5. All peers need access to the metadata of all chaincodes (to verify the endorsement), and require access to all the endorsement policies of all chaincodes. This is distributed to all peers at chaincode deploy time?
  6. Confidential chaincode is supported by having the chaincode deployed only on the endorsers. The assumption is that there are no Byzantine endorsers? Otherwise, a Byzantine endorser can do anything with the chaincode, and communicate it to whoever he wants.
  7. How is a Byzantine submitting / committing peer handled?
  8. In the description of the raw ledger there's the following sentence which I don't understand: It also provides submitting peer with information about invalid transactions ...
  9. As for Binh's earlier remark on non determinism; this is captured by the endorsers and reported back to the submitting peer (INCORRECT_STATE), so the submitting peer can take an appropriate action based on that response. Personally, I'd vote for dropping such a transaction since I don't see the value of supporting non deterministic transactions within a blockchain. The problem that arises than of course is how to handle a Byzantine endorser that replies with such an error.
@vukolic

This comment has been minimized.

Copy link
Author

commented May 31, 2016

@bmandler
Many thanks for very insightful comments! Let me respond quickly - but addressing fully requires more work as discussed below:

  1. Good point - and this was originally the case in the design, until there was a remark that endorsers may be more general as peers that do not necessarily maintain the state. But in principle, I agree - that we could simplify + endorser will most of the time be actually a committing peer.
  2. The idea is that transaction deployment are seen as an invocation of system chaincode, where a certain set of (system chaincode) endorsers may need to endorse that a transaction is deployed. I again agree, the flow of the deploy is todo.
  3. There was already a comment re this from @christo4ferris - will be added (todo).
  4. In principle, this would be an implementation optimization (there will be many of these). But, as you correctly point out - depending on the trust model in the endorsers of this chaincode - this optimization may be applicable only when a certain number (or set) of endorser reply with STALE_VERSION (cf Byzantine endorsers). I suggest to address this by adding a comment along these lines to the design draft.
  5. This is why deployment might need to go to all committing peers not just to endorsers as you suggested in 2) - this will need to be made clear when Deploy flow is outlined.
  6. With the confidentiality as enabled by this design, confidentiality would be violated with Byzantine endorsers specified by the chaincode. Notice that the integrity guarantees of such a chaincode execution could be maintained despite confidentiality violatoins. If this level of confidentiality is insufficient, chaincode can/should resort to application (chaincode) level confidentiality techniques (which the design does not prevent).
  7. Dealing with submitting peer submitting a tx with wrong endorsement - is handled with endorser signatures. As for interactions with consensus - it is the job of the consensus module to deal with Byz submitting peers (as Byz consensus clients) and Byz committing peers. Byz committing peers are simply learners in Paxos wording so it is easy to tolerate any number of Byz committing peers from consensus perspective. Dealing with Byz committing peers for queries will need to be spelled out - when we spell out queries.
  8. This means that a submitting peer, acting as a committing one will be able to deduce an invalid tx and act upon it as agreed w. client (cf retryFlag, etc.)

BTW, if you want to volunteer for writing one of these TODOs (e.g., deployment/queries/anything else) - pls ping me so we coordinate. Thanks again!

@mihaigh

This comment has been minimized.

Copy link

commented May 31, 2016

Typos:

  1. on section 1.2.ii. point b it is written "on behalf one or more clients for executing the transaction." instead of "on behalf of one or more clients for executing the transaction."
  2. on section 1.2.iii. it is written "and can then send messages and obtain see messages that arrive" instead of "and can then send messages and obtain the messages that arrive"
@mm-hl

This comment has been minimized.

Copy link

commented Jun 2, 2016

Hi guys

Thanks for the good work on the Next Consensus Architecture Proposal. I like the general direction of moving away from a monolithic architecture, allowing for more flexibility and separation of concerns. I have a bunch of comments at varying levels of detail. They are listed more or less in order of the relevant parts of the proposal. These comments were written while reading this version. Please excuse the ugly long numbered list but this may facilitate easier referencing for further discussion

  1. The name "submitting peer" seems potentially confusing, because it is in some sense the client that "submits" the transaction. (Note who's doing the submitting in this sentence: "A submitting peer is a committing peer to which a client can directly connect to submit transactions, providing an interface between clients and the blockchain.")

  2. Perhaps the term "accepting peer" (or just "acceptor") would be useful, as it accepts transactions that clients submit.

  3. In Section 1.2, this seems contradictory and is at least confusing: "All peers usually maintain the state, except for endorsers. In a typical deployment endorsers will always be committing peers as well and, hence, maintain the state." (due to use of "usual" and "typically" for two sentences that contradict each other ).

  4. While the comments regarding nondeterminism at the beginning of Section 2 make sense, it's unclear to me that it would ever make sense to trust submitting nodes not to be byzantine while still bothering to have all his other blockchain infrastructure. Unless there is a sensible such scenario, the insistence that determinism isn't required seems to merit a technical footnote at most, not an a high-level, introductory comment that makes it sound like there is some significant advantage due to not requiring determinism. (I'm yet to be convinced that nondeterministic chaincodes are worth the substantial trouble they bring, nor that there are convincing solutions to support them; happy to be convinced otherwise.)

  5. In Section 2.2, there is no discussion of the motivation to convert the transaction into a state update and dependence version representations. As noted below, there are some downsides to this, and the upside is not explicitly stated, making it difficult to weigh the perceived pros and cons.

  6. It is unclear exactly what is state update and exactly what is verdep. One can read between the lines and figure out what is probably intended, but better to be explicitly clear.

  7. What would be the point of having a tran-proposal contain a hash of the txPayload? It provides limited transaction confidentiality, given that state update and version dependencies are included anyway, and it doesn't quite guarantee unique transaction IDs either, given that some transactions might not update any state.

  8. Is it a conscious choice to have some signatures cover the message/statement type (e.g., TRANSACTION-[IN]VALID) and some not (e.g., PROPOSE, SUBMIT)?

  9. The alternative design of having the consensus fabric do the endorsing as well seems like it would not maintain a clean separation of concerns, and also seems that it may undermine one of the stated advantages of having endorsers in the first place, namely improving scalability by allowing different sets of nodes to be endorsers for different chaincodes.

  10. In Section 2.3, would it make sense to allow more generality for rejection reasons, particularly to distinguish between error conditions and policy-based rejection?

  11. Also, many/all of the design alternatives mentioned in Section 2.3 seem reasonable in different scenarios, and also multiple combinations of roles for different peers make sense in different scenarios. So I am wondering if a little more generality in the protocol might accommodate all of them cleanly without the protocol dictating specific choices about who sends what where. For example, messages could include indications of whether notification is required if the tx is invalid, whether TRANSACTION-VALID/TRANSACTION-INVALID statements should be sent to the endorser or directly to a committing peer, evidence of endorsements gathered so far, etc.

  12. I like the generality implied by the consensus fabric just getting "blobs" and ordering them. For example, at first I thought this would allow Hyperledger fabric to support variations on the theme exemplified by BigchainDB, where consensus determines order of blocks, and block validity is determined later by peer voting. But Section 2.4, is more specific than this, seemingly saying that blob=transaction. (More about this below now that I've read Section 4.1.) Similarly, Section 2.5 refers to consensus fabric delivering transactions, which again seems as if it may be overly specific. Maybe I am overgeneralizing the intent of the overall project and the "blob" language is just to emphasize that the contents of the transactions are opaque for consenters. It's an interesting question to what extent the project should attempt to support a broader range of possible implementations. I'm not sure to what extent that kind of thing has been discussed/agreed, but there is clearly a spirit of supporting various pluggable things, so I wonder if that could/should also extend to having sufficient generality to allow, for example, submitting peers to construct blocks of transactions, and committing peers to unpack them and process them. If a decision is made to preclude this possibility, it may warrant more discussion, motivation, validation, etc. For example, to what extent would such a decision impact communication overhead?

  13. This sentence in Section 2.5 is confusing: "For example, once can provide serializability when for all keys in the readset and writeset, the current version in the state is still the same as the one in the readset, and all other transactions are rejected."

    • s/once/one/ ?
    • As far as I can see, there is no assumption that the writeset is a subset of the readset, so it doesn't make sense to require a key in the write set to have a version equal to "the one in the readset"; I think the intention is that, for a key in the writeset, the version in the state equals the one in the writeset.
    • Also the phrasing is funny, such that it seems on first reading like "all other transactions are rejected" is part of the condition described in the sentence, which doesn't make sense.

    How about: "For example, serializability can be provided by requiring the version associated with each key in the read set or writeset to be equal to that key's version in the state, and rejecting transactions that do not satisfy this requirement." ?

  14. The discussion of different isolation levels makes me uneasy. A few observations:

    • The idea of converting transactions into readsets and writesets early on (before endorsements are gathered, for example) and then rejecting a transaction if it does not still apply against the same versions witnessed earlier creates a window in which a transaction may be rejected due to concurrent updates when this is not strictly necessary. For a simple example, imagine transactions that increment a counter stored against some key. In systems such as the current Hyperledger fabric, Ethereum and others, where the chaincode/contract is executed when the transaction is validated, two transactions that increment the counter and are submitted around the same time will be considered in order and can both succeed. In contrast, with the proposed arrangement, the two transactions may see the same state and by the time they are processed, one of them must fail, even though there is no reason from the point of view of the chaincode's semantics why it must fail.
    • "Optimizations" such as considering weaker guarantees (e.g., snapshot isolation) may mitigate this to some degree, but also introduce cognitive overhead for a brand new class of programmers in that they have to be aware of the possible semantic implications, and if they are not acceptable, work around them explicitly in a way that cancels any performance improvement that might have been offered by implementing a weaker guarantee. Most likely, people will often overlook this issue, maintain a mental model of serializability, and it will be fine most of the time. Right up until it isn't.
    • There seems to be no reason for a decision on one guarantee. For example, different smart contracts could provide different guarantees, if it really made sense to offer different ones. (Like the proposal, I am not considering cross-chaincode transactions here, but I think at least common combinations would work and provide the weaker guarantee.)
      Overall, if a decision is made to support a weaker guarantee, this decision should be supported by concrete evidence of a tangible benefit, and explicit acknowledgement of the potential downsides and risks and plans to mitigate them (e.g., via documentation and education). Especially if the decision is to support only a weaker guarantee.
  15. In Section 3.1, "the endorsement policy is a predicate on the transaction, endorsement, and potentially further state" sounds slightly circular to me at first. It might be clearer if "endorsement" were replaced with "endorsement(s)" or "endorsement message(s)". I think the idea is that we could have a policy that states, for example, if any 2 of these 3 endorsers endorse the transaction, then it is considered endorsed. It is not clear to me where these individual endorsements would fall in the list of data to which the policy may refer. Presumably not in the "and potentially more" bullet for the foundations of such a simple policy. Yes, it would involve their keys/identities but that doesn't refer to the fact that they have endorsed this specific transaction. Presumably it is intended that the endorsement policy would depend on the messages sent by endorsers, as described in Section 2.3.

  16. The last bullet in Section 2.5 says "It is important to note that invalid transactions are not committed, do not change the state, and are not recorded.", which appears to contradict the first bullet in Section 4, which says "The raw ledger contains information about both valid and invalid transactions and provides a verifiable history of all successful and unsuccessful state changes and attempts to change state."

  17. The second bullet in Section 4 seems incomplete: in addition to filtering out invalid transactions, the effects of valid ones are applied to the state.

  18. What does it mean for the raw ledger to be "mandatory"? If I don't store it, how can you tell and why do you care? Do I really have to store all invalid transactions (which includes transactions that are perfectly fine except that they became invalid because someone concurrently updated some state that they access)?

  19. What is the added value of the validated ledger if the raw ledger is "mandatory"?

  20. In Section 4.1, we learn that the consensus fabric can form blobs. When I mentioned the BigchainDB approach above, I was imagining that submitters would form blocks of transactions, and submit these as blobs to the consensus fabric, which would simply produce an authenticated ordering of blobs. That seems cleaner to me, keeping the consensus fabric focused on its core business, and not getting involved in policies of what size blobs to create, how to represent them, etc. What's the reasoning behind putting this into the consensus fabric instead?

  21. This seems contradictory: "Consensus batching does not impact the construction of the raw ledger, which remains a hash chain of blobs. But with batching, the raw ledger becomes a hash chain of batches rather than hash chain of individual blobs."

  22. Section 5 refers to "state" but is really talking about ledger data, not the "(Blockchain) state" referred to by the second bullet of Section 4. This could be confusing. And I think there is a deeper issue here than just presentation, which I will explain below, after a few other comments.

  23. While the checkpointing approach allows pruning of raw ledger data so that invalid transactions do not need to be stored indefinitely, the first bullet in Section 5.2.iii has a fallback case that depends on other peers having retained the raw blocks unless a trusted checkpoint is available. So we could not guarantee that both: i) all invalid transactions can eventually be pruned; and ii) a new peer can always start without depending on finding a trusted checkpoint. Maybe that's OK, but these issues really make me question the motivation for storing the invalid transactions in such a way that they affect the hashes required to verify valid transactions.

  24. If we need to retain invalid transactions, why not record them separately? For example, directly storing the validated ledger, and including in each block's header the Merkle root of a structure storing the invalid transactions, would allow the invalid transactions to be discarded at will without interfering with the ability of peers to catch up with the validated ledger. If the invalid transaction sets for each block also contained indications of the order in which invalid txs were processed wrt the valid ones, this would allow reconstruction of the "raw ledger" by anyone who chose to retain the invalid txs. Block headers of the valid ledger could also include hashes of associated raw blocks, allowing verification of the reconstructed raw ledger. (I am not suggesting that all this is needed, just pointing out that it's possible to reconstruct and verify the same information recorded in the raw ledger in the current proposal.)

  25. This approach separates the issues of retention of invalid transactions and recovery from checkpoints (or replaying from the beginning in case no trusted checkpoint can be found). However, this still does not address the deeper issue I alluded to above.

  26. While the proposed checkpoint approach allows a catching-up peer to skip processing invalid transactions for the missed blocks, it still requires it to process all valid transactions. For a new peer, this means requesting and processing all valid transactions since forever, perhaps years worth of transactions, just to be able to start operating.
    Some blockchains, like Ethereum for example, include a Merkle root of the state tree in block headers (and store block's transactions separately from the headers). This enables peers to provide Merkle proofs of state reported to light clients, and, more directly relevant to this proposal, also enables "fast sync", whereby a new node can request just headers from peers and then receive a representation of the recent blockchain state from (a) peer(s), which it can validate against the state Merkle root from the associated block header. In the case of Ethereum, this allows a new node to be started in hours, rather than quite a few days, and this is after less than one year of operation.
    If the validated ledger included such state Merkle roots, then checkpoints would similarly contain sufficient information to allow a peer to request the state (as in the KV store) from (a) peers(s) and verify that it is the correct state for the checkpointed block, thus avoiding the need to receive and process all valid transactions since forever. If a trusted checkpoint cannot be found by a catching-up or new peer, then it can still request all the valid transactions and process them all, regardless of whether everyone else has retained invalid transactions.
    Is there some compelling motivation or advantage associated with the proposed approach that I am missing?

Hope some of this is helpful.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 2, 2016

@mm-hl Thanks! I am again going to focus in the response only on vital things:
 
3) to be fixed - probably by making endorsing peers also committing peers
 
4) ack - to be fixed
 
5 + 14) Let me tackle this one early on: The approach to handling concurrency is a critical one. Version dependencies were chosen after weeks of (sometimes heated :) discussion with @chetmurthy and @cca88. Although this one probably deserves a separate issue - let me elaborate a bit here.
 
In principle, we have two approaches to handling concurrency at hand.
 
a) version dependencies (verDep): these are typically used in the context of databases where a concurrent change to an object is conditional to object not changing in the meantime. This provides liveness-level typically called in distributed computing - obstruction freedom. In DB parlance we need to specify isolation guarantees, which as you correctly point out - may be specified per chaincode and not globally.
 
b) sequencing through a leader (leader): in this approach - taken by many distributed systems, there would be a leader endorser that sequences concurrent requests and resolves concurrency. This approach is taken by the Sieve protocol that is a current HL fabric prototype consensus protocol (http://arxiv.org/abs/1603.07351).  
 
Both approaches have pros and cons. Again, this requires a separate issue but will try to put out the essence here.
 
"verDep" PROS: 

  • easier to code HL fabric than "leader"
  • allows a lot of chaincode endorsement parallelism in absence of concurrent writes to the same object
  • suitable for data models such as Bitcoin UTXO - where there should in fact be no concurrent writes to the same object (these constitute a double-spend attempt)
    "verDep" CONS:
  • obstruction-free liveness: will commit only in case of no concurrent tx modifying the same object
  • for a programmer used to centralized-non-concurrent coding: more difficult to code chaincode
     
    "leader" PROS
  • wait-free liveness: a transaction will be sequenced by a leader regardless of concurrent ones
  • easier to program chaincode: a classical counter increment would be ok
     
    "leader" CONS: 
  • more difficult to code HL fabric - need to have leader election at the peer level - possibly per chaincode
  • limits chaincode execution parallelism - optimizations - beyond sharding - are possible but complex (see Eve paper at OSDI'12)
     
    In the end, which approach is better - will depend on the data model and the level of granularity on which concurrent data modifications are allowed.
     
    Some ideas on how to have best of both worlds:
  • have "verDep" with libraries handling implementations of things such as counter. Counter could mimick voting in the real world, where voters incrementing a counter for one candidate, simply cast their vote - and somebody else tallies the votes later.
  • combine the two - in principle, it is possible to even combine the two approaches in a single system (w. sharding), obviously at the expense of higher system complexity.
     
    7)  this one may as well simply be tid
     
    8) yes - PROPOSE are point to point msgs that need not be transferred (hence no need for digital signature). SUBMIT is actually signed by the client.
     
    9) Indeed - I assume this is your vote towards keeping consenters separate from committers?
     
    10 + 11) in principle more generality could be allowed
     
    12) we tried hard to keep blob!=transaction and to keep the separation - I will do a careful pass to enforce this
     
    15) We intended to say that endorsements are collection of signatures on TRANSACTION-VALID msgs. In a typical case, endorsement policy will take an endorsement (set of endorser signatures) and return true or false. Perhaps it would be more clear to call a single endorser's signature - endorsement and a set of those sth else (e.g., endorsementSet or sth nicer)
     
    16) "and are not recorded" will be dropped
     
    17) ack
     
    18 + 19) agree. Perhaps the right way is to simply remove the mandatory/optional distinction/mention. Esp since RL is "mandatory" but can be pruned (garbage collected)
     
    20 + 21) Consensus fabric can form batches not blobs. Batches are lists of blobs. Will double check to make sure this is clear.
     
    22 + 26) adding blockchain state hash to the checkpoint was again one of the things discussed but dropped (for simplicity). Would you be ok with adding that as an optional field, or would you go all the way to requiring it must enter validated ledger and/or checkpoint?
     
    23) storing invalid transactions in the hashchain is an artifact of separation of committers and consenters. As submitting peer can submit whatever to consensus service, either we: a) have consensus service hashchain also invalid transactions, or b) have consensus service "look into" a tx, effectivelly putting chaincode state (at least the endorsement and versions) into consenters (i.e., mandating that consenters are committers). Please also refer to your comment 9)
     
    24) The problem is in telling which tx is invalid if consensus service is agnostic to state (i.e., if consenters are not committers). This is why we first chain valid+invalid in RL and then filter out with another chaining at peer level (validated ledger).
     
    Many many thanks again for great comments. Pls keep on commenting/discussing.
     
      
@mm-hl

This comment has been minimized.

Copy link

commented Jun 3, 2016

@vukolic thanks for the quick and detailed response. Minor responses in this comment, longer discussion in the next.

  1. Doesn't seem to address my comment, not sure what you are saying.

  2. All of the things mentioned (including PROPOSE and SUBMIT messages) contain signatures, according to my reading of the proposal. My comment is about what is covered by the signature and specifically about why the message type is covered in some cases and not in others. I can see that it's potentially important for the signature to cover the TRANSACTION-[IN]VALID, so nobody can lie about the determination, but I can't see that it's important not to cover PROPOSE and SUBMIT, and all else being equal consistency is probably preferable. No big deal, I was just curious if this was intentional.

  3. My comment doesn't count as a vote to keep the roles of consenters separate from committers. It's a suggestion to keep the functionality separate, and (along with comment 11) to generalise the protocol to cleanly cover a wider range of configurations and not have to specify what gets sent to whom and in what order.

  4. Yes I think that would be clearer.

  5. Sorry, yes I meant "batch" where I said "blob" (first instance in comment 20).

  6. Above typo notwithstanding, I stand by my comment that the sentence seems contradictory. (Minor nit, just trying to help improve presentation.)

22 + 26) Not sure what you mean. If it is "required", for whom does this create an obligation? At what granularity might one opt out if it were options? Per implementation? Per Network? Per peer?

  1. I emphasize "in such a way that they affect the hashes required to verify valid transactions" in my comment. I agree that making the consensus fabric process opaque blobs means that it has no option but to include invalid transactions in its output. That does not prevent committers [edit: this previously mistakenly said "consenters"] (who can determine that they are invalid) from maintaining a ledger representation that keeps them out of the way, thus facilitating easy pruning and fast sync for new peers, which is what I attempted to illustrate in comments 24-26.
@mm-hl

This comment has been minimized.

Copy link

commented Jun 3, 2016

Responding to @vukolic's 5 + 14):

First, I find it difficult to tell from the proposal what the role/purpose of endorsers is intended to be, which I think affects this conversation significantly, because it's hard to figure out what motivates what. Here are some of the reasons I'm struggling with this and some "reading between the lines" that may help me understand and you clarify:

  • From Section 2.2, we can see that the submitting peer is able to execute the transaction, implying that it has the "current" state, knows the chaincode, and has any keys or capabilities needed to execute the transaction if it is a confidential transaction. However, the third bullet of Section 3.1 appears to contradict this.
  • The first "Alternative design" in Section 2.2 suggests that the endorser doesn't need the tx payload, in which case it would not be able to execute the transaction, nor would it be able to verify that the state updates and verdep included in the transaction were correct. But that is what is depicted in Figure 1.

I note that the proposal does not require committing peers to be able to execute transactions. My comments were coming from a perspective where it would be a reasonable for them to be able to do so. Perhaps avoiding this is a key motivation for the proposed design? I can see potential advantages including:

  • optimizing throughput for committing peers
  • allowing chaincodes to be confidential (not known by committers); even though committers are aware of keys & values read and updated, these may be encrypted by the chaincode so that they would be opaque to committers. This would seem to go some way towards addressing one of the concerns I had based on what I understood from the original protocol-spec, namely that all validators would need keys to execute confidential chaincodes, which may be unacceptable in many use cases.

If I reread your response with an assumption that avoiding committers executing transactions is a/the key motivation, it makes a lot more sense to me, so I think I am understanding the proposal and its intentions better now. If this is correct, the proposal would be much improved by upfront discussion of this motivation, probably in Section 1.3. I'll proceed with the assumption that I've got that right.

First, wouldn't it make more sense that the submitter would send the txpayload to endorsers, and the endorsers would execute the chaincode and construct the transaction? That way, the submitter would not need to be able to execute the transaction, thus supporting confidentiality. Presumably endorsement policies would (usually) require that all endorsers produce the same transaction; otherwise, what would make sense to do?

This raises a question: against what state do (submitters and) endorsers run the transaction? Even if chaincodes are deterministic and all participants are honest, they may end up with different state updates and verdeps if they run against different states. Maybe the submitter should nominate a state to run against, and endorsers could endorse only when/if they have caught up to that state?

The key advantage of obstruction-freedom is that it allows us to decouple synchronization from scheulding/contention control, while admitting much simpler mechanims that are required for lock-freedom and wait-freedom. But obstruction freedom without effective contention control is usually a disaster. I vaguely thought about this when considering the retryFlag. Is that its intended purpose? If so, have you thought much about what kinds of retry policies could be implemented and would make sense? Presumably endorsers would be in the best position to implement retry policies, and thinking about this in context of your comments about Eve starts to maybe make some sense. For example, endorsers could potentially work together (perhaps, but not necessarily, via a leader) to combine transactions from the same chaincode into orders or batches that might somehow be processed more efficiently by committers, perhaps somewhat analogous to the "mixer" in Eve. This might be mostly internal to endorsers, but the protocol might help by enabling endorsers to communicate ordering/batching dependencies to submitters (or directly to the consensus fabric).

Coming back to the potential motivation for reducing transactions to state updates and version dependencies pre-consensus, it may help to improve throughput for committers, but:

  • Amdahl is Amdahl and has a pesky habit of being right, so this only pushes the threshold out, it doesn't achieve trule scalability, for which we will need sharding and ideas beyond that.
  • Perhaps it pushes the threshold out considerably and is worthwhile, but I wonder about the baseline assumptions. If we're thinking of chaincodes needing to be executed in Docker containers, etc., this suggests that checking version dependencies and applying state updates would be substantially faster. But is that baseline set in stone?
  • What about other models, ranging from something like the Ethereum approach, to templates or DSLs, or whatever that can remove the need for sandboxing and/or termination enforcement? Might these change the equation significantly? Would it be a goal of the project to be flexible enough to support pluggable committers that support different assumtions about transaction execution, etc?
@mm-hl

This comment has been minimized.

Copy link

commented Jun 3, 2016

One more small comment on terminology. I find it slightly strange in Section 1.2, bullet 2.a that a committing peer "commits the transactions and maintains the state" but is called a "read-only peer" (if is not also playing other roles). I get that the intention is that it can only support read-only queries from clients if it does not play these other roles, but nonetheless the terminology seems slightly strange to me given that it is maintaining the state and applying updates to it.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 3, 2016

@ mm-hl quickly re 5+14)

The idea of HASH(txPayload) in alternative design in 2.2 is misplaced - HASH(txPayload) (or tid) should be sent to consensus service in 2.4 for confidentiality, so basically tran-proposal cannot be reused from 2.2 to 2.4. PROPOSE message (Sec 2.2) HAS TO have full txPayload sent to endorsers for - indeed - execution of the transaction. I will make this clear asap since this is critical.

[EDIT: this is now fixed by having tran-proposal containing HASH of the payload, while PROPOSE contains txPayload, explicitly and outside tran-proposal.]

As for your remarks on confidentiality - precisely - this is one of the main motivations of this approach. I am preparing an extension of the document that will deal with confidential chaincode examples in this architecture so things should become more clear then.

As for synchronization for obstruction freedom - the expectation here is that the data model will permit less synchronization (e.g., "spend-once" UTXO). If this turns out not to be the case - we will need synchronization (e.g., leader) in the vein of Sieve/Eve - perhaps only for state partitions that hold such objects.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 3, 2016

@ mm-hl re 9 + 23)

" That does not prevent consenters (who can determine that they are invalid) "

In fact consenters - in the current proposal - have no way to determine whether a tx is invalid or not. This can be only done by (committing) peers.

The design is currently general as in - it separates consenters from (committing) peers. The main motivation is to allow easily pluggable generic consensus service that do not have to care at all about chaincode state, verifying endorsement policies and the likes. Such a consensus service could inherently be as scalable and performant as possible as execution/validation is totally removed from the critical path.

The cost of this genericity and modularity is the fact that consenters (consensus service) cannot tell valid and invalid transactions apart. This needs to be done at the (committing) peer level. If this is a show-stopper, it is fairly easy to specialize this architecture to have consenter >= committing peer. This would come at the expense of sacrificing advantages listed above.

Besides 5+14 - this is another big design decision that we as HL community need to take.

@corecode

This comment has been minimized.

Copy link
Contributor

commented Jun 3, 2016

Including stateUpdate and (more importantly) verDep in the PROPOSE message to endorsers allows endorsers to be (internally) sharded on the chaincode state keys.

@cca88

This comment has been minimized.

Copy link

commented Jun 3, 2016

@mm-hl: Thanks for your comments and the discussion.

Re 13) Your text says what was meant, this is better.

Re 14) Yes, this would essentially be a transaction manager running together with consensus on blockchain.

Re 26) The proposal mentions this, it's the optional "(Blockchain) State" data structure that can be part of the hash chain, included through the root of a hash tree on the state, just as you suggest in Ethereum. Makes a lot of sense as an optimization, not strictly needed for security. We will need to decide whether to have this.

@mm-hl

This comment has been minimized.

Copy link

commented Jun 3, 2016

@vukolic Sorry, I caught myself a number of times saying "consenter" when I meant "committer" but obviously didn't catch them all. Yes, I totally understand and agree, and my sentence was about what consenters store, nothing to do with committers.

@corecode True, but at the expense of confidentiality from the submitter, since it has to execute the transaction in order to determine the stateupdate.

@cca88 The "(Blockchain) State" mentioned in the proposal is described simply as a KV store. I don't see anything that suggests it's stored as a Merkle structure, or that its Merkle root could/would be included in checkpoints, which is what is needed for supporting fast sync.

Also @cca88 I don't get your comment about 14). Are you agreeing that different consistency conditions for different chaincodes could make sense and commenting on how this would be implemented?

@cca88

This comment has been minimized.

Copy link

commented Jun 3, 2016

@mm-hl: Right, this was confusing: I thought that optionally including the blockchain state also meant to include it inside 4.2 as part of the hash chain. (That hash chain now is built over the VL.) Will clarify this option.

  1. Yes, agree. IMHO the design moves towards a transaction manager (TM) being implemented inside the logic for applying the updates; the TM ensurs a chosen consistency condition (isolation guarantee). The conditions couldn't be implemented by chaincode itself, only hard-coded ones in the fabric seem possible. Chaincode deployment can pick the isolation guarantee that it wants and which programmer understands. There is a line of literature that has investigated how to run a replicated DB without a central point of control (over a reliable group communication system, tolerating crashes only) by Kemme, Jimenez-Peris, Patino-Martinez, Alonso, Schiper and others. It is summarized in a textbook (Database Replication, Morgan & Claypool). When extending the approach here to full generality, it would become the equivalent of their "database replication" but in the BFT model.
@tock-ibm

This comment has been minimized.

Copy link

commented Jun 6, 2016

Issue: committing peers can change the order of blobs from consensus.

The question of Byzantine committing peers was already raised.
However, while it is true that a Byzantine committing peer cannot alter the content of a blob provided by the deliver() event of the consensus, it seems to me that it can change the order between blobs.
That is, the deliver provides {sequence-num, blob, hash}; anyone can compute a new hash for a different {sequence-num', blob} pair. For the ordering provided by the consensus to be resistant to forgery it must be signed by some entity, not just hashed.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 6, 2016

@tock-ibm

Byzantine committing peers can indeed do what they want with their raw ledger producing a hashchain that verifies but is bogus. However, this is not an issue, since such Byzantine peers cannot convince any honest peer of their chain. Namely, in the State transfer and checkpointing protocol (Sec 5) an honest peer gets the head (tip) of the raw ledger hashchain from the consensus service and then it resolves it back via peer-to-peer comm.

Now, whether consensus service delivers multi-signed batches to peers or it delivers via f+1 consenter confirmations - this is up to a consensus service implementation.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 10, 2016

A subset of comments above is included in the new document revision. Notice that this does not mean that other comments will not be taken into account.

Notable change is that an endorsing peer now maintains the state. As a result, there are no committing peers any more, but simply peers, which can have additional roles of submitters and endorsers.

@mihaigh

This comment has been minimized.

Copy link

commented Jun 11, 2016

@vukolic transaction flow should be updated after this peer unification

@mm-hl

This comment has been minimized.

Copy link

commented Jun 13, 2016

These comments are based on this version.

  1. Regarding the changed definition of submitting peers (Section 1.3.2.a), this appears to address the confidentiality issue in previous versions, because submitting peers accept transactions pertaining to specific chaincodes, so submitting peers for a given chaincode can be a subset of endorsers for that chaincode. However, it also puts the burden on clients of knowing which submitting peers can accept transactions for which chaincodes. This means that clients would need to manage issues like figuring out an alternative submitting peer in case the one they have previously remembered is unavailable or misbehaving.

An alternative would be to keep the previous structure of allowing any submitting peer to accept transactions for any chaincode, but to remove the responsibility it previously had to execute the chaincode. Instead, it could identify a "lead" endorser, which would execute the transaction, produce the proposed read/write sets, and request additional endorsers to verify correct execution and endorse the transaction. The submitting peer would only need to wait for a sufficient set of endorsements to show up. (What it does if they don't is a separate issue; maybe it times out and picks another lead endorser.) The submitting peer could inform the client which lead endorser was successfully used, allowing the client to "cache" the result for optimizing subsequent transactions for that chaincode, but would not require such caching or require it to be up to date or consistent. I think this arrangement would better fit the description of "providing an interface between clients and the blockchain", as it removes unnecessary burden from clients. (Also, it would make the "stateless" and "any peer" advertising in Section 1.3.i. more accurate.)

  1. Section 1.3.ii seems repetitive coming so soon after the introductory material in Section 1.3. Furthermore, it isn't quite consistent with the earlier material. For example, the "pertaining to a particular chaincode" part of submitting peers that I addressed above is not reflected here.

  2. In Section 1.3.iii, I think the specificity of "for transactions and state updates" is potentially misleading and confusing, given that the consenters themselves know nothing about the opaque "blobs" they are ordering.

  3. In the second paragraph of the same section, "reliability" is mentioned, but what is described thereafter is really only about atomic broadcast. There is no stated or implied guarantee that all messages offered will eventually be delivered, or anything else that I would consider as "reliability" guarantees. Maybe just say "different implementations may offer different reliability guarantees" or something like that? (Now I see that this is addressed better in later text, so maybe just add "implementation-specific" here.)

  4. Also, given that you're going to the trouble to accurately describe what the "consensus fabric" does (atomic broadcast), why not take the opportunity to excise the common abuse of the word "consensus", maybe putting a footnote to avoid confusion for people who are accustomed to (mis)using "consensus"? Maybe even go so far as to rename consenters and consensus fabric, for example to broadcasters and broadcast fabric? But maybe the terminological abuse is widespread enough that there is so no point resisting it.

  5. I think the remark in Section 1.3.iii is too vague and speculative to be more than a distraction.

  6. I think the Safety guarantee paragraphs would be improved if more precise properties were stated first, and then intuitive summaries and observations included later if they are still needed, rather than starting with imprecise descriptions and then refining them with "this means", "note that", "put differently", "in other words", etc. How about this (to encompass both safety guarantees):

There exists some sequence M consisting of messages m_i = (seqno_i, prevhash_i, blob_i), i=0,1,...:

  • for all i, seqno_i = i
  • the sequence of deliver() events at each correct peer is always a prefix of M
  • for all i, some consensus client calls broadcast(blob_i)
  • prevhash_0 is some default value
  • for i > 0, prevhash_i = HASH(i-1,prevhash_{i-1},blob_{i-1})

What this doesn't address:

    1. It doesn't preclude "out of thin air" blobs, i.e., it doesn't disallow delivering a message containing a blob that hasn't yet been broadcast by any client. Seems unimportant for context in which blobs are expected to contain signatures, etc.
    1. It doesn't preclude the same blob being delivered multiple times, which the existing text seems to imply. However, given that consenters are oblivious to blob contents, does it make sense for their correctness to depend upon this requirement? I think the issue here is really about assumptions on what clients broadcast. A faulty client might attempt to rebroadcast a blob, and I think it makes sense that the consensus fabric will deliver the blob again in this case. The committers will figure out what's going on due to nonces, etc., and reject the duplicate transaction(s), but this is nothing to do with the consenters. It could make sense to require the existence of witness functions that ensure that for each i such that blob_i = v, there is a unique broadcast(v) event. Or more pragmatically, just don't bother specifying this requirement, and leave it to committers to deal with it, which they have to do anyway. This might deviate slightly from more traditional notions of atomic broadcast though.
  1. Regarding checkpoint validity policies, while it is alluded to in an example in a sub-bullet, I think it's worth pointing out explicitly that a weaker checkpoint validity policy (or combination of local checkpoint validity policies) can undermine fault tolerance properties because it may cause a correct peer to behave "incorrectly".

  2. In Section 5.2.i, I suggest s/blocknohash/blockhash/g as the hash is for the block, not the block number, right?

  3. I still don't find the checkpoint and state transfer description very convincing or compelling; the previous comments still mostly apply.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 13, 2016

@mihaigh - fixed
@mm-hl

27+28) The intention was to have submitting peer able to act on any transaction, except those pertaining to confidential chaincodes (that section will appear soon). The text now consistently reflects this.

As for the leader, I would like to treat this separately. After some discussions - and following your concerns as well as following concurrency and programming model impact of version dependencies raised by the community - the idea is as follows: offer both the option of leaderless verDeps (MVCC) as the text now stipulates + leader-based approach. Leader would be elected per chaincode, with the assistance of the consensus service.

While chaincode could implement its own leader election module on top of consensus (total order broadcast) - this may be sub optimal and very complicated to the chaincode developers, so the idea is to have the fabric support for few typical leader implementations.

Notice that chaincode could opt for leaderless variant (default) or some leader-election policy built in the fabric, or simply implement its own leader election.

Notice also that in such an approach, current leaderless MVCC is just a special case of leader function implementation, in which leader election chaincode deployed at every submitting peer would be

leader(chaincodeID)
return myID

For actual leader implementation that leader implementation would look at the blockchain state to determine the current leader and act appropriately (per leader election policy/chaincode) when necessary to help elect the leader.

I will work out the first draft along these lines and submit - so we can discuss further.

29-32) fixed

33-35) Agree, this is TBD properly. Notice that 34 and 35 are addressed in prose already.

36-38) Changes are pending to the checkpointing mechanism to include the state hash. Will post when ready for review.

@mm-hl

This comment has been minimized.

Copy link

commented Jun 13, 2016

@vukolic thanks for the responses.

Regarding 27+28, please note that I was not suggesting a different or additional approach to concurrency, which I agree is a separate topic. My point was only regarding whether a submitting peer could accept a transaction for a confidential chaincode for which it does not have the ability to execute the transaction. The key part of the suggestion is: identify a "lead" endorser, which would execute the transaction, produce the proposed read/write sets, and request additional endorsers to verify correct execution and endorse the transaction. All of that assumes the MVCC model, and just addresses who has the ability to execute and prepare the MVCC-style transaction.

Although I used the word "lead" in "lead endorser", the suggestion does not entail or require any form of agreement or election, and thus none of the complexity you discussed in your response. The "lead endorser" could instead be called the "initiating endorser" and could be chosen at random or via any other policy. A different choice could be made per transaction. Indeed, a submitting peer could even choose to send the "initiation request" to multiple endorsers, again chosen randomly or otherwise, and then wait for a sufficient set of endorsement responses to satisfy the endorsement policy, which might or might not all derive from the same "initiating endorser".

The main change I am suggesting is regarding step 2.2 in the transaction flow: The submitting peer prepares a transaction and sends it to endorsers for obtaining an endorsement. This is what puts the burden on submitting clients of knowing which submitting peers to target for which chaincodes. It sounds like you're preparing more detailed treatment wrt confidential chaincodes, so I will try to review again when that appears.

Thanks for the good work!

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 14, 2016

@mm-hl

ack - the incubating confidentiality draft is precisely doing sth similar - that one will appear for review/comments/discussion soon (will of course post a notice here).

This does not change the story re leader in the context of concurrency as discussed in my previous comment. We agree this is a separate issue and will be treated in the separate section of the proposal - so we can discuss it then in more details.

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2016

I wanted to discuss here on the aspect that causes most departure from the current architecture - which is transforming transaction from the original form (issued by a user - invoke a chaincode method) into a form that includes state updates. Because, this is what requires new functions such as endorsers and committers, new artifacts such as read-write set/verDep, and also changes the transaction execution flow significantly.
At the first sight, to me it appears that the main motivation is the Confidentiality as the other advantages such as separating the consenter nodes are more about separating the functions performed by a single peer in the current architecture.
So, is the main motivation to allow deployment of a chaincode to specific set of peers and still to maintain a single global ledger (transactions and state)? If yes, then why not instead have a separate ledger altogether for each such confidential chaincode (or trust group). The peer that does not have the chaincode anyway can not operate on the corresponding state and moreover it may be be confusing to allow the unrelated peer the access to state data but not the chaincode.
Allowing non-determinism in the chaincode may be another side-advantage but I am not certain whether that could be the main motivation. Similarly, agreeing on state updates beforehand could be another but I guess checkpointing also serves the similar purpose.

So, basically, I wanted to find out the main motivation of including the state updates in the transaction definition which causes a significant change in flow.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 15, 2016

@manish-sethi

The high level motivation is to simplify fabric implementaiton. This motivation is broken down as follows (w/o specific order): 1) handling non-determinism, 2) allowing more parallelism in chaincode execution (endorsement), 3) providing a simple mechanism of ensuring that a transaction is never executed more than once.

  1. Handling non-determinism is an extremely important motivation, esp. when we replicate code coded in high level language such as golang/Java/etc. With model w/o state updates (taken, e.g., by the Sieve protocol - http://arxiv.org/abs/1603.07351 - which is the prototype in the current HL fabric) there is always a case in which a non-deterministic would appear as deterministic during execution and, as such, committed to the ledger. However if one only logs the transaction payload of such a transaction and not its state (updates), a replica repeating the transaction execution sequence (e.g., a new peer) could end up executing the transaction payload and end up in a divergent state (due to non-determinism). This can be of course handled through state transfer instead of sequential execution - but it complicates the system. Hence the choice for state updates which yield a simpler (fabric implementation-wise) way to deal with non-determinism at the fabric level.

(as a side note: state updates are directly applicable to Sieve as well, and would turn Sieve into leader-based protocol for handling non-determinism with state updates)

  1. As for parallel execution, state updates + version dependencies are nice, since they allow leader-free parallel execution. This is not the only way to implement parallel execution (which can be also leader based, cf. Eve - OSDI'12) but again, it appears as a simpler one.

  2. Further motivation is to simplify the implementation with respect to transaction execution as with state updates (and version dependencies) one cannot "accidently" execute the same transaction twice (see ZAB paper on Zookeper atomic broadcast by Junqueira et al. DSN'11 for discussion on this one).

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jun 16, 2016

@vukolic thanks for highlighting the though process behind the proposal. To discuss these further, can you have a look at the following.

  1. For the case of a new peer joining, we can always include the state updates produced by the transaction payload into the block when we execute the transactions in the current architecture. In other words, a block in the blockchain can still look same as it appears in the endorsement based model. However, across live replicas we would rely upon the checkpointing which I think is the case even in the endorsement based model (please correct me if I am wrong here).

  2. About parallelism, the proposed approach primarily tries to compensate the cost of grpc communication between a chaincode and peer (assuming these are not high CPU consuming transactions) at the cost of additional overheads which include signing and collecting the endorsements, performing the additional ledger reads (one during simulation and another during commit - for version matching), roll backs in the case of version conflicts. I am sure that these overheads may be compensated for large networks where endorsement is required from a significantly smaller subset of peers. However, for small-sized networks where a larger subset is involved in the endorsement, the final performance may be poorer because of these overheads. I am just concerned and wanted to know whether we have some measurements of these costs and some sense at what configurations (e.g., network size and ratio of endorsers to the network size) they start showing benefits.

  3. (a) In past, I was thinking of a simple approach for allowing parallelism in the current code base. Let me write it briefly here and run this by you to see if this make sense. Below is a rough functioning of this
    Execute transactions in a block in parallel and maintain a map {k -> TxId} where a transaction TxId reads / modifies the key k. If there is no conflict (against each key only a single TxID entry is present) at the end of batch execution, commit the transaction results. In the case of higher number of conflicts, fall-back to sequential execution. Further, if there are a smaller number of conflicts some optimization can be applied e.g., rolling back and re-executing conflicting transactions in the order of relative order they appear in the block (there are corner cases here but I think - not critical to discuss right now). Let me know if you think that this could be made leader-free because of dynamically monitoring the conflicts and relying on checkpointing at a later stage? Upto a certain network size, some simple parallelism like this may result in better performance. But I am not sure if both the approaches can be merged in a single architecture and employ one vs the other based on deployments.
    As a side note, I believe that the above mentioned approach of dynamically monitoring is orthogonal and could be included in the endorsement based architecture as well where an endorser collects and executes the the endorsement requests in parallel.

  4. I haven't read this paper but did not understand the comment of executing transaction twice by mistake. Is it just about adding a check of looking up txid in the blockchain before executing again or something more than that?

@roger505

This comment has been minimized.

Copy link

commented Jun 23, 2016

Section2.5, 2nd paragraph:
Reading the description I'm wondering if proposed method covers all possible isolation levels.

If I consider a scenario where two transactions (A , B) are entered and part of the chaincode business logic is to see if the other transaction exists and if it is to mark each as matched. The chain code is implemented by storing each of A and B as a key value pair with keys A and B.

When item A executes then it searches for B but does not find it.
If item B executes then it searches for A.

If the consensus model only consider the read variable and the updated variables then if A and B each executed on versions of the world state that don’t contain the applied state of the other transaction, and then the state updates are combined in the same block, it is possible for both A and B will be entered into the system in an unmatched state.

The problem is clearly the “search” for items that don’t yet exist / are not committed.

One solution would be to partially serialise execution. So in the example above then it might be possible to insist that these transactions are executed serially on the same endorsing peer, however this would need endorsing peers to execute subsequent transactions against the proposed world state rather than against the committed world state.

It this the proposal in 2.3?

Another solution would be for the application to store A and B under the same key, say AB, however this imposes restrictions on the programing data model and moves to a model of few keys with large blobs of data containing internal structure, which may itself introduce performance problems.

A final solution might be to record search criteria along with keys read or written and then in the consensus model to see if any keys have been entered into the range search between execution and commit.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 23, 2016

@roger505 I am not sure I understand the issue

your "chaincode" seems to have following two tx
A) A=1; if B==\bot then executeSomeWinnerCode
B) B=1; if A==\bot then executeSomeWinnerCode
i.e., if they execute concurrently on the initial state we have
A.readset={(B,\bot)}, A.writeset={(A,\bot)}, A.stateUpdate={(A,1)}
B.readset={(A,\bot)}, B.writeset={(B,\bot)}, B.stateUpdate={(B,1)}

as consensus orders one transaction before the other, the first transaction (per raw ledger order) is valid and committed (regardless of isolation level). Notice that there is an order among tx in a batch, even if they belong to the same batch.

In case of serializability the second (per raw ledger order) tx is invalid. Under some other isolation level (e.g., SI) the second tx could be valid as well.

let me know if I am missing sth

@mm-hl

This comment has been minimized.

Copy link

commented Jun 24, 2016

@manish-sethi yes, there could be many strategies for optimizing execution of a batch of transactions. The only requirement would be that they are executed "as if" sequentially, which is what's required to avoid the complexity (e.g., in Eve) that arises from multiple honest peers getting different results for the same batch.

@roger505

This comment has been minimized.

Copy link

commented Jun 24, 2016

@vukolic I agree that the example you quote will be safe, however in the use case I was thinking of the matching is done with a range search rather than an explicit read as there is a "matching tolerance". How would the code represent a "RangeQueryState" where no key is found ? I agree that in some isolation levels this would not be protected against, I was just keen understand if in this design it is possible to configure the system so it is safe , even if that is at the expense of some limitations such as forcing potential problem transactions to have the same endorsers.

@learner4

This comment has been minimized.

Copy link

commented Jun 29, 2016

My thoughts - would it be good to consider that Identities involved in transaction be verified during this process?
If consensus and endorsement is done on transactions, the transactors involved should also be verified in a regulated environment. Endorsing the "content" of the transaction may be only a part of the solution where endorsers would also need to validate the Identities; just as an easy example: OFAC checks

@roger505

This comment has been minimized.

Copy link

commented Jun 29, 2016

Is it possible to explain the confidentiality model further please?
I notice that the tran-proposal contains the spID and the clientID, and that the tran-proposal is included in a blob, and the blob is included in the chain.

Would the spID and clientID not reveal something about the submitter of the transaction?

How does this fit with transaction certificates to obscure the origin of the transaction?

@vukolic

This comment has been minimized.

Copy link
Author

commented Jun 30, 2016

@roger505 more details on the confidentiality are following... Pls stay tuned.

UPDATE: In the meantime clientID is gone as actually not necessary. In principle spID should not reveal anything with respect to the client nor leak any useful info. It is however useful, as it denotes a peer who computed transaction results and may reveal useful for accountability (e.g., peer submitting transactions with invalid endorsements) as well as to help uniquely identify a transaction.

@roger505

This comment has been minimized.

Copy link

commented Jul 1, 2016

@vukolic Sorry to jumped the gun.

In your confidentiality update, can you please consider if one possible deployment could be a one to one mapping between clients and submitting peer, in which case would the spID not reveal the clientID. Or do you regard the only supported configuration as being submitting peers having a one to many relationship with clients ?

@srderson

This comment has been minimized.

Copy link
Contributor

commented Jul 1, 2016

All - Please be aware that there is a recommendation that proposals should be moved from the wiki to the repository.

PR: https://github.com/hyperledger/fabric/pull/2082
Mailing list discussion: http://lists.hyperledger.org/pipermail/hyperledger-fabric/2016-June/000127.html

As this is one of the most active proposals, I figured people here may have opinions on the subject. Please weigh in on the mailing list if you have a preference one way or the other.

@JoVerwimp

This comment has been minimized.

Copy link

commented Jul 6, 2016

@roger505, @vukolic "Or do you regard the only supported configuration as being submitting peers having a one to many relationship with clients ?" so would that not move to a centralised model? i.e. where the submitting peers are centralised and servicing multiple clients. Since the submitting peer needs to prepare the transaction, it needs access to all data (possibly including off chain confidential data, including coordinating the updates in off-chain and on-chain data stores).

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

@srderson, @vukolic, @corecode: moving the version related discussion here.
From 2.2 "All k/v entries are versioned, that is, every entry contains ordered version information, which is incremented every time when the value stored under a key is updated”
An alternate to the version is to use the ID of the previous TX that modified the key most recently.
Benefits of using previous transaction-id instead of a number based version are:

  • This does not require introducing a new concept (auto-incrementing version) at the protocol level. It would conceptually be closer to bitcoin data model where previous-transaction-id is similar to input and newValue is similar to UTXO for the transaction.

  • Alone version info may not be sufficient in the case of a consensus protocol that allows forks. For example - when using PoW with endorsement based approach, think of a scenario, where a transaction was endorsed on fork A by all the endorsers and committed by the committers on fork B. Accidentally, if both the fork A and fork B have the same version for a key, the transaction will get committed to fork B with unintended state.

  • At implementation level, the explicit mention of previous transaction points to a structured record at the storage level - an index on transaction-id could be reused for tracking the change history for a key

    However, the downside of using the transaction-id is that it would be significant bigger in size as compared to using a number for the version, leading to increased payload.
    If we finally chose the number based versioning, for the scenario highlighted in the second point above, the endorsement also need to carry the hash of the top block against with the transaction gets simulated (does it call out for a consensus specific info in the endorsement in general?)

Thoughts?

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

@vukolic: while moving the transactions from raw ledger to validated ledger, in addition to dropping the invalid transactions (because of versions conflicts), I think that we should drop the read set info from the valid transactions as well (to reduce the storage requirement).
Because, read-set info is mainly useful only for validation. Do you see any problem in this?

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

@vukolic - from section 1.2 "The state of the blockchain ("world state") has a simple structure and consists of a key/value store (KVS)"
I think that we should not restrict in the description to key/value store. Different implementation of ledger could support different data-models. At the protocol description level, we can be more general (e.g., a transaction simulation produces the info that is necessary for validation and making state changes). KV-store can be one simple data-model we can describe for illustration though.

My though process is that, if some one wants to manage state in a more complex data-model such as full-fledged relational database, though specifying stateUpdate in key-value form should be possible because of key-value being most basic in nature [e.g., one can represent tablename.rowId.columnName combination for the key] however, one may like to specify the state info in more compact form (say in a nested proto message in order to avoid repeating the table name if many rows in a table are updated by a transaction).

Essentially, I just prefer to leave the scope of representation of stateUpdate open ended and let the ledger implementation decide this. In fact, my earlier thoughts on using previous-transaction-id instead of auto-incrementing versions may also fall in this category. What are your thoughts on this?

@ijmitch

This comment has been minimized.

Copy link

commented Jul 6, 2016

I've tried to review the comments here - but there's quite a lot(!) so forgive me any duplication I'm about to create.

  • Would it be useful to settle on one of the terms validation or endorsement? Certainly a brief but specific definition of what either/both of these mean would help me. My working definition is something like agreement with another peer that execution of the chaincode against the current state has resulted in the same modifications to the state. I guess we're in a transition phase from one term to the other - it is explicitly stated "validating (i.e. endorsing)".
  • Is there any meaningful distinction between what is execution of a transaction in a submitting peer (first para of 2.2) and what is the simulation of the transaction during endorsement (2.3 first bullet)? (Other than one happens in a submitting peer and the other during endorsement.)
  • There seems to be a lot of implicit reliance on the disjointedness of state modifiable in a given transaction (ie the keys associated with the targeted chaincode). Could this be made more explicit?
  • For example, I presume that an endorser would report STALE_VERSION if it got a deliver from the consensus service with updates that have yet to make it to the submitter - so the assumption is a low rate of clashing transactions (deliver being the result of a recent transaction for the same chaincode from this submitter or another one).
  • I interpret this separation between endorsement and consensus being a separation of the integrity of the intended actions of the chaincode (the 'app') vs the replication of valid updates to database instances (of a ledger data structure). Taking this as a beneficial change.
  • The separation is so clean that I can't tell anything about how a consensus service would actually work - if you want me to think of it as a black-box then it's met that objective. If you want me to understand the system end-to-end, then that's not so good.
  • 3.2 is confusing me... second para is mixing up peers and consenters in a way I don't understand.
  • how different is a single blockchain with a partitioned consensus service differ from separate blockchains (except for the obvious sharing or not of the database instance)?

I'll post comments about state transfer and checkpointing separately.

@JoVerwimp

This comment has been minimized.

Copy link

commented Jul 6, 2016

@manish-sethi Managing the k/v versioning away from a mere integer does make sense for the arguments you mention. The transaction-id may indeed be longer but I do not believe that to be the problem currently (storage is cheap, right?).
Also I see no issue of stripping the transaction in the validated ledger of all unnecessary info, including its read set which is a subset of the write sets of previous transactions.
Your comment on 1.2 makes sense to me, although for now a simple KVS simplifies documentation & implementation.

@ijmitch I agree with your statement that it would be best to make the assumption explicit that the application should be such that conflicts in write sets and subsequent read sets are naturally rare. Applications that have constant updates to a limited set of data do not fit this paradigm.
From what I have read so far, endorsement is executing (simulating the execution of if you like) chaincode and checking (validating?) that with he same input, the same data is read from the world state & the same data is written to the world state is produced. Consensus is about agreeing an order of transactions in a block and detecting conflicts in their (nor ordered) write sets later read sets. I do like the separation of concerns and the flexibility its buys you for deploying the consenters without having access to any storage (they are not peers; they become stateless).
On the comment around single blockchain with partitioned consensus vs. separate blockchains: a single blockchain would always apply the blocks (and successful transactions therein) to the entire blockchain, even if the consensus was reached with contribution from a subset of the consenters (depending on the selected consensus algorithm). Separate blockchains would (currently?) not be interacting (although with this proposal you could imaging they can use the same contenting nodes without many updates...)

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

@JoVerwimp - about 1.2, I have no issues in the details highlighted using a KV-data model. However, what I prefer is an explicit agreement on the supporting any ledger (data-model) implementation and reflecting that in the write-up - may be just by adding a couple of lines that highlight the intent. In the current writing, it is not very difficult for someone to assume that KV-store is the only data model supported - which may lead to confusion (or wrong assumptions) between folks implementing consensus protocols and implementing a ledger for a data-model.

@ijmitch

This comment has been minimized.

Copy link

commented Jul 7, 2016

Since it's been mentioned, I would very much support moving proposals such as this from a scheme of using a wiki page(s) and a single issue for discussion to a set of docs under change control which we can fork, have more focused issues on, and submit pull requests for suggested updates to.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jul 7, 2016

@manish-sethi

  1. version numbers: You already know my view on version numbers and they are summarized in your excellent comment. In a nutshell, the question boils down to the following: are we willing to pay a price for future-proof support of POW-style forked raw ledgers by accepting the overhead in the size of the raw ledger (more specifically, version numbers). Note that this overhead is not needed if raw ledger is a single, no-forks, chain. I prefer letting other community members voice their opinion here.

  2. "while moving the transactions from raw ledger to validated ledger, " This is true, yet the same argument applies to the writeset, not only readset (this is less obvious since our implementation, both v0.5 and "v2", is having writeset and stateUpdates combined). For example, if version numbers are monotonically incresing integers, keeping writeset in the validated ledger also does not contribute to much, as one could obtain them by simply counting - in validated ledger history how many times a given key changed. In this sense, readset actually contains more information over stateUpdate than writeset. In the end, the "right" answer will depend on how much "history-tracking" we want to facilitate in validated ledger.

  3. re comment on 1.2 - I fully agree, yet the intention was to have 1.2 as a spec that is (trivially) implementable with key-value stores but also with RDBMSs and sth third. It was also added in response to a previous comment that was arguing against the "vague" description of state. I propose to explain this point in English in Section 1.2. and to suggest that implementing this state with sth else than actual KVS is perfectly possible and perhaps remove currently unused KVS-like API calls (list/delete).

@vukolic

This comment has been minimized.

Copy link
Author

commented Jul 7, 2016

@ijmitch

  1. Would it be useful to settle on one of the terms validation or endorsement?... I guess we're in a transition phase from one term to the other - it is explicitly stated "validating (i.e. endorsing).

Precisely so (transition period). Will try to state that once and then use "endorsement".

  1. Is there any meaningful distinction between what is execution of a transaction in a submitting peer (first para of 2.2) and what is the simulation of the transaction during endorsement (2.3 first bullet)? (Other than one happens in a submitting peer and the other during endorsement.)

In principle there is no difference (same chaincode is executed, with same transaction payload), except that the result of the execution needs not be the same, for different reasons including but not limited to: a) non-deterministic executions - including calls to local off-chain state, b) executing different code branches depending on say own endorser id.

  1. There seems to be a lot of implicit reliance on the disjointedness of state modifiable in a given transaction (ie the keys associated with the targeted chaincode). Could this be made more explicit?

This is implicit as we felt hard coding this may be premature until we start talking about cross-chaincode transactions. In principle there are at least two approaches here
i) partition the state across chaincodeIDs, but allow chaincode invoking other chaincode. We need to facilitate cross-chaincode transactions.
ii) Have chaincode declare some variables as "private" and some variables as "public" where "public" variables could be modified by any chaincode.

ii) seems more difficult to manage than i) hence there is implicit bias to i). But not explicit one yet.

  1. For example, I presume that an endorser would report STALE_VERSION if it got a deliver from the consensus service with updates that have yet to make it to the submitter - so the assumption is a low rate of clashing transactions (deliver being the result of a recent transaction for the same chaincode from this submitter or another one).

Precisely. That said, and also as discussed in previous responses to @mm-hl, the plan is to add, in future, some ability to chaincodes to reason about the concept of a leader that would handle the higher rate of clashing transactions.

  1. The separation is so clean that I can't tell anything about how a consensus service would actually work - if you want me to think of it as a black-box then it's met that objective. If you want me to understand the system end-to-end, then that's not so good.

This is actually on purpose so I am glad that the objective is met :) Now, for internals of the consensus service, these will look very much like classical consensus protocols (be them Byzantine fault tolerant (BFT) or crash-fault tolerant (CFT), e.g., PBFT, paxos, raft, etc.) augmented with Kafka-like (pub-sub like) notification of all peers of the new raw-ledger block deliver() event and hashchain that puts together consecutive deliver events. Implementation details are omitted as the goal was to specify the API of the consensus service, not to mandate how the implementation (and the fault model) will look like. In principle, we wanted the spec flexible enough to easily accommodate in future not only BFT/CFT "classical" consensus but also PoW fork-style consensus protocols (although, strictly speaking, the proposal - as it currently stands - does not allow PoW). Hence - not too many details of the box internals.

  1. 3.2 is confusing me... second para is mixing up peers and consenters in a way I don't understand.

Will have a look.

  1. how different is a single blockchain with a partitioned consensus service differ from separate blockchains (except for the obvious sharing or not of the database instance)?

This is related to 3) and will be more obvious once we spell out cross-chaincode tx.

@manish-sethi

This comment has been minimized.

Copy link
Contributor

commented Jul 7, 2016

@vukolic about your point 2) above - yes write-set also belongs to same category as read-set. I just wanted to confirm with you that you are OK about a transformed representation of transaction when moving from the raw ledger to the final ledger from consensus point of view and let it decide by the different ledger implementations.

In essence, I wanted to check whether you are fine with the consensus being oblivious of the following:

  • Representation of the simulation results in the blob (to support different data models and different versioning techniques such as previous_tx_id vs numbers)
  • transformation of transactions when moving from raw to validated ledger (as long as they have sufficient details for reproducing the state without executing the transactions)

Because, both of the above are produced and consumed by a specific ledger implementation and wanted to make sure that consensus is not dependent on these in any manner - so as to the specific details of these can be moved to the ledger discussion page (just trying to get to a better separation of concerns between consensus and ledger).

To take a specific example of version info representation - if a ledger implementation supports forks and wants to maintain numbering based versions, it can include the hash of the top block in the simulation results for endorsement and later validation. Similarly, another ledger implementation (or merely a configuration parameter in the same implementation) may decide to use previous tx_id. So, potentially we can discuss these things on ledger page.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jul 7, 2016

@manish-sethi indeed, consensus couldn't care less what would one do transforming raw ledger to validated ledger. In any transformation however (not only the one you suggest), the raw ledger hashchain is "broken" so it needs to be re-established by the validated ledger (e.g., in a way proposed in Sections 4 and 5). We can carry on the ledger page.

@ijmitch

This comment has been minimized.

Copy link

commented Jul 7, 2016

@vukolic : thanks for the comprehensive replies.

For my 5), I'm glad that objective is met too. In particular, I was looking at deliver() being an opportunity to integrate what we might have been looking for from the event framework. Since I'm looking at integration with existing systems, particularly on z Systems, I'm wondering whether this makes it plausible to register with the consensus service solely to get deliver() calls to be informed of what's going on. If a consensus client works in this mode, never broadcasting an update, then it wouldn't actually need to maintain a ledger or KVS in the classic sense.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jul 7, 2016

@ijmitch sure, a peer can purely be listener, never submitting/endorsing transactions.

That said, such a peer may want to maintain a ledger for efficiency. For example, if such a peer crashes and recovers it would need to fetch state from other peers as if it was a new peer joining the network. Possible - but not sure it is practical.

@ijmitch

This comment has been minimized.

Copy link

commented Jul 7, 2016

@vukolic : my use case is that the listening peer is actually something like a gateway to the outside world which simply consumes the data on deliver() - there could be non-blockchain systems that are 'interested' in the transactions on which consensus has agreed. This needs some consideration of confidentiality and identity propagation, but deliver() is starting to sound attractive for SoR integration.

@elli-androulaki

This comment has been minimized.

Copy link

commented Jul 12, 2016

Hi, We just updated the new architecture proposal with a section referring to transaction (chaincode) confidentiality. The link to the new section (section 6) is this one:
https://github.com/hyperledger/fabric/wiki/Next-Consensus-Architecture-Proposal#6-confidentiality
In addition, more details were added to Section 2 on transaction lifecycle.

@vukolic

This comment has been minimized.

Copy link
Author

commented Jul 12, 2016

Also, the latest revisions contains updates to Sec.1.2 and Sec.1.3.3, in response to some previous comments here - notably from @manish-sethi (e.g., no 3) @ijmitch (e.g., no 3) and @mm-hl (e.g., no 33)

@ijmitch

This comment has been minimized.

Copy link

commented Jul 25, 2016

It's not obvious (to me!) from the new Section 6 how deployment of confidential chain code is limited to the submitting and endorsing peers - ie why committing peers don't see it. Perhaps it's that the deliver from the consensus service of the deploy transaction to the committing peers just asks them to write the encrypted blob to their copies of the ledger, whereas to the endorsers and submitters, it means do that and make the chain code operational - is that correct?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.