Skip to content
This repository has been archived by the owner on Jun 11, 2022. It is now read-only.

Upstreaming OBFT stuff #693

Closed
wants to merge 137 commits into from
Closed
Show file tree
Hide file tree
Changes from 134 commits
Commits
Show all changes
137 commits
Select commit Hold shift + click to select a range
922115a
Created new handlers `height` and `block_by_height` for the bridge, a…
vsubhuman Feb 15, 2019
becb365
Implemented `get_slotid` and `get_epoch_and_slot` for block date
vsubhuman Feb 15, 2019
4f62091
Replace `BTreeMap` with `LinkedHashMap` to iterate lookups in inserti…
vsubhuman Feb 15, 2019
7a5948e
Implemented the `block_location_by_height` function
vsubhuman Feb 15, 2019
29aa5cf
Added parameter `ordinal` to `Lookup` and implemented storing epoch-n…
vsubhuman Feb 15, 2019
f3c474f
Added storing unsorted offsets additionally in index files, and imple…
vsubhuman Feb 16, 2019
b8f2e12
Implemented new enum `IndexOffsetType` to reduce code duplication
vsubhuman Feb 17, 2019
3c1b6d5
Removed new function from deprecated type and renamed new function ac…
vsubhuman Feb 17, 2019
d044816
Removed `IndexOffsetType` logic and implemented storing unsorted offs…
vsubhuman Feb 17, 2019
b84f9cc
Implemented new type `BlockLocation::Offset` and split-up some logic …
vsubhuman Feb 17, 2019
e465d9c
Implemented `storage::epoch::epoch_read_block_offset` which reads N-t…
vsubhuman Feb 17, 2019
559daf1
Added new field `Storage.loose_idx` and implemented `Storage.build_lo…
vsubhuman Feb 17, 2019
25d5b33
Implemented functions to work with loose index and also added syncing…
vsubhuman Feb 17, 2019
3f6edd2
Extended `Storage.block_location_by_height` to check loose index first
vsubhuman Feb 17, 2019
217d99e
Fixed reference
vsubhuman Feb 18, 2019
a990857
Removed unnecessary `pub`s
vsubhuman Feb 18, 2019
85dabea
Fixed Iterator type spec in `write_offsets_to_file` function
vsubhuman Feb 18, 2019
c351032
Made `ChainDifficulty` great again. (With private `u64` field)
vsubhuman Feb 19, 2019
6577089
Made `FanoutTotal` great again. (With private `u32` field)
vsubhuman Feb 19, 2019
120949c
Fixed function naming to Rust conventions
vsubhuman Feb 19, 2019
76a3f47
Replaced `match` with `if let`
vsubhuman Feb 19, 2019
380f063
Fixed loose index initial construction order, and added checks for ne…
vsubhuman Feb 19, 2019
4fa16ff
Simplified integer casting in `block_location_by_height`
vsubhuman Feb 19, 2019
46d0b27
Updated `get_from_loose_index` to handle the case when requested heig…
vsubhuman Feb 19, 2019
89aadc3
Simplified match case in `block_location_by_height` to work with pote…
vsubhuman Feb 19, 2019
650544b
Changed asserts as loose index construction\update to validate it doe…
vsubhuman Feb 19, 2019
63c7472
Prettified some code working with `storage.loose_idx`
vsubhuman Feb 19, 2019
7142cb7
Fixed loose_idx entry destruct
vsubhuman Feb 19, 2019
b5dbbf2
Simplified `block_location_by_height` using the magic `?`
vsubhuman Feb 19, 2019
ae06de4
Simplified `block_location_by_height` even more with use of magic `if…
vsubhuman Feb 19, 2019
6c586b1
Removed use of `ordinal` in index and removed use of `Lookups`. Imple…
vsubhuman Feb 20, 2019
c4d3a01
Fixed styling errors
vsubhuman Feb 20, 2019
ea69bdb
Fixed storage-locking
vsubhuman Feb 22, 2019
ed17614
Merge remote-tracking branch 'remotes/upstream/prop-up-leaning-tower'…
vsubhuman Feb 22, 2019
152c3e9
Changed the terminal case check in `sync::get_unpacked_blocks_in_epoc…
vsubhuman Feb 12, 2019
998436b
Added new OBFT-specific comments to the `sync::get_unpacked_blocks_in…
vsubhuman Feb 12, 2019
1030315
Changed the epoch-number validation code in `verify_chain::ChainState…
vsubhuman Feb 13, 2019
2998e8c
Changed the epoch-number validation if-else in `verify_chain::ChainSt…
vsubhuman Feb 13, 2019
8f9e98f
Reworked the epoch-number validation if-else in `verify_chain::ChainS…
vsubhuman Feb 13, 2019
1ba3b3c
Added new field `chain_state::ChainState::last_boundary_block_epoch` …
vsubhuman Feb 13, 2019
0e2101f
Fixed last EBB check in `verify_chain::ChainState::verify_block`
vsubhuman Feb 13, 2019
d21ed19
Updated loop in `sync::net_sync_to` to be ready to epochs with no EBB
vsubhuman Feb 13, 2019
8a40915
Updated epoch-switch in `sync::net_sync_to` in `net.get_blocks`-callb…
vsubhuman Feb 13, 2019
19e85fe
Updated code in `sync::net_sync_to` in `net.get_blocks`-callback to c…
vsubhuman Feb 13, 2019
8f7aab3
Fixed failing checks
vsubhuman Feb 15, 2019
ce705a9
Added `.clone()` to value
vsubhuman Feb 24, 2019
7b8efe8
Merge remote-tracking branch 'remotes/upstream/prop-up-leaning-tower'…
vsubhuman Feb 24, 2019
ed71ab0
Introduced enum flags for callbacks to be able to continue or stop th…
vsubhuman Feb 22, 2019
778eeda
Changed `get_blocks` callback in `sync.rs` so it first validates the …
vsubhuman Feb 22, 2019
3c191bd
Fixed compilation problems
vsubhuman Feb 24, 2019
167b73a
Fixed style problems
vsubhuman Feb 24, 2019
85383cf
Added comment about possible rollback types
vsubhuman Feb 24, 2019
2a3d00a
Implemented rollback detection in two possible places
vsubhuman Feb 24, 2019
070d480
Merge remote-tracking branch 'remotes/origin/feature/block-by-height-…
vsubhuman Feb 24, 2019
9477cd2
Implemented rollback for case when more thank K loose blocks
vsubhuman Feb 24, 2019
a3cc46b
Implemented rollback for loose tail case
vsubhuman Feb 25, 2019
5a83de5
Extracted some common file-reading code in `epoch.rs`
vsubhuman Feb 25, 2019
aecb6e4
Implemented rollback for case when we are in packed tail
vsubhuman Feb 25, 2019
c4e91d8
Changed the terminal case check in `sync::get_unpacked_blocks_in_epoc…
vsubhuman Feb 12, 2019
cc3c1a8
Added new OBFT-specific comments to the `sync::get_unpacked_blocks_in…
vsubhuman Feb 12, 2019
158750b
Changed the epoch-number validation code in `verify_chain::ChainState…
vsubhuman Feb 13, 2019
b7ec9ba
Changed the epoch-number validation if-else in `verify_chain::ChainSt…
vsubhuman Feb 13, 2019
aa8220a
Reworked the epoch-number validation if-else in `verify_chain::ChainS…
vsubhuman Feb 13, 2019
afd9bd9
Added new field `chain_state::ChainState::last_boundary_block_epoch` …
vsubhuman Feb 13, 2019
fcc9b78
Fixed last EBB check in `verify_chain::ChainState::verify_block`
vsubhuman Feb 13, 2019
7be8a08
Updated loop in `sync::net_sync_to` to be ready to epochs with no EBB
vsubhuman Feb 13, 2019
78857ac
Updated epoch-switch in `sync::net_sync_to` in `net.get_blocks`-callb…
vsubhuman Feb 13, 2019
174d3ae
Updated code in `sync::net_sync_to` in `net.get_blocks`-callback to c…
vsubhuman Feb 13, 2019
fc6659b
Fixed failing checks
vsubhuman Feb 15, 2019
1be2dae
Added `.clone()` to value
vsubhuman Feb 24, 2019
b249c71
Fixed style checks
vsubhuman Feb 24, 2019
cf3b81d
Created new handlers `height` and `block_by_height` for the bridge, a…
vsubhuman Feb 15, 2019
0478b13
Implemented `get_slotid` and `get_epoch_and_slot` for block date
vsubhuman Feb 15, 2019
909d5f9
Replace `BTreeMap` with `LinkedHashMap` to iterate lookups in inserti…
vsubhuman Feb 15, 2019
47cdf8d
Implemented the `block_location_by_height` function
vsubhuman Feb 15, 2019
704d434
Added parameter `ordinal` to `Lookup` and implemented storing epoch-n…
vsubhuman Feb 15, 2019
818b296
Added storing unsorted offsets additionally in index files, and imple…
vsubhuman Feb 16, 2019
dc11956
Implemented new enum `IndexOffsetType` to reduce code duplication
vsubhuman Feb 17, 2019
157b0fa
Removed new function from deprecated type and renamed new function ac…
vsubhuman Feb 17, 2019
33aa84a
Removed `IndexOffsetType` logic and implemented storing unsorted offs…
vsubhuman Feb 17, 2019
c4c7c44
Implemented new type `BlockLocation::Offset` and split-up some logic …
vsubhuman Feb 17, 2019
1c04ce7
Implemented `storage::epoch::epoch_read_block_offset` which reads N-t…
vsubhuman Feb 17, 2019
374c951
Added new field `Storage.loose_idx` and implemented `Storage.build_lo…
vsubhuman Feb 17, 2019
53def45
Implemented functions to work with loose index and also added syncing…
vsubhuman Feb 17, 2019
a3d1429
Extended `Storage.block_location_by_height` to check loose index first
vsubhuman Feb 17, 2019
f9a815e
Fixed reference
vsubhuman Feb 18, 2019
0413df7
Removed unnecessary `pub`s
vsubhuman Feb 18, 2019
39bb751
Fixed Iterator type spec in `write_offsets_to_file` function
vsubhuman Feb 18, 2019
0e44280
Made `ChainDifficulty` great again. (With private `u64` field)
vsubhuman Feb 19, 2019
6753ede
Made `FanoutTotal` great again. (With private `u32` field)
vsubhuman Feb 19, 2019
ddad1f9
Fixed function naming to Rust conventions
vsubhuman Feb 19, 2019
8e091a0
Replaced `match` with `if let`
vsubhuman Feb 19, 2019
a6485a5
Fixed loose index initial construction order, and added checks for ne…
vsubhuman Feb 19, 2019
48dbf6b
Simplified integer casting in `block_location_by_height`
vsubhuman Feb 19, 2019
8d878c3
Updated `get_from_loose_index` to handle the case when requested heig…
vsubhuman Feb 19, 2019
8dcd6d5
Simplified match case in `block_location_by_height` to work with pote…
vsubhuman Feb 19, 2019
30e6581
Changed asserts as loose index construction\update to validate it doe…
vsubhuman Feb 19, 2019
02d5e94
Prettified some code working with `storage.loose_idx`
vsubhuman Feb 19, 2019
ae9892f
Fixed loose_idx entry destruct
vsubhuman Feb 19, 2019
73c8e78
Simplified `block_location_by_height` using the magic `?`
vsubhuman Feb 19, 2019
113bcb2
Simplified `block_location_by_height` even more with use of magic `if…
vsubhuman Feb 19, 2019
25fe837
Removed use of `ordinal` in index and removed use of `Lookups`. Imple…
vsubhuman Feb 20, 2019
bcae641
Fixed styling errors
vsubhuman Feb 20, 2019
f9496af
Fixed storage-locking
vsubhuman Feb 22, 2019
dd942a0
Merge `block-by-height-2` into `rollbacks`
vsubhuman Mar 20, 2019
8019fbb
Removed duplicated code
vsubhuman Mar 20, 2019
8324156
Extracted `perform_rollback` function
vsubhuman Mar 21, 2019
ce70808
Merge remote-tracking branch 'remotes/origin/master' into feature/blo…
vsubhuman Mar 21, 2019
c1562f2
Fixed min/max problem. Extracted common code into a function.
vsubhuman Apr 3, 2019
cc0869e
Codestyle
vsubhuman Apr 3, 2019
25baf6d
Merge remote-tracking branch 'remotes/origin/master' into emurgo/bloc…
vsubhuman Apr 4, 2019
b0b70aa
Merge remote-tracking branch 'remotes/origin/emurgo/block-by-height' …
vsubhuman Apr 4, 2019
ba175f9
Merge remote-tracking branch 'remotes/origin/master' into emurgo/bloc…
vsubhuman Apr 4, 2019
e7682e8
Merge pull request #1 from Emurgo/emurgo/block-by-height
vsubhuman Apr 4, 2019
773719a
Merge pull request #2 from Emurgo/emurgo/sync-obft-ready
vsubhuman Apr 4, 2019
55767a4
Merge pull request #3 from Emurgo/emurgo/rollback
vsubhuman Apr 4, 2019
e30d57a
Implemented `net_tip` in storage. Expanded `status` endpoint to retur…
vsubhuman Apr 4, 2019
0f496ce
Merge pull request #4 from Emurgo/emurgo/net-tip
vsubhuman Apr 4, 2019
d78dcb0
Added OBFT-testnet genesis block and add new network config for it
vsubhuman Apr 7, 2019
f86732a
Fixed genesis loading and first non-genesis hash config
vsubhuman Apr 8, 2019
48cead1
Fixed genesis/ebb
vsubhuman Apr 8, 2019
c5cb50c
Merge pull request #5 from Emurgo/obft-testnet
vsubhuman Apr 23, 2019
afb6092
Merge remote-tracking branch 'remotes/upstream/master' into master-em…
vsubhuman Apr 23, 2019
5f01985
Implemented epoch-flags and ebb-check for packed epochs
vsubhuman Apr 24, 2019
3529f60
Implemented epoch-flags and ebb-check for packed epochs
vsubhuman Apr 24, 2019
f387cbf
Fixed last EBB date assert
vsubhuman Apr 25, 2019
5e19600
Merge pull request #6 from Emurgo/block-by-height-no-ebb
vsubhuman May 13, 2019
5b63769
Merge pull request #7 from Emurgo/master
vsubhuman May 16, 2019
1882284
Fix LooseChainHeightEntry::header_hash()
rooooooooob May 16, 2019
3e827e7
Fix storage asserts regarding chain difficulty/EBB
rooooooooob May 16, 2019
217e5ef
Merge pull request #8 from Emurgo/loose-entry-hash-fix
vsubhuman May 17, 2019
a770975
Fixed formatting
vsubhuman May 17, 2019
cb32c71
Added comment about genesis hash hack
vsubhuman May 17, 2019
a66028d
Merge pull request #10 from Emurgo/fix-fmt
vsubhuman May 17, 2019
9355e07
Removed OBFT-testnet config and genesis hash hack
vsubhuman May 18, 2019
dfb91b9
Removed mention of obft-testnet genesis hash in sources
vsubhuman May 18, 2019
4a04081
Fixed variable reference
vsubhuman May 18, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
17 changes: 16 additions & 1 deletion cardano/src/block/block.rs
Expand Up @@ -12,7 +12,8 @@ use super::super::config::ProtocolMagic;
use super::boundary;
use super::date::BlockDate;
use super::normal;
use super::types::{BlockVersion, HeaderHash};
use super::types::{BlockVersion, ChainDifficulty, HeaderHash};
use crate::tx::TxAux;
use cbor_event::{self, de::Deserialize, de::Deserializer, se::Serializer};
use chain_core;

Expand Down Expand Up @@ -204,6 +205,13 @@ impl<'a> BlockHeaderView<'a> {
pub fn compute_hash(&self) -> HeaderHash {
HeaderHash::new(&self.to_cbor())
}

pub fn difficulty(&self) -> ChainDifficulty {
match self {
BlockHeaderView::Boundary(h) => h.consensus.chain_difficulty,
BlockHeaderView::Normal(h) => h.consensus.chain_difficulty,
}
}
}

impl BlockHeader {
Expand Down Expand Up @@ -248,6 +256,13 @@ impl BlockHeader {
let v = cbor!(self).unwrap();
HeaderHash::new(&v[..])
}

pub fn difficulty(&self) -> ChainDifficulty {
match self {
BlockHeader::BoundaryBlockHeader(h) => h.consensus.chain_difficulty,
BlockHeader::MainBlockHeader(h) => h.consensus.chain_difficulty,
}
}
}

impl fmt::Display for BlockHeader {
Expand Down
2 changes: 2 additions & 0 deletions cardano/src/block/chain_state.rs
Expand Up @@ -18,6 +18,7 @@ pub struct ChainState {
pub last_block: HeaderHash,
pub last_date: Option<super::BlockDate>,
pub last_boundary_block: Option<HeaderHash>,
pub last_boundary_block_epoch: Option<EpochId>,
pub slot_leaders: Vec<address::StakeholderId>,
pub utxos: Utxos,
pub chain_length: u64,
Expand Down Expand Up @@ -62,6 +63,7 @@ impl ChainState {
last_block: genesis_data.genesis_prev.clone(),
last_date: None,
last_boundary_block: None,
last_boundary_block_epoch: None,
slot_leaders: vec![],
utxos,
chain_length: 0,
Expand Down
9 changes: 9 additions & 0 deletions cardano/src/block/date.rs
Expand Up @@ -65,6 +65,15 @@ impl BlockDate {
&BlockDate::Normal(ref s) => s.epoch,
}
}
pub fn slotid(&self) -> Option<SlotId> {
match self {
&BlockDate::Boundary(_) => None,
&BlockDate::Normal(ref s) => Some(s.slotid),
}
}
pub fn epoch_and_slot(&self) -> (EpochId, Option<SlotId>) {
(self.get_epochid(), self.slotid())
}
pub fn next(&self) -> Self {
match self {
&BlockDate::Boundary(e) => BlockDate::Normal(EpochSlotId {
Expand Down
31 changes: 30 additions & 1 deletion cardano/src/block/types.rs
@@ -1,7 +1,7 @@
use super::normal::SscPayload;
use cbor_event::{self, de::Deserializer, se::Serializer};
use hash::Blake2b256;
use util::try_from_slice::TryFromSlice;
use util::{hex, try_from_slice::TryFromSlice};

use std::{
fmt,
Expand Down Expand Up @@ -46,6 +46,10 @@ impl HeaderHash {
pub fn as_hash_bytes(&self) -> &[u8; Blake2b256::HASH_SIZE] {
self.0.as_hash_bytes()
}

pub fn as_hex(&self) -> String {
hex::encode(self.as_hash_bytes())
}
}

impl fmt::Display for HeaderHash {
Expand Down Expand Up @@ -240,10 +244,35 @@ impl From<u64> for ChainDifficulty {
ChainDifficulty(f)
}
}
impl From<ChainDifficulty> for u64 {
fn from(cd: ChainDifficulty) -> Self {
cd.0
}
}

pub type EpochId = u64; // == EpochIndex
pub type SlotId = u16; // == LocalSlotIndex

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct EpochFlags {
pub is_ebb: bool,
}
impl EpochFlags {
pub fn to_mask(&self) -> u8 {
let mut mask = 0u8;
if self.is_ebb {
mask |= 1;
}
return mask;
}

pub fn from_mask(mask: u8) -> EpochFlags {
EpochFlags {
is_ebb: (mask & 1) > 0,
}
}
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[cfg_attr(feature = "generic-serialization", derive(Serialize, Deserialize))]
pub struct EpochSlotId {
Expand Down
80 changes: 56 additions & 24 deletions cardano/src/block/verify_chain.rs
Expand Up @@ -14,20 +14,34 @@ impl ChainState {

add_error(&mut res, self.do_verify(block_hash, blk));

self.last_block = block_hash.clone();
self.last_date = Some(blk.header().blockdate());
// FIXME: count boundary blocks as part of the chain length?
self.chain_length += 1;

match blk {
Block::BoundaryBlock(blk) => {
self.last_boundary_block = Some(block_hash.clone());
self.last_boundary_block_epoch = Some(blk.header.consensus.epoch);
self.slot_leaders = blk.body.slot_leaders.clone();
}

Block::MainBlock(_) => {}
Block::MainBlock(blk) => {
if self.last_boundary_block.is_some() {
let block_epoch = blk.header.consensus.slot_id.epoch;
let local_epoch = self.last_date.map(|d| d.get_epochid()).unwrap_or(0);
if block_epoch > local_epoch {
// We are in OBFT and switched epochs without EBB
// Remove last boundary block state

// TODO: should cleanup EBB state here?
//self.last_boundary_block = None;
//self.last_boundary_block_epoch = None;
//self.slot_leaders = vec![];
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: gotta decide whether we need to drop "Last EBB" state here after switch to OBFT. Cuz right now it just hangs there forever.

ℹ️ Gotta remember that "Last EBB" state is used to determine whether current epoch is an EBB or not, when storing epoch packs with meta-flags.

}
}
}
};

self.last_block = block_hash.clone();
self.last_date = Some(blk.header().blockdate());
// FIXME: count boundary blocks as part of the chain length?
self.chain_length += 1;

// Update the utxos from the transactions.
if let Block::MainBlock(blk) = blk {
for txaux in blk.body.tx.iter() {
Expand Down Expand Up @@ -65,17 +79,21 @@ impl ChainState {
return Err(Error::BlockDateInPast);
}

// If this is a genesis block, it should be the next
// epoch; otherwise it should be in the current epoch.
if date.get_epochid()
!= (last_date.get_epochid() + if date.is_boundary() { 1 } else { 0 })
{
return Err(Error::BlockDateInFuture);
// New date should be this or next epoch, not further
if date.get_epochid() > last_date.get_epochid() {
if date.get_epochid() > last_date.get_epochid() + 1 {
return Err(Error::BlockDateInFuture);
}
// First next-epoch block supposed to be EBB,
// unless we are in OBFT era
if !date.is_boundary() {
// TODO: validate we are in OBFT?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: gotta decide whether we need to add additional checks for proper OBFT switch (e.g. block version, or some other weirdo parameters from blocks). Right now we just allow new epochs to have no EBB, and that's it.

}
}
}

None => {
if date != BlockDate::Boundary(0) {
if date.get_epochid() != 0 {
// FIXME: use epoch_start
return Err(Error::BlockDateInFuture);
}
Expand All @@ -87,19 +105,33 @@ impl ChainState {
Block::BoundaryBlock(_) => {}

Block::MainBlock(blk) => {
let slot_id = blk.header.consensus.slot_id.slotid as usize;
let epoch_id = blk.header.consensus.slot_id.epoch;
let epoch_with_ebb = match &self.last_boundary_block_epoch {
Some(last_ebb_epoch) => epoch_id == *last_ebb_epoch,
_ => false,
};

if slot_id >= self.slot_leaders.len() {
return Err(Error::NonExistentSlot);
}
if epoch_with_ebb {
// If epoch contains EBB - validate block using the leader-list

let slot_id = blk.header.consensus.slot_id.slotid as usize;

if slot_id >= self.slot_leaders.len() {
return Err(Error::NonExistentSlot);
}

let slot_leader = &self.slot_leaders[slot_id];

let slot_leader = &self.slot_leaders[slot_id];
// Note: the block signature was already checked in
// verify_block, so here we only check the leader key
// against the genesis block.
if slot_leader != &address::StakeholderId::new(&blk.header.consensus.leader_key)
{
return Err(Error::WrongSlotLeader);
}
} else {

// Note: the block signature was already checked in
// verify_block, so here we only check the leader key
// against the genesis block.
if slot_leader != &address::StakeholderId::new(&blk.header.consensus.leader_key) {
return Err(Error::WrongSlotLeader);
// TODO: validate OBFT leader
Copy link
Contributor Author

@vsubhuman vsubhuman May 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: No OBFT leader validation present. We don't have the documentation yet on how it's done.

}
}
};
Expand Down