Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow genesis tool to specify shards to track #1726

Merged
merged 5 commits into from Nov 20, 2019
Merged

Allow genesis tool to specify shards to track #1726

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
27dfe94
Allow genesis tool to specify shards to track
bowenwang1996 Nov 19, 2019
bbe08be
Add help for tracked-shards
bowenwang1996 Nov 19, 2019
0b0c9c2
check shard id
bowenwang1996 Nov 19, 2019
6ed7763
Merge branch 'staging' into track-shards-for-genesis
bowenwang1996 Nov 19, 2019
60dd536
fix
bowenwang1996 Nov 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions genesis-tools/genesis-csv-to-json/src/csv_to_json_configs.rs
View file
Open in desktop
Expand Up @@ -6,7 +6,7 @@ use near::config::{
CONFIG_FILENAME, DEVELOPER_PERCENT, EXPECTED_EPOCH_LENGTH, GAS_PRICE_ADJUSTMENT_RATE,
GENESIS_CONFIG_FILENAME, INITIAL_GAS_LIMIT, INITIAL_GAS_PRICE, MAX_INFLATION_RATE,
NODE_KEY_FILE, NUM_BLOCKS_PER_YEAR, NUM_BLOCK_PRODUCERS, PROTOCOL_PERCENT,
TRANSACTION_VALIDITY_PERIOD, VALIDATOR_KEY_FILE,
TRANSACTION_VALIDITY_PERIOD,
};
use near::{GenesisConfig, NEAR_BASE};
use near_network::types::PROTOCOL_VERSION;
Expand All @@ -29,13 +29,13 @@ fn verify_total_supply(total_supply: Balance, chain_id: &String) {

/// Generates `config.json` and `genesis.config` from csv files.
/// Verifies that `validator_key.json`, and `node_key.json` are present.
pub fn csv_to_json_configs(home: &Path, chain_id: String) {
pub fn csv_to_json_configs(home: &Path, chain_id: String, tracked_shards: Vec<ShardId>) {
// Verify that key files exist.
assert!(home.join(VALIDATOR_KEY_FILE).as_path().exists(), "Validator key file should exist");
assert!(home.join(NODE_KEY_FILE).as_path().exists(), "Node key file should exist");

// Construct `config.json`.
let mut config = Config::default();
config.tracked_shards = tracked_shards;
config.telemetry.endpoints.push(near::config::DEFAULT_TELEMETRY_URL.to_string());

// Construct genesis config.
Expand Down
18 changes: 17 additions & 1 deletion genesis-tools/genesis-csv-to-json/src/main.rs
View file
Open in desktop
@@ -1,5 +1,7 @@
use clap::{App, Arg};
use near::get_default_home;
use near_primitives::types::ShardId;
use std::collections::HashSet;
use std::path::Path;

pub mod csv_parser;
Expand All @@ -18,9 +20,23 @@ fn main() {
.takes_value(true),
)
.arg(Arg::with_name("chain-id").long("chain-id").takes_value(true))
.arg(
Arg::with_name("tracked-shards")
.long("tracked-shards")
.takes_value(true)
.help("Set of shards that this node wants to track (default empty)"),
)
.get_matches();

let home_dir = matches.value_of("home").map(|dir| Path::new(dir)).unwrap();
let chain_id = matches.value_of("chain-id").expect("Chain id is requried");
csv_to_json_configs::csv_to_json_configs(home_dir, chain_id.to_string());
let tracked_shards: HashSet<ShardId> = match matches.value_of("tracked-shards") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this arg used? The usage of this command line tool is to allow a group of participants who share the same csv file to create the same genesis config. Are they all going to track the same set of shards? Will this set of shards include all shards existing, if not who is going to track the remaining shards?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add explanation to the arg's description.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracked shards are independent of genesis. They are specified by the node if the node wants to always track some shards. I'll add an explanation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also answer:

Are they all going to track the same set of shards? Will this set of shards include all shards existing, if not who is going to track the remaining shards?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And add this explanation to the arg description.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. As I said tracked-shards is specified by the node. If a node wants to track some shards, it is their decision. It is not related in any way to how any other nodes make this decision.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do people decide what shards to track? Let's say we have stake wars participants, how does participant Alice decide that she wants to track some set of shards, e.g. "1,5,6"? How do they know what is the total number of shards?

Some(s) => s.split(',').map(|v| v.parse::<ShardId>().unwrap()).collect(),
bowenwang1996 marked this conversation as resolved.
Show resolved Hide resolved
None => HashSet::default(),
};
csv_to_json_configs::csv_to_json_configs(
home_dir,
chain_id.to_string(),
tracked_shards.into_iter().collect(),
);
}
1 change: 0 additions & 1 deletion near/src/runtime.rs
View file
Open in desktop
Expand Up @@ -921,7 +921,6 @@ impl RuntimeAdapter for NightshadeRuntime {
}

fn get_validator_info(&self, block_hash: &CryptoHash) -> Result<EpochValidatorInfo, Error> {
println!("get validator info");
let mut epoch_manager = self.epoch_manager.write().expect(POISONED_LOCK_ERR);
epoch_manager.get_validator_info(block_hash).map_err(|e| e.into())
}
Expand Down
10 changes: 6 additions & 4 deletions scripts/nodelib.py
View file
Open in desktop
Expand Up @@ -257,7 +257,7 @@ def initialize_keys(home, is_release, nodocker, image, account_id, generate_sign
if account_id:
generate_validator_key(home, is_release, nodocker, image, account_id)

def create_genesis(home, is_release, nodocker, image, chain_id):
def create_genesis(home, is_release, nodocker, image, chain_id, tracked_shards):
if os.path.exists(os.path.join(home, 'genesis.json')):
print("Genesis already exists")
return
Expand All @@ -268,15 +268,17 @@ def create_genesis(home, is_release, nodocker, image, chain_id):
cmd = ['./target/%s/genesis-csv-to-json' % ('release' if is_release else 'debug')]
cmd.extend(['--home', home])
cmd.extend(['--chain-id', chain_id])
if len(tracked_shards) > 0:
cmd.extend(['--tracked-shards', tracked_shards])
try:
subprocess.call(cmd)
except KeyboardInterrupt:
print("\nStopping NEARCore.")
else:
subprocess.check_output(['docker', 'run', '-v', '%s:/srv/genesis-csv-to-json' % home, '-it', image, 'genesis-csv-to-json', '--home=/srv/genesis-csv-to-json', '--chain-id=%s' % chain_id])
subprocess.check_output(['docker', 'run', '-v', '%s:/srv/genesis-csv-to-json' % home, '-it', image, 'genesis-csv-to-json', '--home=/srv/genesis-csv-to-json', '--chain-id=%s' % chain_id, '--tracked-shards=%s' % tracked_shards])
print("Genesis created")

def start_stakewars(home, is_release, nodocker, image, telemetry_url, verbose):
def start_stakewars(home, is_release, nodocker, image, telemetry_url, verbose, tracked_shards):
if nodocker:
install_cargo()
compile_package('genesis-csv-to-json', is_release)
Expand All @@ -287,7 +289,7 @@ def start_stakewars(home, is_release, nodocker, image, telemetry_url, verbose):
except subprocess.CalledProcessError as exc:
print("Failed to fetch docker containers: %s" % exc)
exit(1)
create_genesis(home, is_release, nodocker, image, 'stakewars')
create_genesis(home, is_release, nodocker, image, 'stakewars', tracked_shards)
if nodocker:
run_nodocker(home, is_release, boot_nodes='', telemetry_url=telemetry_url, verbose=verbose)
else:
Expand Down
3 changes: 2 additions & 1 deletion scripts/start_stakewars.py
View file
Open in desktop
Expand Up @@ -20,6 +20,7 @@
parser.add_argument(
'--image', default='nearprotocol/nearcore:stakewars',
help='Image to run in docker (default: nearprotocol/nearcore:stakewars)')
parser.add_argument('--tracked-shards', default='', help='The shards that this node wants to track')
args = parser.parse_args()

TELEMETRY_URL = 'https://explorer.tatooine.nearprotocol.com/api/nodes'
Expand All @@ -34,4 +35,4 @@
print("****************************************************")
print("* Running NEAR validator node for Stake Wars *")
print("****************************************************")
start_stakewars(args.home, not args.debug, nodocker, args.image, telemetry_url=TELEMETRY_URL, verbose=args.verbose)
start_stakewars(args.home, not args.debug, nodocker, args.image, telemetry_url=TELEMETRY_URL, verbose=args.verbose, tracked_shards=args.tracked_shards)