feature/refactor: integrate `NomadConfig` #30

luketchang · 2022-03-16T19:43:49Z

Changes

NomadIdentifier and NameOrDomain moved to new nomad-types crate
Configuration crate moved into rust repo
All pure configuration structs/enums moved to configuration crate (e.g. SignerConf, ChainConf)
Agent-specific settings are mix of base settings + agent settings from configuration crate
Adds AgentsSecrets struct which is deserialized from secrets.json
Full base + agent-specific settings built from NomadConfig + AgentSecrets
Paging settings made per-chain not single for entire agent

Chores

All base and agent-specific configurations unit tested
Agents run locally to ensure proper bootup

Closes #28, #31, #14

…rs + rpc

…gs decl for relayer

…o configuration

…data only)

…rom config types

…nted out

luketchang · 2022-03-23T01:50:10Z

Cargo.toml

@@ -14,5 +15,5 @@ members = [
    "agents/processor",
    "tools/kms-cli",
    "tools/nomad-cli",
-    "tools/balance-exporter"
+    # "tools/balance-exporter"


Will delete in separate PR: #38

luketchang · 2022-03-23T02:34:56Z

agents/processor/src/settings.rs

+mod test {
+    use super::*;
+    use nomad_base::{AgentSecrets, NomadAgent};
+
+    const RUN_ENV: &str = "test";
+    const AGENT_HOME: &str = "ethereum";
+    const SECRETS_PATH: &str = "../../fixtures/secrets.json";
+
+    #[test]
+    fn it_builds_settings_from_config_and_secrets() {
+        std::env::set_var("RUN_ENV", RUN_ENV);
+        std::env::set_var("AGENT_HOME", AGENT_HOME);
+        std::env::set_var("SECRETS_PATH", SECRETS_PATH);

-decl_settings!(Processor {
-    /// The polling interval (in seconds)
-    interval: String,
-    /// An allow list of message senders
-    allowed: Option<HashSet<H256>>,
-    /// A deny list of message senders
-    denied: Option<HashSet<H256>>,
-    /// Only index transactions if this key is set
-    indexon: Option<String>,
-    /// An amazon aws s3 bucket to push proofs to
-    s3: Option<S3Config>,
-});
+        let settings = ProcessorSettings::new().unwrap();
+
+        let config = nomad_xyz_configuration::get_builtin(RUN_ENV).unwrap();
+        let secrets = AgentSecrets::from_file(SECRETS_PATH).unwrap();
+
+        settings
+            .base
+            .validate_against_config_and_secrets(
+                crate::Processor::AGENT_NAME,
+                AGENT_HOME,
+                config,
+                &secrets,
+            )
+            .unwrap();
+
+        let agent_config = &config.agent().get("ethereum").unwrap().processor;
+        assert_eq!(settings.agent.interval, agent_config.interval);
+        assert_eq!(settings.agent.allowed, agent_config.allowed);
+        assert_eq!(settings.agent.denied, agent_config.denied);
+        assert_eq!(settings.agent.index_only, agent_config.index_only);
+        assert_eq!(settings.agent.s3, agent_config.s3);
+    }
+}


Have issue up to DRY up agent settings tests: #34

luketchang · 2022-03-23T02:35:49Z

configuration/data/definitions.ts

@@ -0,0 +1,138 @@
+export type NomadIdentifier = string;


Need to update definitions. Leaving for later PR due to avoid adding more scope to already overloaded PR: #39

luketchang · 2022-03-23T02:37:36Z

nomad-types/src/utils.rs

+/// # Arguments
+///
+/// * `s` - The hex str
+pub fn strip_0x_prefix(s: &str) -> &str {


agents/processor/src/processor.rs

prestwich · 2022-03-23T16:31:34Z

nomad-base/src/settings/secrets.rs

+    pub attestation_signer: Option<SignerConf>,
+}
+
+impl AgentSecrets {


open an issue for from_env

prestwich · 2022-03-23T16:38:32Z

chains/nomad-ethereum/Cargo.toml

@@ -11,6 +11,8 @@ edition = "2021"
 serde = "1.0.120"
 serde_json = { version = "1.0.61", default-features = false }
 ethers = {git = "https://github.com/gakonst/ethers-rs", branch = "master", features = ["abigen"]}
+ethers-core = {git = "https://github.com/gakonst/ethers-rs", branch = "master"}
+ethers-providers = {git = "https://github.com/gakonst/ethers-rs", branch = "master"}


open an issue for simplifiying our ethers dependencies 😅

configuration/LICENSE-APACHE

configuration/LICENSE-MIT

nomad-base/src/settings/secrets.rs

prestwich · 2022-03-23T16:44:42Z

nomad-base/bin/example.rs

-}
-
-#[allow(dead_code)]
+// use nomad_base::{decl_settings, NomadAgent};


maybe just delete the file at this point. no need for it

prestwich · 2022-03-23T16:46:03Z

nomad-base/src/contract_sync/mod.rs

@@ -46,12 +48,14 @@ where

 impl<I> ContractSync<I> {
    /// Instantiate new ContractSync
+    #[allow(clippy::too_many_arguments)]


ouch 💀 put this on our tech debt plate

nomad-base/src/settings/chains.rs

prestwich · 2022-03-23T16:48:58Z

nomad-base/src/settings/mod.rs

-            .and_then(|s| s.parse::<u32>().ok())
-            .unwrap_or(1999)
+    /// Get agent-specific index settings unique to that agent
+    pub fn from_agent_name(agent_name: &str) -> Self {


future: this should be type-associated information on the agent type. open an issue here

prestwich · 2022-03-23T16:50:37Z

nomad-base/src/settings/mod.rs

@@ -221,39 +153,47 @@ pub struct Settings {
    /// The path to use for the DB file


the longterm goal should be to deprecate and remove this type in favor of using the config types more directly

Yep I think having a struct BaseAgentSettings in configuration would be good and then all the types with lots of functionality (e.g. CachingHome) implement From<BaseAgentSettings>. Something like that where instead of implementing into_<some struct> we implement from_<some config struct>

prestwich · 2022-03-23T16:52:50Z

nomad-core/Cargo.toml

@@ -25,6 +25,12 @@ rocksdb = { git = "https://github.com/rust-rocksdb/rust-rocksdb" }
 prometheus = "0.12.0"
 bytes = { version = "1", features = ["serde"]}
 num = {version="0", features=["serde"]}
+rusoto_core = "0.47.0"


I don't like moving the aws deps into core 🤔 where's a better place to put these in the future?

Yeah only reason this went here is so we could build a Signers from SignerConf. Best solution is probably to keep SignerExt in nomad-core and move Signers (enum over Aws and Local types) into nomad-base

Well turns out ethers technically imports rusoto_core anyways thru AwsSigner types. So regardless, aws stuff has to be a dependency of nomad-core if we import ethers

ack oh well

luketchang added 7 commits March 16, 2022 10:51

prog: home instantiation from NomadConfig, clippy failing

95183d8

prog: instantiates replicas, missing index settings + tracing + signe…

ee47f5d

…rs + rpc

fix: simplify replica creation

2c617a7

chore: delete jaeger and zipkin

01d484d

refactor: move NomadIdentifier into nomad-types crate

1787765

refactor: move agent config types to nomad-types

f835740

feature: retrieve index settings from agent name

f71ffa8

luketchang force-pushed the luke/integrate-config branch from 55723d5 to f71ffa8 Compare March 17, 2022 20:43

luketchang added 4 commits March 17, 2022 13:59

chore: rename tracing to log config

fb39989

feature: add unused secrets module

419f756

refactor: get transaction signers and logging from config; new settin…

0abcc7a

…gs decl for relayer

prog: agents using new config scheme; todo add agent-specific types t…

ade8310

…o configuration

luketchang force-pushed the luke/integrate-config branch from da39e85 to ade8310 Compare March 18, 2022 05:30

feature: add all agent-specific config blocks to nomad-types (public …

ffae9b7

…data only)

luketchang force-pushed the luke/integrate-config branch from d3ca042 to ffae9b7 Compare March 18, 2022 06:49

luketchang added 5 commits March 18, 2022 12:30

refactor: use config agent types, instantiate agent settings blocks f…

57aec19

…rom config types

fix: watcher gets managers from configuration

0c67a89

refactor: building settings draws on secret rpcs; example agent comme…

3a6f754

…nted out

fix: make metrics port optional

c366580

test: add base settings and per-agent config to settings tests

a2117ad

luketchang force-pushed the luke/integrate-config branch from ee1d25d to a2117ad Compare March 20, 2022 06:23

luketchang added 3 commits March 20, 2022 09:18

fix: check rpcs for replicas and xapps

d074ae5

refactor: DRY up deser_nomad_number with macro

53ea4fb

fix: deserialization default reads u64 to fix failing tests

6f5f96b

luketchang force-pushed the luke/integrate-config branch from 9a7757b to 6f5f96b Compare March 20, 2022 20:46

luketchang changed the title ~~WIP: integrate NomadConfig~~ feature/refactor: integrate NomadConfig Mar 20, 2022

luketchang added 2 commits March 20, 2022 16:27

fix: tests use updated uints from config crate

0373bb5

chore: put example bin agent back

d4d02aa

luketchang force-pushed the luke/integrate-config branch from 44997d2 to d4d02aa Compare March 20, 2022 22:37

luketchang marked this pull request as ready for review March 20, 2022 22:39

feature: allow config to be overridable

99d0c76

luketchang commented Mar 23, 2022

View reviewed changes

chore: nits

49d5fb3

luketchang force-pushed the luke/integrate-config branch from a793b2b to 49d5fb3 Compare March 23, 2022 02:14

luketchang commented Mar 23, 2022

View reviewed changes

prestwich assigned luketchang Mar 23, 2022

luketchang commented Mar 23, 2022

View reviewed changes

nomad-types/src/utils.rs

/// # Arguments

///

/// * `s` - The hex str

pub fn strip_0x_prefix(s: &str) -> &str {

Copy link

Collaborator Author

luketchang Mar 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#32

fix: new repo link for configuration

7ba98a4

prestwich marked this pull request as ready for review March 23, 2022 16:04

docs: add docs for agent deploy flow

3f5b752