feat(chain-listener): initial refactoring + tests #2232

gurinderu · 2024-04-19T14:49:54Z

Description

Added chain listener test for cc activation flow.

Additional Notes

Added traits for the dependencies
Added the ability to inject topology into the dummy core manager

Checklist

The code follows the project's coding conventions and style guidelines.
All tests related to the changes have passed successfully.
Documentation has been updated to reflect the changes (if applicable).
All new and existing unit tests have passed.
I have self-reviewed my code and ensured its quality.
I have added/updated necessary comments to aid understanding.

Reviewer Checklist

Code has been reviewed for quality and adherence to guidelines.
Tests have been reviewed and are sufficient to validate the changes.
Documentation has been reviewed and is up to date.
Any questions or concerns have been addressed.

crates/chain-listener/src/ccp.rs

crates/chain-listener/src/event/mod.rs

crates/chain-listener/src/listener.rs

crates/core-manager/src/dummy.rs

crates/core-manager/src/manager.rs

justprosh · 2024-05-09T15:50:30Z

.github/workflows/e2e.yml

@@ -85,6 +85,7 @@ jobs:
    uses: fluencelabs/cli/.github/workflows/tests.yml@main
    with:
      nox-image: "${{ needs.nox-snapshot.outputs.nox-image }}"
+      ref: "timeout"


Don't forget to revert it after merge

CLAassistant · 2024-05-10T13:53:12Z

All committers have signed the CLA.

crates/chain-listener/src/ccp.rs

mikevoronov · 2024-05-14T20:14:53Z

nox/src/node.rs

+            config.network_id,
+            config.wallet_key,
+            config.default_base_fee,
+            config.default_base_fee,


Suggested change

config.default_base_fee,

config.default_priority_fee,

mikevoronov · 2024-05-14T20:17:48Z

nox/src/node.rs

-        let ccp_client = if let Some(ccp_endpoint) = listener_config.ccp_endpoint.clone() {
+        let ccp_client: Option<Arc<dyn CCPClient>> = if let Some(ccp_endpoint) =
+            listener_config.ccp_endpoint.clone()
+        {
            let ccp_client = CCPRpcHttpClient::new(ccp_endpoint.clone())


This method could receive AsRef<str> instead, it'll allow not to clone the ccp_endpoint string several times.

mikevoronov · 2024-05-14T20:22:51Z

nox/src/node.rs

@@ -123,28 +125,40 @@ async fn setup_listener(
        config.chain_config.clone(),
        config.chain_listener_config.clone(),
    ) {
-        let ccp_client = if let Some(ccp_endpoint) = listener_config.ccp_endpoint.clone() {
+        let ccp_client: Option<Arc<dyn CCPClient>> = if let Some(ccp_endpoint) =


Looks like listener_config.ccp_endpoint.map(|ccp_endpoint| ....) would be more readable.

Do you need typing in ccp_client?

afaik clippy advices to use if let Some in these cases

mikevoronov · 2024-05-14T20:26:44Z

nox/src/node.rs

            Some(ccp_client)
        } else {
            None
        };

-        let ws_client = ChainListener::create_ws_client(&listener_config.ws_endpoint).await?;
        let cc_events_dir = config.dir_config.cc_events_dir.clone();


It better to move this definition close to usage of this variable or even remove it and use this statement directly in the ChainListener::new.

mikevoronov · 2024-05-14T21:10:45Z

crates/chain-listener/src/subscription.rs

+           let ws_client = self.ws_client.read().await;
+                ws_client.subscribe("eth_subscribe", params.clone(), "eth_unsubscribe")


Are you sure that this is formatted?

mikevoronov · 2024-05-14T21:11:14Z

crates/chain-listener/src/subscription.rs

+                if let RestartNeeded(_) = err {
+                    tracing::error!(target: "chain-listener", "Failed to subscribe to {method}: {err};");
+                    Permanent(err)
+                } else {
+                    tracing::warn!(target: "chain-listener", "Failed to subscribe to {method}: {err}; Retrying...");
+                    backoff::Error::transient(err)
+                }})


The match block will be more readable.

mikevoronov · 2024-05-14T21:14:41Z

crates/chain-connector/src/connector.rs

@@ -72,9 +71,47 @@ pub trait ChainConnector: Send + Sync {
    ) -> Result<Vec<Result<Value, ConnectorError>>, ConnectorError>;
 }

+#[derive(Clone)]
+pub struct HttpChainConnectorConfig {


I still not sure that we need such a structure.

mikevoronov · 2024-05-14T21:22:37Z

crates/chain-listener/src/listener.rs

@@ -773,7 +674,7 @@ impl ChainListener {
    /// Unit goes to Deal
    async fn process_unit_deactivated(
        &mut self,


In the acquire_cores_for_cc function you can make core more readable by decoupling assignments from match to make it smth like

let assignment = match cores { Ok(assignment) => assignment, Err(AcquireError::NotFoundAvailableCores { required, available, .. }) => { tracing::warn!(target: "chain-listener", "Found {required} CUs in the Capacity Commitment, but Nox has only {available} Cores available for CC"); let assign_units = units.iter().take(available).cloned().collect(); let assignment = self.core_manager.acquire_worker_core(AcquireRequest::new( assign_units, WorkType::CapacityCommitment, ))?; assignment }; let priority_units = filter(&cu_groups.priority_units, &assignment); let non_priority_units = filter(&cu_groups.non_priority_units, &assignment); let pending_units = filter(&cu_groups.pending_units, &assignment); let finished_units = filter(&cu_groups.finished_units, &assignment); Ok(PhysicalCoreGroups { priority_cores: priority_units, non_priority_cores: non_priority_units, pending_cores: pending_units, finished_cores: finished_units, })

mikevoronov · 2024-05-14T21:25:54Z

crates/chain-listener/src/listener.rs

            .name("ChainListener")
-            .spawn(async move {
+            .spawn(
+                async move {



excess empty line here

mikevoronov · 2024-05-14T22:21:54Z

crates/chain-listener/src/ccp.rs

+use std::collections::HashMap;
+
+#[async_trait]
+pub trait CCPClient: Sync + Send {


There is the same trait with name NoxCCPApi inside CCP intended to use in Nox.

And nobody uses it) and CCPRpcHttpClient also doesn't)

Maybe we can avoid using this upper-case mess? Like, CcpClient instead of CCPC...?

github-actions · 2024-05-17T15:22:00Z

crates/core-manager/src/core_range.rs

+impl CoreRange {
+    pub fn is_subset(&self, cores: &NonEmpty<PhysicalCoreId>) -> bool {
+        let range: RangeSetBlaze<usize> =
+            RangeSetBlaze::from_iter(cores.into_iter().map(|core| <usize>::from(core.clone())));


🚫 [clippy] _{reported by reviewdog 🐶}

error: using `clone` on type `PhysicalCoreId` which implements the `Copy` trait --> crates/core-manager/src/core_range.rs:15:81 | 15 | RangeSetBlaze::from_iter(cores.into_iter().map(|core| ::from(core.clone()))); | ^^^^^^^^^^^^ help: try dereferencing it: `*core` | = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy = note: `-D clippy::clone-on-copy` implied by `-D warnings` = help: to override `-D warnings` add `#[allow(clippy::clone_on_copy)]`

kmd-fl · 2024-05-28T09:14:49Z

aquamarine/src/log.rs

@@ -12,7 +12,16 @@ fn truncate(s: &str) -> &str {
 pub fn builtin_log_fn(service: &str, args: &str, elapsed: FormattedDuration, particle_id: String) {
    let args = truncate(args);
    match service {
-        "array" | "cmp" | "debug" | "math" | "op" | "getDataSrv" | "run-console" | "json" => {
+        "run-console" => {


sure you want to leave it INFO?

kmd-fl · 2024-05-28T11:04:21Z

crates/chain-listener/src/subscription.rs

+        self.subscribe("logs", self.unit_matched_params()).await
+    }
+
+    async fn refresh(&self) -> Result<(), Error> {


Looks like this is unused

kmd-fl · 2024-05-28T11:06:57Z

crates/chain-listener/src/ccp.rs

+use std::collections::HashMap;
+
+#[async_trait]
+pub trait CCPClient: Sync + Send {


Maybe we can avoid using this upper-case mess? Like, CcpClient instead of CCPC...?

gurinderu added the e2e Run e2e workflow label Apr 24, 2024

gurinderu marked this pull request as ready for review April 24, 2024 14:41

gurinderu requested review from justprosh and kmd-fl April 25, 2024 19:12

kmd-fl reviewed Apr 26, 2024

View reviewed changes

crates/chain-listener/src/ccp.rs Show resolved Hide resolved

crates/chain-listener/src/event/mod.rs Outdated Show resolved Hide resolved

crates/chain-listener/src/listener.rs Show resolved Hide resolved

crates/core-manager/src/dummy.rs Show resolved Hide resolved

gurinderu requested a review from kmd-fl April 26, 2024 09:40

github-actions bot reviewed Apr 26, 2024

View reviewed changes

crates/core-manager/src/manager.rs Outdated Show resolved Hide resolved

justprosh reviewed May 9, 2024

View reviewed changes

justprosh mentioned this pull request May 10, 2024

fix(chain-listener): reset proof counters on new epoch in refresh [NET-813] #2246

Merged

10 tasks

shamsartem added e2e Run e2e workflow and removed e2e Run e2e workflow labels May 10, 2024

gurinderu added e2e Run e2e workflow and removed e2e Run e2e workflow labels May 13, 2024

shamsartem added e2e Run e2e workflow and removed e2e Run e2e workflow labels May 13, 2024

gurinderu force-pushed the listener-test branch 2 times, most recently from 897f5e7 to fe9caea Compare May 14, 2024 18:18

mikevoronov reviewed May 14, 2024

View reviewed changes

gurinderu added 10 commits May 17, 2024 11:39

wip

b6a2304

wip

0bba9a3

wip

30de5f2

wip

9a6840d

wip

8cafb89

init bevaviour

f5423ba

init bevaviour

07b2429

init bevaviour

eaf2434

switch to mockall

b301a56

switch to mockall

a02a14c

gurinderu and others added 23 commits May 17, 2024 11:40

config fix

6a70628

fix test

e235124

fix fmt

539e4d2

review fixes

eb6a46f

review fixes

a8cabcb

fix fmt

c428e85

fix logging target

f2b0fcf

fix accuire

067c8a7

fix accuire

781e46d

cli tests timeout change

fdb9395

fix accuire

b532fd8

wip

2bd7f7a

wip

7cf0076

wip

853060e

wip

3ce6213

fix

a13b0b8

fix

f1c1436

update decider

325f4f8

wip

0923193

wip

9385327

wip

58bb8e7

wip

f69b1df

fmt fix

6f8355f

gurinderu force-pushed the listener-test branch from 7553afb to 6f8355f Compare May 17, 2024 09:20

fix

ad79f59

github-actions bot reviewed May 17, 2024

View reviewed changes

gurinderu added 2 commits May 17, 2024 18:22

fix

530ed0c

clippy fixes

144a444

kmd-fl reviewed May 28, 2024

View reviewed changes

gurinderu marked this pull request as draft May 28, 2024 13:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(chain-listener): initial refactoring + tests #2232

feat(chain-listener): initial refactoring + tests #2232

gurinderu commented Apr 19, 2024 •

edited

justprosh May 9, 2024 •

edited

CLAassistant commented May 10, 2024 •

edited

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

kmd-fl May 15, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

mikevoronov May 14, 2024

gurinderu May 15, 2024

kmd-fl May 28, 2024

github-actions bot May 17, 2024

kmd-fl May 28, 2024

kmd-fl May 28, 2024

kmd-fl May 28, 2024

		let ws_client = self.ws_client.read().await;
		ws_client.subscribe("eth_subscribe", params.clone(), "eth_unsubscribe")

feat(chain-listener): initial refactoring + tests #2232

Are you sure you want to change the base?

feat(chain-listener): initial refactoring + tests #2232

Conversation

gurinderu commented Apr 19, 2024 • edited

Description

Additional Notes

Checklist

Reviewer Checklist

justprosh May 9, 2024 • edited

Choose a reason for hiding this comment

CLAassistant commented May 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot May 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gurinderu commented Apr 19, 2024 •

edited

justprosh May 9, 2024 •

edited

CLAassistant commented May 10, 2024 •

edited