Performance Monitor Plugin #188

meevee98 · 2020-03-03T20:30:30Z

This is a plugin created to monitor some aspects of Neo.
The metrics that the plugin measures are the ones discussed in neo#920

cloud8little · 2020-03-09T09:50:26Z

@meevee98 Gread work! But there are some issues need to be fixed.
test commit : 45b10a0 with
neo-cli: master e5400583a41bd19d4f2db5e05ca59a6c04160853
neo: master ff850c5e170a8a132132de9aea6168b0668125aa
neo-vm: master 754bca3568c50695c571cb796c500fd66c07d0fe

block time <index/hash>
current block height is 310, result of block time 301, 303 is incorrect.

neo> block time 301
Block Hash: 0x1642a8e70d21a1651cba7fd9f22e4d6d05b69a93fe09119bc753a082db5194f7
      Index: 301
      Time: 127 seconds
neo> block time 302
Block Hash: 0x35716ca313de935d230d6e5959028a98697374918b56743a95c0eeadd7f1f3ad
      Index: 302
      Time: 15 seconds
neo> block time 303
Block Hash: 0x5367caa58473c2d82fdf340d715e9381d612f180fd8cd95f254ee4467072e276
      Index: 303
      Time: 303 seconds

When the height increased to 550, block avgtime 550, the result is incorrect.

neo> block avgtime 7
Average time/block: 15.06 seconds
neo> block avgtime 8
Average time/block: 15.06 seconds
neo> block avgtime 9
Average time/block: 15.06 seconds
neo> block avgtime 30
Average time/block: 18.77 seconds
neo> block avgtime 100
Average time/block: 16.76 seconds
neo> block avgtime 101
Average time/block: 16.75 seconds
neo> block avgtime 102
Average time/block: 16.75 seconds
neo> block avgtime 103
Average time/block: 16.75 seconds
neo> block avgtime 104
Average time/block: 16.73 seconds
neo> block avgtime 550
Average time/block: 1112.98 seconds
neo> block avgtime 551
Average time/block: 1112.98 seconds

check disk
Both on Win10/Ubuntu, check disk return error: Performance counters are not supported on this platform.
check memory
on Ubuntu 18.04, check memory return 0.00 MB
tx avgsize [1 - 10000]
There are total 2 txs for the test private chain. while tx avgsize 3 is not equal to tx avgsize 2.
confirmation time & payload time

always return Timeout.

neo> confirmation time
Waiting for the next commit...
Timeout
neo> confirmation time
Waiting for the next commit...
Timeout

neo> payload time
Waiting for the next payload...
Timeout

Network Commands: connected, when current height update, height in the result of "connected" does not update. for example, height is always 578, while getconnectioncount is 580.

neo> connected
Connected nodes: 1
  ip: 127.0.0.1         height: 578

meevee98 · 2020-03-09T20:50:46Z

Thank you for your tests @cloud8little

block time <index/hash>
current block height is 310, result of block time 301, 303 is incorrect.

When the height increased to 550, block avgtime 550, the result is incorrect.

I couldn't reproduce, can you give me more details?

check disk
Both on Win10/Ubuntu, check disk return error: Performance counters are not supported on this platform.

For this to work, you need to add System.Diagnostics.PerformanceCounter as a dependency in neo-cli, otherwise it won't not work, even if you include the dll in the folder.
I've added it in neo-node#549

confirmation time & payload time
always return Timeout.

Confirmation and payload time only work after start consensus, that's why it always returns a timeout. I've changed the printed message to make this condition clearer.

bettybao1209 · 2020-03-10T02:53:36Z

@meevee98 hi, I wrote a readme giving the description about the commands of the plugin, as well as some notable points to ease the use. Hope to have your opinion.

cloud8little · 2020-03-11T07:48:23Z

@meevee98

you can try "block time 147" in this private chain.
block-time-147-incorrect.zip

already add this reference. Update dependencies for Performance Plugin neo-node#549, while the result is the same: error: Performance counters are not supported on this platform.
the same chain with 1, you can wait the block height to 170+.

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

meevee98 · 2020-03-11T17:32:31Z

@meevee98 hi, I wrote a readme giving the description about the commands of the plugin, as well as some notable points to ease the use. Hope to have your opinion.

Thank you for your help @bettybao1209. I think the readme is very good.

you can try "block time 147" in this private chain.
block-time-147-incorrect.zip

I investigated the problem and I think this is not wrong at all.
Using the show block command from this pull request to show the information of the block, we can see that there is a more than 4 minutes gap between the blocks 147 and 148.

That's why the result of block time was 270 seconds (4 minutes and 30 seconds). Also, that's why I think it's not the wrong result: this result means that block 148 delayed, so the 147 remained active longer than it should.

the same chain with 1, you can wait the block height to 170+.

For this, the problem was that some blocks had a much longer time in seconds, increasing the average. I found that these three blocks had a time greater than 4 hours, with the longest between blocks 100 and 101, with a gap of almost 2 days:

Probably that happened because the blockchain was paused and continued later.

I'm still working on the problem with Performance Counter.

add readme

vncoelho

Looks good, I see that new performance indicators were added.

src/PerformanceMonitor/PerformanceMonitor.csproj

shargon · 2020-06-16T07:27:20Z

@meevee98 it's this solved? #188 (comment)

shargon · 2020-07-07T07:50:35Z

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

+        [ConsoleCommand("block avgtime", Category = "Block Commands", Description = "Show the average time in seconds the latest blocks are active.")]
+        private void OnBlockAverageTimeCommand(uint blockCount = 1000)
+        {
+            uint desiredCount = blockCount;


Reuse blockCount

shargon · 2020-07-07T07:51:50Z

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

+                    throw new RpcException(-100, "Minimum 1 block");
+                }
+
+                if (desiredCount > 10000)


How much time spend with this limit?

Using RPC, I got 889 ms in total time, with this method using 168 ms in the RPC Server

Do you think we should decrease this maximum as well?

shargon · 2020-07-07T07:52:08Z

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

+        [ConsoleCommand("block timestamp", Category = "Block Commands", Description = "Show the block timestamp for each of the n latest blocks.")]
+        private void OnBlockTimestampCommand(uint blockCount)
+        {
+            uint desiredCount = blockCount;


Reuse blockCount

shargon · 2020-07-07T07:56:15Z

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

+            uint maxNBlocksPerDay = 24 * 60 * 60 / (Blockchain.MillisecondsPerBlock / 1000);
+            if (desiredCount > maxNBlocksPerDay)
+            {
+                throw new RpcException(-100, maxNBlocksPerDay.ToString("Maximum 0 blocks"));


It was the format syntax. I changed it to make clearer what the message should be.

shargon · 2020-07-07T07:57:02Z

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs

+            using (var snapshot = Blockchain.Singleton.GetSnapshot())
+            {
+                var block = GetBlock(height.ToString());
+                var blocksTime = snapshot.GetBlocksTimestamp(desiredCount, block);


Did you test the time it spend with the maximum values?

I've tested the cli's method that returns the same information. The cli's method is a bit lazier than the rpc's because of the printed messages. In my PC I got around 0.5 seconds for getting and printing the information.

I think that it should be lower than 1000 in rpc, in console it's ok

Sure, I have no problem decreasing it.

src/PerformanceMonitor/PerformanceMonitor.cs

shargon · 2020-07-07T07:59:56Z

src/PerformanceMonitor/PerformanceMonitor.cs

+        /// <param name="height">
+        /// The block height to be passed in the ping message
+        /// </param>
+        private void SendBlockchainPingMessage(uint height)


We should send the ping always with our hight, why it's an argument?

This method is used in the block sync command. In the case that the local node is not updated, the ping message sends the max height of the remote nodes.
It was needed before the implementation of the ping command, but now it can just send the ping with the current height. I will change it.

src/PerformanceMonitor/PerformanceMonitor.cs

src/PerformanceMonitor/CpuUsageMonitor.cs

src/PerformanceMonitor/PerformanceMonitor.Consensus.cs

src/PerformanceMonitor/PerformanceMonitor.IO.cs

shargon · 2020-07-07T08:13:03Z

src/PerformanceMonitor/PerformanceMonitor.NetworkProtocol.cs

+
+            try
+            {
+                client.GetBlockCount();


MaxTimeout? what's happend if the server never answer and it doesn't close the connection?

It's handled by SendAsync method of RpcClient. PostAsync throws HttpRequestException on timeout. ref

By default it's 100 seconds https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.timeout?view=netcore-3.1 I think that it should be 30 or less

I see your point.
I'm going to set the timeout in the method then.

shargon · 2020-07-10T07:23:16Z

@neo-project/ngd-shanghai could someone test it?

superboyiii · 2020-07-10T08:41:03Z

@neo-project/ngd-shanghai could someone test it?

@cloud8little will help test this.

cloud8little · 2020-07-14T03:47:23Z

help message: PASS

block avgtime PASS
block sync PASS

block time PASS
block timesincelast PASS
block timestamp PASS
connected PASS
ping PASS
rpc time PASS
check state PASS
tx avgsize xx PASS
tx size PASS
without RpcServer PASS
commit time/confirmation time/payload time @meevee98 seems when starting a consensus node, payload/confirmation always timeout. when starting consensus as watch only, it returns value.

cloud8little · 2020-07-22T09:40:07Z

@meevee98 could you help answer the #14 please? #188 (comment)

lock9 · 2023-12-15T15:29:00Z

Hi @Liaojinghui,
@meevee98 is my colleague. Do you guys still have interest in this PR? I don't know why it was forgotten 😅

@meevee98 could you help answer the #14 please? #188 (comment)

Is this still relevant?

cschuchardt88 · 2023-12-15T20:47:41Z

it doesnt support other OS, like linux. And never fixed anything, from what i can see.

Jim8y · 2024-02-17T10:34:31Z

close as inactive

meevee98 and others added 14 commits February 27, 2020 10:26

block properties metrics

a88d0c1

helper methods

57c80ff

other metrics

a111972

fix time in percentage calculation

efa546e

cpu usage monitor

28cd1f9

io metrics

8031850

listen blockchain and remotes new blocks

93d9ea7

consensus algorithm metrics

5d14b34

network protocol metrics

3038243

rename plugin

91f41ed

format

84d12bc

separate main class in different files

0ffa0b8

change comment and reorder methods

45b10a0

add readme

d968149

meevee98 added 3 commits March 9, 2020 14:24

fix remote height on connected and ping commands

d8a3d9b

handle io exceptions

7048b9a

fix results

3a7add8

meevee98 mentioned this pull request Mar 9, 2020

Update dependencies for Performance Plugin neo-project/neo-node#549

Closed

additional info

63f1c32

fix typo

ad1bef9

shargon reviewed Mar 11, 2020

View reviewed changes

src/PerformanceMonitor/PerformanceMonitor.BlockProperties.cs Outdated Show resolved Hide resolved

meevee98 added 2 commits March 11, 2020 12:00

Merge branch 'master' into metrics-plugin

3af0a35

change precision

8aef748

bettybao1209 and others added 2 commits March 12, 2020 10:08

add note for consensus commands

09d615f

Merge pull request #7 from bettybao1209/metrics-plugin

f90f684

add readme

vncoelho reviewed Jun 1, 2020

View reviewed changes

shargon reviewed Jun 16, 2020

View reviewed changes

src/PerformanceMonitor/PerformanceMonitor.csproj Outdated Show resolved Hide resolved

fix cpu high usage on block sync

8a4d004

meevee98 requested a review from shargon July 3, 2020 19:15

Merge branch 'master' into metrics-plugin

910cf6f

shargon reviewed Jul 7, 2020

View reviewed changes

src/PerformanceMonitor/PerformanceMonitor.cs Outdated Show resolved Hide resolved

shargon reviewed Jul 7, 2020

View reviewed changes

src/PerformanceMonitor/PerformanceMonitor.cs Outdated Show resolved Hide resolved

src/PerformanceMonitor/PerformanceMonitor.cs Show resolved Hide resolved

src/PerformanceMonitor/PerformanceMonitor.cs Show resolved Hide resolved

shargon reviewed Jul 7, 2020

View reviewed changes

src/PerformanceMonitor/PerformanceMonitor.cs Show resolved Hide resolved

src/PerformanceMonitor/CpuUsageMonitor.cs Outdated Show resolved Hide resolved

shargon reviewed Jul 7, 2020

View reviewed changes

meevee98 added 3 commits July 9, 2020 10:16

Merge branch 'master' into metrics-plugin

14c3159

code review

c800081

Merge check cpu, threads and memory

fcfd339

meevee98 requested a review from shargon July 9, 2020 20:33

meevee98 added 2 commits July 13, 2020 12:33

Merge branch 'master' into metrics-plugin

4efd244

update commands limits and timeouts

b4e4399

meevee98 added 2 commits August 4, 2020 15:31

Merge branch 'master' into metrics-plugin

4245576

fix payload/confirmation time

203bd38

Jim8y added the Need Active Pr will be closed after one week if no new activity. label Feb 12, 2024

Jim8y closed this Feb 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance Monitor Plugin #188

Performance Monitor Plugin #188

meevee98 commented Mar 3, 2020

cloud8little commented Mar 9, 2020 •

edited

meevee98 commented Mar 9, 2020 •

edited

bettybao1209 commented Mar 10, 2020

cloud8little commented Mar 11, 2020 •

edited

meevee98 commented Mar 11, 2020

vncoelho left a comment

shargon commented Jun 16, 2020

shargon Jul 7, 2020

shargon Jul 7, 2020

meevee98 Jul 13, 2020

shargon Jul 7, 2020

shargon Jul 7, 2020

meevee98 Jul 13, 2020

shargon Jul 7, 2020

meevee98 Jul 9, 2020

shargon Jul 10, 2020

meevee98 Jul 13, 2020

shargon Jul 7, 2020

meevee98 Jul 7, 2020

shargon Jul 7, 2020

meevee98 Jul 7, 2020

shargon Jul 10, 2020

meevee98 Jul 13, 2020

shargon commented Jul 10, 2020

superboyiii commented Jul 10, 2020

cloud8little commented Jul 14, 2020

cloud8little commented Jul 22, 2020

lock9 commented Dec 15, 2023

cschuchardt88 commented Dec 15, 2023 •

edited

Jim8y commented Feb 17, 2024

Performance Monitor Plugin #188

Performance Monitor Plugin #188

Conversation

meevee98 commented Mar 3, 2020

cloud8little commented Mar 9, 2020 • edited

meevee98 commented Mar 9, 2020 • edited

bettybao1209 commented Mar 10, 2020

cloud8little commented Mar 11, 2020 • edited

meevee98 commented Mar 11, 2020

vncoelho left a comment

Choose a reason for hiding this comment

shargon commented Jun 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shargon commented Jul 10, 2020

superboyiii commented Jul 10, 2020

cloud8little commented Jul 14, 2020

cloud8little commented Jul 22, 2020

lock9 commented Dec 15, 2023

cschuchardt88 commented Dec 15, 2023 • edited

Jim8y commented Feb 17, 2024

cloud8little commented Mar 9, 2020 •

edited

meevee98 commented Mar 9, 2020 •

edited

cloud8little commented Mar 11, 2020 •

edited

cschuchardt88 commented Dec 15, 2023 •

edited