Add chain database size to telemetry #3928

ygao76 · 2023-05-24T05:30:46Z

Summary

Create size() function for level DB
Create a new chainDatabaseSize metric

Testing Plan

unit test

Documentation

Does this change require any updates to the Iron Fish Docs (ex. the RPC API
Reference)? If yes, link a
related documentation pull request for the website.

[ ] Yes

Breaking Change

Is this a breaking change? If yes, add notes below on why this is breaking and
what additional work is required, if any.

[ ] Yes

@rohanjadvani @NullSoldier @dguenther

ironfish/src/blockchain/blockchain.ts

rohanjadvani · 2023-05-24T21:44:29Z

ironfish/src/metrics/metricsMonitor.ts

@@ -19,6 +19,7 @@ export class MetricsMonitor {

  readonly mining_newBlockTemplate: Meter
  readonly chain_newBlock: Meter
+  readonly chainDatabaseSize: Gauge


Suggested change

readonly chainDatabaseSize: Gauge

readonly chain_databaseSize: Gauge

I will change it, curious about the reason for this particular name convention. Is this for better parsing for telemetry?

ironfish/src/blockchain/blockchain.ts

danield9tqh · 2023-05-25T21:07:30Z

ironfish/src/blockchain/blockchain.test.ts

@@ -103,6 +103,8 @@ describe('Blockchain', () => {
    expect((await chain.getHashAtSequence(2))?.equals(headerA1.hash)).toBe(true)
    expect((await chain.getHashAtSequence(3))?.equals(headerB2.hash)).toBe(true)
    expect((await chain.getHashAtSequence(4))?.equals(headerB3.hash)).toBe(true)
+
+    expect(await chain.db.size()).toBeGreaterThanOrEqual(size)


I don't think this is testing much if we're accepting it can be equal as well. They could both be 0. I think removing this would be fine or just check strictly greater than.

Updated to check the chain db size > 0. The size function is the approximate size https://github.com/Level/leveldown#dbapproximatesizestart-end-callback There is disclaimer that the result might not include the recent writes. In the previous test, the size value actually does not change due to this.

danield9tqh · 2023-05-25T21:09:18Z

ironfish/src/blockchain/blockchain.ts

@@ -406,6 +406,8 @@ export class Blockchain {

        this.resolveOrphans(block)

+        this.metrics.chain_databaseSize.value = await this.db.size()


Do we know what performance of this size check is like? Just want to make sure we're not adding extra time to addBlock

I just checked the operation takes 0.031458 milliseconds, it should be negligible.

danield9tqh · 2023-05-25T21:18:02Z

ironfish/src/metrics/metricsMonitor.ts

@@ -66,6 +67,7 @@ export class MetricsMonitor {

    this.mining_newBlockTemplate = this.addMeter()
    this.chain_newBlock = this.addMeter()
+    this.chain_databaseSize = new Gauge()


We should probably find a way to initialize this to the size of the chain db. Maybe set it when the database starts up so that we don't get 0 values sent to telemetry if it's not initialized

Added in the blockchain.open()

Add test:perf command Remove flags Use jest-junit test reporter Rename Update path Add permission to github action parent fce9eee author ygao76 <yajun@ironfish.network> 1684974186 -0700 committer ygao76 <yajun@ironfish.network> 1686272927 -0700 Set fail on error to true Upload test report Increase perf test timeout Create new cmd test:perf:report Refactor transaction perf tests Only run transaciont Add test baseline metrics and refactor tests checks if submitted block is heavier when chain changed (#3923) * checks if submitted block is heavier when chain changed when a miner submits a block to the mining manager the manager discards the block if the chain has changed. that is, if the submitted block no longer connects to the chain head. however, if the submitted block is heavier than the curremt chain head then the manager should add this block to the chain. - removes unnecessary check that chain head exists - does not discard submitted block if it is heavier than the current chain head * sets work on submitted block before checking isBlockHeavier the 'work' on a block header is only set in 'chain.connect', so checking 'isBlockHeavier' in 'submitBlockTemplate' will always return false unless we set the work first. only sets the work and checks if the block is heavier if the chain has changed. this is to avoid duplicating this work if possible and avoid looking up the previous block if possible adds rpc to follow transaction gossip stream (#3925) * adds rpc to follow transaction gossip stream creates an event rpc endpoint to stream serialized transactions as the peer network receives them in gossip messages. adds an event to the peer network that emits each transaction that the node receives. emits all transaction gossip without verifying or waiting for the mempool to accept the transaction. supports syncing pending transactions to the api. * emits transaction gossip after filtering for duplicates removes need for LRU in RPC endpoint if we leverage the filtering in the peer network logic matches the logic for onBlockGossipReceived event, but emits event before validating transaction fixes flaky onTransactionGossipReceived test (#3933) emits event from peer network directly in test logic for RPC endpoint instead of indirectly trying to add the transaction into the peer network to force the event to emit. regenerates fixtures and fixes outdated wallet tests (#3931) changes in #3830 inadvertently fixed a bug whereby transactions that did not send any iron (e.g., mint transactions with 0 fee) would still include a spend to cover the fee of 0. this bug created fixtures for several tests in which fee was 0, but a spend was included anyway. after regenerating fixtures following the fix, these tests failed. - do not expect transaction assetBalanceDeltas to include the native asset when fee is 0 and no iron is sent or received (i.e., mint or burn with 0 fee) - fixes getTransactionType to check whether the account was the sender on any notes in case the transaction does not include spends from the account bumps package versions for v1.3.0 (#3934) Add chain database size to telemetry (#3928) * Add chain database size to telemetry * Add unit test * Rename * Log chainDB size in chain.open() Increase timeout" Avoid teardown DB after each test case List all test cases in report Use JUnit Report Action for github action because the previous one does not have detailed report for each test case Use Publish Test Results for github action Show test result output Use default json file Add json output formatter" Reduce output Print only test cases output Only print test cases Update thredshold

* Add chain database size to telemetry * Add unit test * Rename * Log chainDB size in chain.open()

ygao76 added 2 commits May 23, 2023 22:19

Add chain database size to telemetry

c36bc18

Add unit test

d4dc208

ygao76 requested a review from a team as a code owner May 24, 2023 05:30

ygao76 commented May 24, 2023

View reviewed changes

ironfish/src/blockchain/blockchain.ts Outdated Show resolved Hide resolved

rohanjadvani reviewed May 24, 2023

View reviewed changes

Rename

c6a30b6

danield9tqh reviewed May 25, 2023

View reviewed changes

Log chainDB size in chain.open()

2ced575

danield9tqh approved these changes May 25, 2023

View reviewed changes

ygao76 merged commit ae6cb2b into staging May 25, 2023
3 checks passed

ygao76 deleted the yajun/ifl-1006-create-metric-for-chaindb-size branch May 25, 2023 22:10

rohanjadvani approved these changes May 25, 2023

View reviewed changes

jowparks pushed a commit that referenced this pull request Jun 21, 2023

Add chain database size to telemetry (#3928)

edf9340

* Add chain database size to telemetry * Add unit test * Rename * Log chainDB size in chain.open()

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add chain database size to telemetry #3928

Add chain database size to telemetry #3928

ygao76 commented May 24, 2023 •

edited

rohanjadvani May 24, 2023

ygao76 May 24, 2023 •

edited

danield9tqh May 25, 2023 •

edited

ygao76 May 25, 2023

danield9tqh May 25, 2023

ygao76 May 25, 2023

danield9tqh May 25, 2023

ygao76 May 25, 2023

	readonly chainDatabaseSize: Gauge
	readonly chain_databaseSize: Gauge

		@@ -406,6 +406,8 @@ export class Blockchain {

		this.resolveOrphans(block)

		this.metrics.chain_databaseSize.value = await this.db.size()

Add chain database size to telemetry #3928

Add chain database size to telemetry #3928

Conversation

ygao76 commented May 24, 2023 • edited

Summary

Testing Plan

Documentation

Breaking Change

rohanjadvani May 24, 2023

Choose a reason for hiding this comment

ygao76 May 24, 2023 • edited

Choose a reason for hiding this comment

danield9tqh May 25, 2023 • edited

Choose a reason for hiding this comment

ygao76 May 25, 2023

Choose a reason for hiding this comment

danield9tqh May 25, 2023

Choose a reason for hiding this comment

ygao76 May 25, 2023

Choose a reason for hiding this comment

danield9tqh May 25, 2023

Choose a reason for hiding this comment

ygao76 May 25, 2023

Choose a reason for hiding this comment

ygao76 commented May 24, 2023 •

edited

ygao76 May 24, 2023 •

edited

danield9tqh May 25, 2023 •

edited