[Core Manager] Watchers (Developer Mode)

[faustbrian]

Introduction

Watchers are a helpful tool for developers as it shows a lot of relevant information about what their node is doing like what block payloads were received, what errors recently were logged, if their database queries are executing slowly or if their Webhook job queue is clogged with a long waiting time for upcoming jobs.

This feature will only be enabled if developer mode is enabled as it takes up extra resources from Core to pass all this information around. It's advised to not use developer mode on weak nodes or nodes with small disks as a lot of information can accumulate rather quickly under high load.

All of those events that a watcher captures will be logged with a full stack-trace of their execution origin and the data that has been processed. This makes it very easy for developers and node maintainers to figure out where something went wrong without having to dig through their logs or adding more log statements.

Block Watcher

The block watcher will listen for all block events which are things like applying, reverting and rejecting.

Those logs will contain relevant information like the event name, id, height, number of transactions and the total value of a block in ARK based on amounts and fees and a full dump of the processed data for advanced analysis or use.

Transaction Watcher

The transaction watcher will listen for all transaction events which are things like applying, reverting and rejecting.

Those logs will contain all relevant information like the event name, id, type, sender, recipient, time and a full dump of the processed data for advanced analysis or use.

Wallet Watcher

The wallet watcher will listen for all wallet events which are things like applying transactions, updating vote balances, updating signatures or balances after incoming or outgoing transactions.

Those logs will contain all relevant information like the event name, address, public key, type of signatures and a full dump of the processed data for advanced analysis or use.

Round Watcher

The round watcher will listen for all round events which are things like applying and reverting a round which also involves reverted blocks.

Those logs will contain all relevant information like the event name, round number and all delegates that participated in that round and a full dump of the processed data for advanced analysis or use.

Error Watcher

The error watcher will simply log all errors that occur with a full stack-trace and the location from which the error originated. This will include information like the filename, line number and type of error that occurred together with the severity of it which would generally be a handled or unhandled exception.

Log Watcher

The log watcher will collect every line that is written into the log and parse it to present it in a human-readable format without the cognitive work of having to go through long logs on a server or grepping them for specific keywords.

A search will be provided to quickly filter logs for specific keywords.

Query Watcher

The query watcher will log all database queries with the function that issued them followed by some metadata like execution time and a full log of the raw sql query and what database engine was used.

Job & Queue Watcher

The job and queue watcher will monitor all jobs that are queued via the newly introduced jobs which have been mentioned in the Core 3.0 document. Those logs will include information like when a job was executed, how long it took, how many retries occurred and the type of queue driver that was used.

This will initially be mainly useful to see the transaction API and pool doing its job after the introduction of a transaction queue in 2.6.

Schedule Watcher

The schedule watcher will monitor all tasks that are scheduled via the newly introduced scheduler that has been mentioned in the Core 3.0 document. Those logs will contain information about which command was executed, how long it took and an output log that the command itself can provide.

Webhook Watcher

The webhook watcher will be similar to the job and queue watcher as webhooks are essentially just jobs. Additionally, to jobs the webhooks will contain information like which event and conditions triggered them, to where they were sent, if the transmission failed or succeeded and how long it took.

[faustbrian]

@sebastijankuzner last thing for this is that each type of watcher should have the ability to be disabled. For example I should be able to disable watching all transaction and round events.

[sebastijankuzner]

There are 3 main options. We can filter out events that starts with given prefix (transactions,rounds) before they enter the database with blacklisting. Same thing can be achieved with proper querying on database. Another way is that we disable some structures, that are responsible for emitting. In our case we can disable custom watcher emitters:

• logger emitter
• database log emitter
• wallet emitter

Events originating from the other parts of the system cannot be disabled that way.

I am considering that we need to disable structures responsible for emitting to increase node performance. Filtering on other side can be achieved with querying the database. In that case we can work more on the query support (NOT LIKE and NOT EQUAL for blacklisting).

[faustbrian]

#3829