Help filter out logs

[pgrange]

Why

It is hard to read through the logs as there are quite a lot of them and, depending on the situation, it can be hard to separate noise from information.

Since it's json stuff we should explore how to help filter out interesting logs more easily and, at least, give some advice in documentation.

What

Explore how to filter out interesting logs more easily and share some advice in documentation.

[ch1bo]

Besides using tools like grep and jq, there is also the log-filter executable in the hydra-cluster package. But I have never used it by myself. @abailly-iohk did create it.

[GeorgeFlerovsky]

More options.

Simpler for dev: https://lnav.org/

Advanced for prod: https://prometheus.io/

[ghost]

@GeorgeFlerovsky Actually, we already expose Prometheus compatible metrics and it does not by itself solves the log verbosity issue.

From our recent experience troubleshooting opening heads, I'd say this issue is high priority.

[ghost]

I would like to act upon this red bin item but I think the proper way to go is rather to work on improving the logs. Here are a couple ideas:

• Log BeginXXX with an eventId and use that in the corresponding EndXXX log entry to reduce amount of data in the latter and easily correlate pair of events/effects
• Review the data logged with each entry and reduce it to things which are actually interesting to log
• Do not log full transactions wherever possible but only txids, or trim down the amount of information in the tx (eg. full scripts might not be useful?)
• Remove chatty Ping network messages and only log state transitions (eg. when a node becomes connected/disconnected)

[GeorgeFlerovsky]

I think those are sensible ideas.

@uhbif19 what do you think?

[uhbif19]

@GeorgeFlerovsky @pgrange

That is cool but does not cover "share some advice in documentation" part. I do not understand what is the supposed algorithm for debugging Hydra issues using logs. Like issues on node connectivity or PostTxOnChainFailed.

[GeorgeFlerovsky]

@GeorgeFlerovsky @pgrange

That is cool but does not cover "share some advice in documentation" part. I do not understand what is the supposed algorithm for debugging Hydra issues using logs. Like issues on node connectivity or PostTxOnChainFailed.

@uhbif19
What would be your preferred way to debug hydra head issues?

[ghost]

@uhbif19 I hear you. I plan to start a section in the docs on Operating & Troubleshooting Hydra which shall contain some advices and tips. More will come as gain experience and feedback from users.

[uhbif19]

@GeorgeFlerovsky

I do not know, this is why I am asking. Like I want to know for which error/success messages I should look for in first place to check if something is broke.

[ghost]

We went through some troubleshooting session lastly while trying to open a head on mainnet, I have some notes I will transfer to docs.

[ghost]

@uhbif19 @GeorgeFlerovsky I have started an Operating Hydra Node page with some information about logs and how to troubleshoot common issues: https://github.com/input-output-hk/hydra/blob/abailly-iohk/add-operating-node-section/docs/docs/getting-started/operating-hydra.md

Do you think it's missing some important points? What kind of information could be added/changed?