Add optional prometheus metrics

[rklaehn]

I had a hard time figuring out a set of metrics that is not overwhelming, but this seems to work reasonably well. It is not very fine grained, but it gives you a rough idea what the thing is doing overall.

E.g. this is what you get when running the pond integration test with 8 local linux hosts yields the following metrics (dumping more frequently so I see them):

AX_CI_HOSTS=8linux.yaml npm test pond | grep -A 300 got_metrics | grep -A 1 time_sum

...

2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_block_get_time_sum 11.24307093500005
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_block_get_time_count 48033
--
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_block_put_time_sum 1.7594724890000006
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_block_put_time_count 1559
--
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_branch_load_time_sum 2.9276649789999785
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_branch_load_time_count 8291
--
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_branch_store_time_sum 1.3235776810000015
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_branch_store_time_count 1037
--
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_leaf_load_time_sum 5.6611070490000515
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_leaf_load_time_count 34957
--
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_leaf_store_time_sum 0.6494339429999996
2021-07-14T07:17:13.669Z node local-linux-8 Actyx stdout: banyan_leaf_store_time_count 522

so it spends most of the db time getting blocks, and focused on banyan it spends most of the time loading leafs (this metric is the total time including decryption, decompression and decoding).

We are not going to optimize things just now, but if we wanted to this would give us a good idea where to start.

[dvc94ch]

LGTM

[rklaehn]

@rkuhn given the metrics discussion, I hope you agree that this is still in principle the right thing to do. I don't really care that much if this ever ends up in a proper prometheus graph, I just use it as a tool to get a rough idea what the thing is doing, see above.

Note: We have kinda standardized on prometheus as a tool for gathering metrics in ipfs-embed and below ecosystem, see https://github.com/ipfs-rust/ipfs-embed/blob/master/src/telemetry.rs .

[wngr]

You can also use the weight-caches metrics feature, see https://github.com/Actyx/weight-cache/blob/master/Cargo.toml#L20-L22

[rkuhn]

So the approach would be to pass the default registry to Banyan to get the metrics of ipfs-embed as well?

Overall looks reasonable, but I haven’t yet got my hands dirty actually using prometheus gathering functionality.

[rklaehn]

So the approach would be to pass the default registry to Banyan to get the metrics of ipfs-embed as well?

We already have ipfs-embed metrics. You would just pass the default registry to this register as well. Which is what I did to get the numbers above, in addition to a bunch of one-off changes in cosmos to get the emission frequency up...

Not sure what is the overhead of registering multiple registries. If it is low overhead, we could have 2 registries, one with just the essential stuff that goes via ephemeral events to troubleshoot in case of serious problems, and one with everything and the kitchen sink that can only be locally polled by prometheus.

[rkuhn]

It seems to me that the ecosystem has already made a different choice: since some libraries already use the default registry, we are locked into using that as well (which is what I meant above). A secondary registry wouldn’t make sense to me.