Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discussion: Potential USDTs (User Static Defined Traces) for Core #20981

Closed
0xB10C opened this issue Jan 21, 2021 · 21 comments
Closed

Discussion: Potential USDTs (User Static Defined Traces) for Core #20981

0xB10C opened this issue Jan 21, 2021 · 21 comments
Labels

Comments

@0xB10C
Copy link
Contributor

0xB10C commented Jan 21, 2021

USDTs are tracepoints and behave similar to trace logging (no performance impact if not used), but can pass data into scripts running in a Linux kernel VM for collection and further evaluation. The required build system changes and a tracing framework for User Static Defined Traces (USDTs) were merged in #19866 and some additional discussion took place in #20960. Simpler scripts, such as collecting and printing metrics as histogram, can be created with the bpftrace tool. Complexer bcc scripts are written in C and loaded into the kernel with, for example, the bcc Python module where the passed data can be processed.

Following the core-dev-meeting discussion on 21st Jan 2021, the goal here is to list potentially interesting USDTs and what data they should pass. The USDT should have:

  1. a clear use-case and motivation
  2. a semi-stable API (order of arguments don't change so people can rely on these for scripting)
  3. no expensive operations added specifically for this USDT (no parsing, serialization, ... - just passing of already avaliable data)
  4. a documentation in docs/tracing.md
  5. (ideally) a usage example as bftrace or bcc script
  6. (need to double check this) a maximum of 512 bytes passed (max stack size in bpftrace)

While some kind of testing of the USDT examples would be good, root privileges are required to load programs in the kernel VM. Might still be possible to run them in a CI, though?

What could be useful USDTs? What data should they pass? Where should they be placed in the code?

@0xB10C 0xB10C added the Feature label Jan 21, 2021
@0xB10C
Copy link
Contributor Author

0xB10C commented Jan 23, 2021

Some of the previously proposed USDTs by @jb55 from #19866:

net:push_message

  • traces all sent P2P messages as TRACE5:
    • int node_id
    • char* node_addr
    • char* msg_type
    • size_t msg_size
    • uint8_t* msg_data

void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg&& msg)


net:process_message

Traces all received P2P messages as TRACE5:

  • int node_id
  • char* node_addr
  • char* msg_type
  • size_t msg_size
  • uint8_t* msg_data

bool PeerManager::ProcessMessages(CNode* pfrom, std::atomic<bool>& interruptMsgProc)

#19866 has an example bpftrace script using net:process_message and net:push_message.


validation:connect_block

Traces all blocks for which ConnectBlock() is called as TRACE4:

  • uint32 height
  • int tx_count
  • uint8_t[32] hash
  • uint8_t[32] prevhash
  • (more header fields?)

Passing the full serialized block is not possible as this could be bigger than both the bcc and bftrace stack (the kernel VM crashes gracefully, no stack overflow). While the serialized block header would not be bigger than the stack size, header serialization would need to happen even when no USDTs are attached. This has an performance impact and should not be done.

@jb55 used an earlier version of validation:connect_block to measure IBD performance.

Depending on placement in ConnectBlock(), this could either trace to-be-connected blocks or connected blocks (no error while connecting). In either way, the documentation should make it clear for which blocks the data is passed.

bool CChainState::ConnectBlock(const CBlock& block, BlockValidationState& state, CBlockIndex* pindex,

@0xB10C
Copy link
Contributor Author

0xB10C commented Jan 23, 2021

@laanwj mentioned that traces for dis- and connections of peers would be useful too. I'll have a look where these could be placed. I guess there are some places where there is already net debug logging which can be used for orientation. @jlopp's statoshi diff could be helpful to look at too.

Also interesting:

@laanwj
Copy link
Member

laanwj commented Jan 24, 2021

@jlopp's statoshi diff could be helpful to look at too.

FWIW the statoshi diff can be found here: https://github.com/jlopp/statoshi/compare/1b90a7b61a37d450bbe5b485dcc9b8c591567e0d..28b88353a78ae19e3c5379bbda6b72f4fef65a8c

@jb55
Copy link
Contributor

jb55 commented Jan 25, 2021 via email

@laanwj
Copy link
Member

laanwj commented Jan 26, 2021

Would be an interesting project.

I agree! This would be my preferred way to do this kind of reporting, to be honest. It'd flexible and built into the operating system. It has low impact on the code and low impact on performance. It could potentially work for all software.

One reason Statoshi never got merged is because it's just too much hassle to integrate and maintain a Bitcoin Core specific framework for this. Not because it's not a great thing to have.

@laanwj
Copy link
Member

laanwj commented Mar 19, 2021

Besides networking, one major thing that would be interesting to monitor through tracepoints is dbcache usage. This is useful to get metrics for hardware optimization of validation (e.g. to find where bottlenecks are). A few things come to mind:

  • Gross statistics on changes (this is in the log but it's good to have a well defined interface for it)
    • how much memory is used of how much
    • how many UTXOs are cached
  • Before/afters for timing:
    • Insertions and deletions of UTXOs in the CoinsCache with associated information
    • Queries: hits and misses
    • Flushes

@jb55
Copy link
Contributor

jb55 commented Apr 12, 2021 via email

@laanwj
Copy link
Member

laanwj commented Apr 12, 2021

I've been using heaptrack[1] for memory tracing, it hooks into malloc calls which saves us from putting allocation traces everywhere.

That sounds useful, however I think it would show gross malloc usage, not specifically that of the utxo set? Or can it drill down to more specific things as well?

@jb55
Copy link
Contributor

jb55 commented Apr 12, 2021 via email

@0xB10C
Copy link
Contributor Author

0xB10C commented May 5, 2021

I've started working on a patch that adds tracing documentation (based on laanwj's work) and the net:push_message + net:process_message + more (based on jb55's work) together with both bpftrace and bcc examples. Goal is to have some initial and easy to work with examples of how USDT in Core could look like. There will be Bitcoin Core Review Club on USDTs later this month.

@jb55
Copy link
Contributor

jb55 commented May 5, 2021 via email

@0xB10C
Copy link
Contributor Author

0xB10C commented May 21, 2021

laanwj mentioned that traces for dis- and connections of peers would be useful too. I'll have a look where these could be placed. I guess there are some places where there is already net debug logging which can be used for orientation.

@jonatack added that next to normal disconnects, tracepoints for peer evictions would be useful too. He provided input on what tracepoint arguments he'd find useful:

  • message
  • connection direction and type,
  • peer id
  • address

@jonatack
Copy link
Contributor

jonatack commented May 21, 2021

@0xB10C thanks! Yes, my wishlist ties in a bit with issue #20576. I wish we had those in our net logging, separated into high/low frequency/level events.

Edit: could add the network to the args list.

@0xB10C
Copy link
Contributor Author

0xB10C commented Jul 27, 2021

I was looking at this a long time ago. In theory you could get the same functionality as statoshi (stats gathering via prometheus) with https://github.com/cloudflare/ebpf_exporter Would be an interesting project.

I agree! This would be my preferred way to do this kind of reporting, to be honest. It'd flexible and built into the operating system. It has low impact on the code and low impact on performance. It could potentially work for all software.

One reason Statoshi never got merged is because it's just too much hassle to integrate and maintain a Bitcoin Core specific framework for this. Not because it's not a great thing to have.

I have a demo for Prometheus metrics via eBPF based on the tracepoints added in #22006 in 0xB10C/bitcoind-observer. It's written in Rust and uses the bcc crate under the hood which recently added USDT support.

A live demo can be fund on bitcoind.observer. Currently showing block validation during IBD and per message type P2P traffic stats. I was surprised who well it works. Might be a real alternative to statoshi once we add more tracepoints about e.g. UTXO set, peer dis/connections, and so on.

laanwj added a commit to bitcoin-core/gui that referenced this issue Jul 27, 2021
…tion on User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin/bitcoin#19866 earlier this year. Issue bitcoin/bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin/bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
sidhujag pushed a commit to syscoin/syscoin that referenced this issue Jul 28, 2021
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
@0xB10C
Copy link
Contributor Author

0xB10C commented Aug 10, 2021

Tracepoints for Bitcoin Core startup and shutdown could be useful too. IDB latching to true/false too.

@0xB10C
Copy link
Contributor Author

0xB10C commented Aug 13, 2021

Banman tracepoints should be interesting too.

maflcko pushed a commit to maflcko/bitcoin-core-with-ci that referenced this issue Sep 9, 2021
15b8ba5 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
56f0a04 [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in bitcoin/bitcoin#20981 (comment). The first tracepoints were added in bitcoin#22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

Top commit has no ACKs.

Tree-SHA512: 53ba6c2811c7fed08aa608a888bff83214e754b0f50621e83e4f81e95bc22d21f7f956096d12858d8e85240aa226431845995682eaaf55e93016c5b8f4c9282b
maflcko pushed a commit to maflcko/bitcoin-core-with-ci that referenced this issue Sep 9, 2021
15b8ba5 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
56f0a04 [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in bitcoin/bitcoin#20981 (comment). The first tracepoints were added in bitcoin#22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

Top commit has no ACKs.

Tree-SHA512: 53ba6c2811c7fed08aa608a888bff83214e754b0f50621e83e4f81e95bc22d21f7f956096d12858d8e85240aa226431845995682eaaf55e93016c5b8f4c9282b
laanwj added a commit to bitcoin-core/gui that referenced this issue Nov 29, 2021
2bc51c5 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
a26e8ee [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in bitcoin/bitcoin#20981 (comment). The first tracepoints were added in bitcoin#22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

ACKs for top commit:
  laanwj:
    Code and documentation review ACK 2bc51c5

Tree-SHA512: d6b4f435d3260de4c48b36956f9311f65ab3b52cd03b1e0a4ba9cf47a774d8c4b31878e222b11e0ba5d233a68f7567f8a367b12a6392f688c10c11529341e837
RandyMcMillan pushed a commit to RandyMcMillan/mempool-tab that referenced this issue Dec 23, 2021
4769342 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
7d79993 [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in bitcoin/bitcoin#20981 (comment). The first tracepoints were added in bitcoin#22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

ACKs for top commit:
  laanwj:
    Code and documentation review ACK 4769342

Tree-SHA512: d6b4f435d3260de4c48b36956f9311f65ab3b52cd03b1e0a4ba9cf47a774d8c4b31878e222b11e0ba5d233a68f7567f8a367b12a6392f688c10c11529341e837
@0xB10C
Copy link
Contributor Author

0xB10C commented Apr 7, 2022

NYDIG has a $4500 bounty for mempool tracepoints.

https://nydig.com/bounties/add-bpf-tracepoints-for-the-mempool

@glozow
Copy link
Member

glozow commented Apr 22, 2022

Some mempool-related telemetry that I think would be helpful:

  • Transaction traversal count and memory usage when calculating mempool descendants and ancestors. This can inform us about performance/memory benefits of epochs and our current ancestor/descendant limits. I'm particularly interested to see information about this for reorgs.
  • Records of prevouts of transactions added to the mempool. This would let us track things like "how many times is a tx replaced / how many bytes were relayed before this payment confirmed?"
  • Signature and script cache usage.
  • {Resource usage, rate of valid transactions} of our data structures holding invalid/unvalidated transactions: m_orphanage, vExtraTxnForCompact, m_recent_rejects filter
  • Query and hit rate for LockPoints cached in mempool entries
  • The "shape" of packages in mempools, e.g. number of direct parents vs total ancestors

NYDIG has a $4500 bounty for mempool tracepoints.

It's nice that there's a bounty to encourage contributions. I'm glad the bounty stresses the importance of a clearly-defined use case. We already have logs for {added, removed} mempool transactions, but there's definitely room for improvement. Hopefully NYDIG would be okay with different but still useful instrumentation?

@russeree
Copy link
Contributor

russeree commented Apr 23, 2022

@glozow those would be some awesome use cases for ebpf tracepoints. Quite technical in their scope. I would like to take to working on a few of these points if desired. Number 1 has most of my interest.

In regards to #24935

In regards to my existing PR for mempool tracepoints should I remove mempool additions/eviction tracepoints as something to even pursue? Even if the NYDIG bounty no longer qualifies I want to contribute. Learning about the power of ebpf,usdt, and bcc has been a great addition to my skillset and has captured my interest.

Existing logging/tracing solutions in the mempool (adds & evictions)

The current in-mempool implementation seems to be using the ZMQ interface but doesn't currently provide a complete solution to trace all adds/removals from the pool.

There are 2 functions for broadcasting mempool events referenced in txmempool.cpp and one of them is not even used. These are CValidationInterface::TransactionAddedToMempool and CZMQNotificationInterface::TransactionRemovedFromMempool

  • Currently CValidationInterface::TransactionAddedToMempool is not called anywhere in this repo
  • CZMQNotificationInterface::TransactionRemovedFromMempool is only called if the tx was not removed for the reason of 'BLOCK'. The current implementation instead leaves a multiline comment stating that to find the evicted txs from a block, look at the latest txs in the mined block. A valid solution but why not leverage a trace to to a sandboxed kernel program with extremely little overhead. Even if the user doesn't need this functionality it can even be compiled out into a nop instruction if built without tracing support.

On the topics of block evictions, they seem to be one hardest events to trace/log if the 256 bit txid is included since the volume of data being streamed is quite high with up to thousands of evictions in a very short time. Using a ringbuffer and a bpffs queue in tandem all mempool evictions can be logged in completeness and I am pretty sure they maintain the order eviction was requested in.

#24935 eBPF solution to logging mempool additions/evictions

Firstly by using eBPF tracepoints the overhead of tracing an event is minimal and isolated. bpftrace, python, and bcc scripts can be written in any manner desired to accommodate the collection of performance stats, testing, and custom user metrics. These custom scripts have no impact on the underlying functionality of the users current bitcoind build as all data is being passed though the kernel sandbox and the bitcoin core traces are a unidirectional event including a trace of all the txs being passed in and out of the mempool.

The example .py script implemented to trace mempool events in #24935 logs each request for a tx addition/removal from the mempool and the relevant information to validate/compare the current mempool with it's output. These logs are complete without erroneous data. The whole extended test suite was ran and It is possible to log all 328416 function calls to add or evict txs from the mempool. As a note even if these traces are determined to not be a candidate using some of the methods included in log_mempool_activity.py would be very useful in fixing some of the existing scripts that display incomplete data and/or have dropped events.

Thanks

Edit: a word...phrases

@0xB10C
Copy link
Contributor Author

0xB10C commented Dec 1, 2022

ref: #26531 which is adds mempool tracepoints too (#24935 has been closed)

vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Nov 1, 2023
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Nov 6, 2023
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Nov 21, 2023
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Nov 25, 2023
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
@0xB10C
Copy link
Contributor Author

0xB10C commented Apr 10, 2024

Can probably close this as there hasn't been much activity. Many of the tracepoints proposed here have been implemented in some form - other's haven't sparked enough interest.

@0xB10C 0xB10C closed this as completed Apr 10, 2024
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Jun 15, 2024
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Jun 15, 2024
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Jun 15, 2024
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
vijaydasmp pushed a commit to vijaydasmp/dash that referenced this issue Jun 17, 2024
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants
@jb55 @laanwj @jonatack @russeree @0xB10C @glozow and others