Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tracing: first tracepoints and documentation on User-Space, Statically Defined Tracing (USDT) #22006

Merged
merged 4 commits into from Jul 27, 2021

Conversation

0xB10C
Copy link
Contributor

@0xB10C 0xB10C commented May 20, 2021

This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

Context

The TRACEx macros for tracepoints and build system changes for USDT were merged in #19866 earlier this year. Issue #20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a core-dev-meeting discussion on 21st Jan, 2021.

USDT? Stablecoin?

User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute eBPF programs in a kernel VM once the tracepoint is called.

This PR includes, for example, tracepoints for in- and outbound P2P messages.

USDT and eBPF Overview
======================

                ┌──────────────────┐            ┌──────────────┐
                │ tracing script   │            │ bitcoind     │
                │==================│      2.    │==============│
                │  eBPF  │ tracing │      hooks │              │
                │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
            1.       │      │ 4.              │ │ ...          │   │ to eBPF
    User    compiles │      │ pass data to    │ └──────────────┘   │ program
    Space    & loads │      │ tracing script  │                    │
    ─────────────────┼──────┼─────────────────┼────────────────────┼───
    Kernel           │      │                 │                    │
    Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                │  │  eBPF program                         │◄──────┘
                │  └───────────────────────────────────────┤
                │ eBPF kernel Virtual Machine (sandboxed)  │
                └──────────────────────────────────────────┘

1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
2. The eBPF program hooks into one or more tracepoints
3. When the tracepoint is called, the arguments are passed to the eBPF program
4. The eBPF program processes the arguments and returns data to the tracing script

The two main eBPF front-ends with support for USDT are bpftrace an BPF Compiler Collection (BCC). BCC is used for complex tools and daemons and bpftrace is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

This PR adds three tracepoints:

  • net:inbound_message
  • net:outbound_message
  • valildation:block_connected

See doc/tracing.md and contrib/tracing/ for documentation and example tracing scripts.

Open Questions (Not in scope for this PR)

  • How to use these tracepoints under macOS?
  • Release builds with USDT support?
  • Should and can the tracepoints be automatically tested?

Todo (before undraft)

  • bcc example showing how to read raw P2P messages up to 32kb
  • document that you need sys/sdt.h from systemtap for USDT support in Bitcoin Core (apt install systemtap-sdt-dev on debian-like). See 933ab8a
  • release notes?

@practicalswift
Copy link
Contributor

Concept ACK

Excellent work! :)

@adamjonas
Copy link
Member

Concept ACK. Nice work!

@jonatack
Copy link
Contributor

jonatack commented May 21, 2021

Concept ACK. Saw a live demo of this and it seems to be very cool.

contrib/tracing/README.md Outdated Show resolved Hide resolved
Copy link
Contributor

@jnewbery jnewbery left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. Excellent work, @0xB10C!

src/net.cpp Outdated
@@ -2959,11 +2960,21 @@ bool CConnman::NodeFullyConnected(const CNode* pnode)
void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg&& msg)
{
size_t nMessageSize = msg.data.size();
LogPrint(BCLog::NET, "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.m_type), nMessageSize, pnode->GetId());
auto sanitizedType = SanitizeString(msg.m_type);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to sanitize this? the message types for all outgoing messages are hardcoded in our binary so should all be safe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, will remove it from the tracepoint and logging.

@0xB10C 0xB10C force-pushed the 2021-05-initial-usdt-support branch from 92dc3a6 to cdb4005 Compare May 21, 2021 10:32
doc/tracing.md Outdated Show resolved Hide resolved
@0xB10C
Copy link
Contributor Author

0xB10C commented May 24, 2021

We'll cover this PR in the PR Review Club this Wednesday (May, 26th).

https://bitcoincore.reviews/22006

@0xB10C 0xB10C force-pushed the 2021-05-initial-usdt-support branch 5 times, most recently from 0cc7540 to 037ece9 Compare May 24, 2021 14:18
@0xB10C
Copy link
Contributor Author

0xB10C commented May 24, 2021

I added an example (log_raw_p2p_msgs.py) showcasing and documenting some USDT and eBPF limitations. Tracing P2P messages was one of the motivations for USDT in #19866 as an potential alternative to #19509. Some P2P messages like tx and block can be larger than the maximum allocation size of 32kb in eBPF. This means, we can't submit the complete P2P message to tracing scripts from the eBPF VM. Smaller messages aren't a problem. Larger messages are cut off after about 32kb in the log_raw_p2p_msgs.py example and a warning is printed.

Something to be aware of when adding tracepoints and developing tracing scripts.

@0xB10C 0xB10C force-pushed the 2021-05-initial-usdt-support branch from 037ece9 to a076eb6 Compare May 24, 2021 17:33
@laanwj
Copy link
Member

laanwj commented May 25, 2021

Impressive work! Thanks for getting back to this.

How to use these tracepoints under macOS?

FWIW I think it's ok to keep this out of the scope of this PR. Can extend this later. Let's focus on Linux first.

Release builds with USDT support?

Same for this. I think it's a no-brainier to have these probes in the release builds eventually, but no need to do so here. Mind that currently we have a duplicate build system (guix and gitian) and the focus for 0.22 (which is not that far away!) is ironing out issues for the guix build, as well as trying to do a guix build for the release. This is a temporary situation that complicates adding anything.

Should and can the tracepoints be automatically tested?

I'd say "yes". These are meant to be a documented and semi-stable APIs, and APIs not covered by tests tend to code rot over time.

Copy link
Contributor

@jb55 jb55 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK!

src/validation.cpp Show resolved Hide resolved
@@ -1975,6 +1976,16 @@ bool CChainState::ConnectBlock(const CBlock& block, BlockValidationState& state,
int64_t nTime6 = GetTimeMicros(); nTimeCallbacks += nTime6 - nTime5;
LogPrint(BCLog::BENCH, " - Callbacks: %.2fms [%.2fs (%.2fms/blk)]\n", MILLI * (nTime6 - nTime5), nTimeCallbacks * MICRO, nTimeCallbacks * MILLI / nBlocksTotal);

TRACE7(validation, block_connected,
block.GetHash().ToString().c_str(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably fine in this case, but we should avoid passing formatted data to USDTs when we can format it on the bpf side for performance reasons, especially in tight loops. It looks like we're already passing GetHash().data(), can we format this on the ebpf side? I remember having trouble doing this in bpftrace, but perhaps that's a temporary issue until bpftrace gets better buffer formatting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, ideally we'd only pass the GetHash().data() as first argument.

It looks like we're already passing GetHash().data(), can we format this on the ebpf side? I remember having trouble doing this in bpftrace, but perhaps that's a temporary issue until bpftrace gets better buffer formatting.

Right, the current %r printf formatting for buffers in bpftrace isn't suitable for printing header hashes and txids. Adding a new formatting function to bfptrace similar to this as added in bpftrace/bpftrace#1107 doesn't look to hard. However, the block hashes and txids would still be in little-endian.

Copy link
Contributor

@rajarshimaitra rajarshimaitra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Concept ACK.

I have tested and ran the examples.

Below are some nits and bumps I faced.

Overall I think having some standard tracing points and readily available processing scripts can be very helpful for developers.

I liked the p2p message monitor, and excited to see what other tracing usage others come up with.

contrib/tracing/log_raw_p2p_msgs.py Outdated Show resolved Hide resolved
contrib/tracing/README.md Outdated Show resolved Hide resolved
contrib/tracing/log_raw_p2p_msgs.py Outdated Show resolved Hide resolved
doc/tracing.md Show resolved Hide resolved
works fine).

```
$ bpftrace contrib/tracing/log_p2p_traffic.bt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am getting the following error here

$ sudo bpftrace ./contrib/tracing/log_p2p_traffic.bt
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted

trace points seem to be defined. Node is running and talking to peers in testnet

Not sure what's throwing this error here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reporting, I'll have a look. Whats your bpftrace version (bpftrace --v)?

Copy link
Member

@laanwj laanwj Jul 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is this supposed to find the process to attach to? Or would it log metrics for any bitcoind instance running on the system? (it seems so!) Unless you specify -p, I guess.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bpftrace scripts include a relative path to the bitcoind binary in ./src/bitcoind e.g. usdt:./src/bitcoind:validation:block_connected.

This is mentioned on the examples page.

Copy link
Contributor

@jb55 jb55 Jul 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think something more general would be usdt:*bitcoind:net:inbound_message and then

bpftrace -p $(pgrep bitcoind) log_p2p_traffic.bt

eg this uprobe:

sudo bpftrace -p $(pgrep bitcoind) -e '
  uprobe:*bitcoind:"PeerManager::ProcessMessages(CNode*, std::atomic<bool>&)" { printf("node %p %d\n", arg1, *arg2) }
'

seems to work for me...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh hmm I don't think you need *bitcoind, this also worked for me?

sudo bpftrace -p $(pgrep bitcoind) -e '
  uprobe:"PeerManager::ProcessMessages(CNode*, std::atomic<bool>&)" { printf("node %p %d\n", arg1, *arg2) }
'

contrib/tracing/README.md Show resolved Hide resolved
@jamesob
Copy link
Member

jamesob commented May 29, 2021

Concept ACK - very cool. Will review soon.

@0xB10C 0xB10C force-pushed the 2021-05-initial-usdt-support branch from a076eb6 to 3e57d64 Compare June 7, 2021 11:09
@0xB10C 0xB10C deleted the 2021-05-initial-usdt-support branch July 27, 2021 18:51
@@ -3017,11 +3018,20 @@ bool CConnman::NodeFullyConnected(const CNode* pnode)
void CConnman::PushMessage(CNode* pnode, CSerializedNetMsg&& msg)
{
size_t nMessageSize = msg.data.size();
LogPrint(BCLog::NET, "sending %s (%d bytes) peer=%d\n", SanitizeString(msg.m_type), nMessageSize, pnode->GetId());
LogPrint(BCLog::NET, "sending %s (%d bytes) peer=%d\n", msg.m_type, nMessageSize, pnode->GetId());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this refactor(?) mixed in a non-refactor commit without any explanation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have been good to split the refactor from the feature commit or at least mention it in the commit body. Otherwise reviewing locally is impossible and one needs to rely on GitHub, which showed in the past to be corrupting both code and metadata or be unreachable.

Copy link
Contributor Author

@0xB10C 0xB10C Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't consider that this needs a explanation in the commit message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have been good to split the refactor from the feature commit or at least mention it in the commit body. Otherwise reviewing locally is impossible and one needs to rely on GitHub, which showed in the past to be corrupting both code and metadata or be unreachable.

Yes, makes sense.

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Jul 28, 2021
…User-Space, Statically Defined Tracing (USDT)

8f37f5c tracing: Tracepoint for connected blocks (0xb10c)
4224dec tracing: Tracepoints for in- and outbound P2P msgs (0xb10c)
469b71a doc: document systemtap dependency (0xb10c)
84ace9a doc: Add initial USDT documentation (0xb10c)

Pull request description:

  This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).

  ## Context
  The `TRACEx` macros for tracepoints and build system changes for USDT were merged in bitcoin#19866 earlier this year. Issue bitcoin#20981 contains discussion about potential tracepoints and guidelines for adding them (also documented with this PR). USDT was a topic in a [core-dev-meeting discussion](https://bitcoin.jonasschnelli.ch/ircmeetings/logs/bitcoin-core-dev/2021/bitcoin-core-dev.2021-01-21-19.00.moin.txt) on 21st Jan, 2021.

  - [collabora.com: An eBPF overview, part 1: Introduction](https://www.collabora.com/news-and-blog/blog/2019/04/05/an-ebpf-overview-part-1-introduction/)
  - [collabora.com: An eBPF overview, part 2: Machine & bytecode](https://www.collabora.com/news-and-blog/blog/2019/04/15/an-ebpf-overview-part-2-machine-and-bytecode/)
  - [Brendan D. Gregg's blog posts, and book on on eBPF](http://www.brendangregg.com/)
  - [Brendan D. Gregg: Linux bcc/BPF Node.js USDT Tracing](http://www.brendangregg.com/blog/2016-10-12/linux-bcc-nodejs-usdt.html)

  ## USDT? Stablecoin?

  User-space, Statically Defined Tracing (USDT) allows for more observability during development, debugging, code review, and production usage. The tracepoints make it possible to keep track of custom statistics and enable detailed monitoring of otherwise hidden internals and have little to no performance impact when unused. Linux kernels (4.x or newer) can hook into the tracepoints and execute [eBPF] programs in a kernel VM once the tracepoint is called.

  This PR includes, for example, tracepoints for in- and outbound P2P messages.

  ```
  USDT and eBPF Overview
  ======================

                  ┌──────────────────┐            ┌──────────────┐
                  │ tracing script   │            │ bitcoind     │
                  │==================│      2.    │==============│
                  │  eBPF  │ tracing │      hooks │              │
                  │  code  │ logic   │      into┌─┤►tracepoint 1─┼───┐ 3.
                  └────┬───┴──▲──────┘          ├─┤►tracepoint 2 │   │ pass args
              1.       │      │ 4.              │ │ ...          │   │ to eBPF
      User    compiles │      │ pass data to    │ └──────────────┘   │ program
      Space    & loads │      │ tracing script  │                    │
      ─────────────────┼──────┼─────────────────┼────────────────────┼───
      Kernel           │      │                 │                    │
      Space       ┌──┬─▼──────┴─────────────────┴────────────┐       │
                  │  │  eBPF program                         │◄──────┘
                  │  └───────────────────────────────────────┤
                  │ eBPF kernel Virtual Machine (sandboxed)  │
                  └──────────────────────────────────────────┘

  1. The tracing script compiles the eBPF code and loads the eBFP program into a kernel VM
  2. The eBPF program hooks into one or more tracepoints
  3. When the tracepoint is called, the arguments are passed to the eBPF program
  4. The eBPF program processes the arguments and returns data to the tracing script
  ```

  The two main [eBPF] front-ends with support for USDT are [bpftrace] an [BPF Compiler Collection (BCC)]. BCC is used for complex tools and daemons and `bpftrace` is preferred for one-liners and shorter scripts. Example tracing scripts for both are provided with this PR.

  [eBPF]: https://ebpf.io/
  [bpftrace]: https://github.com/iovisor/bpftrace
  [BPF Compiler Collection (BCC)]: https://github.com/iovisor/bcc

  This PR adds three tracepoints:
  - `net:inbound_message`
  - `net:outbound_message`
  - `valildation:block_connected`

  See `doc/tracing.md` and `contrib/tracing/` for documentation and example tracing scripts.

  ## Open Questions (Not in scope for this PR)
  -  How to use these tracepoints under macOS?
  -  Release builds with USDT support?
  -  Should and can the tracepoints be automatically tested?

  ## Todo (before undraft)
  - [x] bcc example showing how to read raw P2P messages up to 32kb
  - [x] document that you need `sys/sdt.h` from `systemtap` for USDT support in Bitcoin Core (`apt install systemtap-sdt-dev` on debian-like). See bitcoin@933ab8a
  - [ ] release notes?

ACKs for top commit:
  laanwj:
    re-ACK 8f37f5c
  jb55:
    ACK 8f37f5c

Tree-SHA512: a92a8a2dfcd28465f58a6e5f50d39486817ef5f51214ec40bdb02a6843b9c08ea154fadb31558825ff3a4687477b90f2a5da5d6451989eef978e128a264c289d
0xB10C added a commit to 0xB10C/bitcoin that referenced this pull request Oct 18, 2021
This tracepoint was introduced in bitcoin#22006. The first argument was the
hash of the connected block as a pointer to a C-like String. The last
argument passed the hash of the connected block as pointer to 32
bytes. The hash was passed both as pointer to a string and to bytes
to allow `bpftrace` scripts to print the String. It was (incorrectly)
assumed that `bpftrace` is not able to hex-format and print the block
hash given only the hash bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash it can be printed in in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the connect_block tracepoint API, however
there exist no release including the tracepoint yet.
0xB10C added a commit to 0xB10C/bitcoin that referenced this pull request Oct 18, 2021
This tracepoint was introduced in bitcoin#22006. The first argument was the
hash of the connected block as a pointer to a C-like String. The last
argument passed the hash of the connected block as a pointer to 32
bytes. The hash was passed both as a pointer to a string and bytes
to allow `bpftrace` scripts to print the String. It was (incorrectly)
assumed that `bpftrace` cannot hex-format and print the block hash
given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the connect_block tracepoint API, however
there exist no release, including the tracepoint, yet.
0xB10C added a commit to 0xB10C/bitcoin that referenced this pull request Oct 18, 2021
The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was passed both as
a pointer to a string and bytes to allow `bpftrace` scripts to print
the String. It was (incorrectly) assumed that `bpftrace` cannot hex-
format and print the block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.
0xB10C added a commit to 0xB10C/bitcoin that referenced this pull request Oct 18, 2021
The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was only passed as
string to allow `bpftrace` scripts to print the hash. It was
(incorrectly) assumed that `bpftrace` cannot hex-format and print the
block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p-=1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.
0xB10C added a commit to 0xB10C/bitcoin that referenced this pull request Oct 18, 2021
The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
The first argument was the hash of the connected block as a pointer
to a C-like String. The last argument passed the hash of the
connected block as a pointer to 32 bytes. The hash was only passed as
string to allow `bpftrace` scripts to print the hash. It was
(incorrectly) assumed that `bpftrace` cannot hex-format and print the
block hash given only the hash as bytes.

The block hash can be printed in `bpftrace` by calling
`printf("%02x")` for each byte of the hash in an `unroll () {...}`.
By starting from the last byte of the hash, it can be printed in
big-endian (the block-explorer format).

```C
  $p = $hash + 31;
  unroll(32) {
      $b = *(uint8*)$p;
      printf("%02x", $b);
      $p -= 1;
  }
```

See also: bitcoin#22902 (comment)

This is a breaking change to the block_connected tracepoint API, however
this tracepoint has not yet been included in a release.
fanquake added a commit that referenced this pull request Oct 19, 2021
…alidation:block_connected` tracepoint

53c9fa9 tracing: drop block_connected hash.toString() arg (0xb10c)

Pull request description:

  The tracepoint `validation:block_connected` was introduced in #22006.
  The first argument was the hash of the connected block as a pointer
  to a C-like String. The last argument passed the hash of the
  connected block as a pointer to 32 bytes. The hash was only passed as
  string to allow `bpftrace` scripts to print the hash. It was
  (incorrectly) assumed that `bpftrace` cannot hex-format and print the
  block hash given only the hash as bytes.

  The block hash can be printed in `bpftrace` by calling
  `printf("%02x")` for each byte of the hash in an `unroll () {...}`.
  By starting from the last byte of the hash, it can be printed in
  big-endian (the block-explorer format).

  ```C
    $p = $hash + 31;
    unroll(32) {
        $b = *(uint8*)$p;
        printf("%02x", $b);
        $p -= 1;
    }
  ```

  See also: #22902 (comment)

  This is a breaking change to the block_connected tracepoint API, however
  this tracepoint has not yet been included in a release.

ACKs for top commit:
  laanwj:
    Concept and code review ACK 53c9fa9
  jb55:
    ACK 53c9fa9

Tree-SHA512: f1b9e4e0ee45aae892e8bf38e04b5ee5fbc643d6e7e27d011b829ed8701dacf966a99b7c877c46cca8666b894a375633e62582c552c8203614c6f2b9c4087585
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Oct 19, 2021
… the `validation:block_connected` tracepoint

53c9fa9 tracing: drop block_connected hash.toString() arg (0xb10c)

Pull request description:

  The tracepoint `validation:block_connected` was introduced in bitcoin#22006.
  The first argument was the hash of the connected block as a pointer
  to a C-like String. The last argument passed the hash of the
  connected block as a pointer to 32 bytes. The hash was only passed as
  string to allow `bpftrace` scripts to print the hash. It was
  (incorrectly) assumed that `bpftrace` cannot hex-format and print the
  block hash given only the hash as bytes.

  The block hash can be printed in `bpftrace` by calling
  `printf("%02x")` for each byte of the hash in an `unroll () {...}`.
  By starting from the last byte of the hash, it can be printed in
  big-endian (the block-explorer format).

  ```C
    $p = $hash + 31;
    unroll(32) {
        $b = *(uint8*)$p;
        printf("%02x", $b);
        $p -= 1;
    }
  ```

  See also: bitcoin#22902 (comment)

  This is a breaking change to the block_connected tracepoint API, however
  this tracepoint has not yet been included in a release.

ACKs for top commit:
  laanwj:
    Concept and code review ACK 53c9fa9
  jb55:
    ACK 53c9fa9

Tree-SHA512: f1b9e4e0ee45aae892e8bf38e04b5ee5fbc643d6e7e27d011b829ed8701dacf966a99b7c877c46cca8666b894a375633e62582c552c8203614c6f2b9c4087585
laanwj added a commit that referenced this pull request Nov 29, 2021
2bc51c5 [tracing] tracepoints to utxocache add, spent and uncache (Arnab Sen)
a26e8ee [tracing] tracepoint for utxocache flushes (Arnab Sen)

Pull request description:

  This PR adds some of the UTXO set cache tracepoints proposed in #20981 (comment). The first tracepoints were added in #22006.

  tracepoint | description
  -- | --
  `utxocache:flush` | Is called after the caches and indexes are flushed
  `utxocache:add` | when a new coin is added to the UTXO cache
  `utxocache:spent` | when a coin is spent
  `utxocache:uncache` | when coin is removed from the UTXO cache

  The tracepoints are further documented in `docs/tracing.md` and the usage is shown via the two newly added example scripts in `contrib/tracing/`.

ACKs for top commit:
  laanwj:
    Code and documentation review ACK 2bc51c5

Tree-SHA512: d6b4f435d3260de4c48b36956f9311f65ab3b52cd03b1e0a4ba9cf47a774d8c4b31878e222b11e0ba5d233a68f7567f8a367b12a6392f688c10c11529341e837
@rob-scheepens
Copy link

As BPF CO-RE arguably is the future direction for BPF, it's perhaps interesting to explore using libbpf/CO-RE instead of BCC. See https://nakryiko.com/posts/bpf-portability-and-co-re/#bpf-co-re-as-of-2021 and https://nakryiko.com/posts/bcc-to-libbpf-howto-guide/#bpf-code-conversion.

@jb55
Copy link
Contributor

jb55 commented Mar 18, 2022 via email

@0xB10C
Copy link
Contributor Author

0xB10C commented Mar 18, 2022

@rob-scheepens agree that BPF CO-RE and libbpf is the way to go for future tracing programs. I've looked into this a while ago, but it seems libbpf doesn't support USDT tracepoints yet (see iovisor/bcc#3007). However, there was a mention of a soon to be open sourced libusdt library developed and used in production at Meta by @anakryiko here. Looking forward to testing this once available!

@anakryiko
Copy link

I'm actually cleaning up the code a bit and preparing patches for submission to include proper USDT support in libbpf itself. So be on the lookout for upstream patches this week (or at worst next week, if I get distracted with other stuff this week).

laanwj added a commit to bitcoin-core/gui that referenced this pull request Apr 6, 2022
76c60d7 test: validation:block_connected tracepoint test (0xb10c)
260e28e test: utxocache:* tracepoint tests (0xb10c)
34b27ba test: net:in/out_message tracepoint tests (0xb10c)
c934087 test: checks for tracepoint tests (0xb10c)

Pull request description:

  This adds functional tests for the USDT tracepoints added in bitcoin/bitcoin#22006 and bitcoin/bitcoin#22902. This partially fixes #23296. The tests **are probably skipped** on most systems as these tests require:
  - a Linux system with a kernel that supports BPF (and available kernel headers)
  - that Bitcoin Core is compiled with tracepoints for USDT support (default when compiled with depends)
  - [bcc](https://github.com/iovisor/bcc) installed
  - the tests are run with a privileged user that is able to e.g. do BPF syscalls and load BPF maps

  The tests are not yet run in our CI as the CirrusCI containers lack the required permissions (see bitcoin/bitcoin#23296 (comment)). Running the tests in a VM in the CI could work, but I haven't experimented with this yet. The priority was to get the actual tests done first to ensure the tracepoints work as intended for the v23.0 release. Running the tracepoint tests in the CI is planned as the next step to finish #23296.

  The tests can, however, be run against e.g. release candidates by hand. Additionally, they provide a starting point for tests for future tracepoints. PRs adding new tracepoint should include tests. This makes reviewing these PRs easier.

  The tests require privileges to execute BPF sycalls (`CAP_SYS_ADMIN` before Linux kernel 5.8 and `CAP_BPF` and `CAP_PERFMON` on 5.8+) and permissions to `/sys/kernel/debug/tracing/`. It's currently recommended to run the tests in a virtual machine (or on a VPS) where it's sensible to use the `root` user to gain these privileges. Never run python scripts you haven't carefully reviewed with `root` permissions! It's unclear if a non-root user can even gain the required privileges. This needs more experimenting.

  The goal here is to test the tracepoint interface to make sure the [documented interface](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#tracepoint-documentation) does not break by accident. The tracepoints expose implementation details. This means we also need to rely on implementation details of Bitcoin Core in these functional tests to trigger the tracepoints. An example is the test of the `utxocache:flush` tracepoint: On Bitcoin Core shutdown, the UTXO cache is flushed twice. The corresponding tracepoint test expects two flushes, too - if not, the test fails. Changing implementation details could cause these tests to fail and the tracepoint API to break. However, we purposefully treat the tracepoints only as [**semi-stable**](https://github.com/bitcoin/bitcoin/blob/master/doc/tracing.md#semi-stable-api). The tracepoints should not block refactors or changes to other internals.

ACKs for top commit:
  jb55:
    tACK 76c60d7
  laanwj:
    Tested ACK 76c60d7

Tree-SHA512: 9a63d945c68102e59d751bd8d2805ddd7b37185408fa831d28a9cb6641b701961389b55f216c475df7d4771154e735625067ee957fc74f454ad7a7921255364c
@anakryiko
Copy link

I'm actually cleaning up the code a bit and preparing patches for submission to include proper USDT support in libbpf itself. So be on the lookout for upstream patches this week (or at worst next week, if I get distracted with other stuff this week).

BTW, USDT support was added into libbpf a while ago. See bpf_program__attach_usdt() API and SEC("usdt") examples in kernel selftests/bpf. libbpf-bootstrap also has USDT example (https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/usdt.c and https://github.com/libbpf/libbpf-bootstrap/blob/master/examples/c/usdt.bpf.c).

@fanquake
Copy link
Member

This change was part of 23.0, and had a release note, so removing "Needs release note".

Fabcien added a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 25, 2022
Summary:
```
Instead of writing ad-hoc logging everywhere (eg: #19509), we can take advantage of linux user static defined traces, aka. USDTs ( not the stablecoin sweat_smile )

The linux kernel can hook into these tracepoints at runtime, but otherwise they have little to no performance impact. Traces can pass data which can be printed externally via tools such as bpftrace. For example, here's one that prints incoming and outgoing network messages:
```

Backport of [[bitcoin/bitcoin#19866 | core#19866]].

```
This PR adds documentation for User-Space, Statically Defined Tracing (USDT) as well as three tracepoints (including documentation and usage examples).
```

Backport of [[bitcoin/bitcoin#22006 | core#22006]].

Test Plan:
  cmake -GNinja .. -DENABLE_TRACING=ON
  ninja
  ./src/bitcoind -daemon
  sudo bpftrace ../contrib/tracing/log_p2p_traffic.bt
  sudo ../contrib/tracing/p2p_monitor.py ./src/bitcoind

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Subscribers: PiRK

Differential Revision: https://reviews.bitcoinabc.org/D12623
gwillen added a commit to gwillen/elements that referenced this pull request Mar 28, 2023
@bitcoin bitcoin locked and limited conversation to collaborators Sep 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet