Examples and hands-on labs for Linux tracing tools workshops
Switch branches/tags
Nothing to show
Clone or download
Permalink
Failed to load latest commit information.
Gatos Makefile: don't echo the echo Sep 7, 2017
buggy Add Node.js MySQL demo and generalize mysqlsniff script Feb 12, 2017
dotnet Add .NET Core demo project Jun 7, 2017
nodey Pass through additional arguments in run.sh Jul 24, 2017
perros Fix typos in Makefile steps Mar 8, 2018
slowy Added slowy Java app Jun 16, 2016
whales Add demos of container tracing Nov 20, 2017
.gitignore Add www script for Node app startup Feb 12, 2017
LICENSE Update year on readme and license Jan 29, 2017
README.md node-slowdown: Slowdown lab caused by bad hasher Jul 20, 2017
a.mat Add matexp program for demonstrating perf on a C++ application Jan 29, 2017
blocky.c Add on- and off-CPU investigation lab Jan 31, 2017
bpf-contention.md Fix prompt symbol Jan 31, 2017
bpf-cpu.md doco - fix for’ loop initial declarations are only allowed in C99 mode ( Jul 14, 2018
bpf-dbslower.md Fix typos Oct 16, 2016
bpf-files.md Minor fixes to files I/O lab following PR Jan 17, 2017
bpf-io.md Update instructions to work on FC/Ubuntu, Strigo/custom setup Oct 13, 2016
bpf-issues.md Miscellaneous fixes across labs Jan 17, 2017
bpf-memleak.md Add command for piping memleak through c++filt Feb 23, 2017
bpf-nodeblocked.md bpf-nodeblocked: Add mention of --trace-sync-io Jul 21, 2017
bpf-nodegc.md bpf-nodegc: Fixed link to gc-profiler module Jul 21, 2017
bpf-nodemysql.md Add link to mysqlsniff.py Jul 14, 2017
bpf-nodeopens.md bpf-nodeopens: Add exercise for syscall tracing and failed open()s Jul 13, 2017
bpf-oneliners.md update postgres INSERT to use single quotes (#2) Jun 21, 2017
bpf-opens.md bpf-opens: Snoop syscalls using BCC syscount instead of perf Jul 13, 2017
bpf-setuidsnoop.md Fix formatting Oct 16, 2016
bpf-usdt.md Fix tplist command to use -vv Nov 12, 2016
create-db.sql Add Node.js MySQL demo and generalize mysqlsniff script Feb 12, 2017
create-user.sql Added IF EXISTS to drop user Mar 8, 2017
data_access.py Update instructions to work on FC/Ubuntu, Strigo/custom setup Oct 13, 2016
dbslower.py Add dbslower lab solution Oct 16, 2016
dbstat.py dbstat: MySQL/PostgreSQL query latency histogram tool Feb 8, 2017
ftrace.md ftrace lab done Jun 13, 2016
heapsnapshot1.png node-memleak: Memory leak debugging with heap snapshots Jul 16, 2017
heapsnapshot2.png node-memleak: Memory leak debugging with heap snapshots Jul 16, 2017
lockstat-solution.py Update lockstat to use BPF symbol resolution (#3) Jun 29, 2017
lockstat.py Update lockstat to use BPF symbol resolution (#3) Jun 29, 2017
logger.c Move logger writes to separate function for easier tracing Feb 2, 2017
matexp.cc Add matexp program for demonstrating perf on a C++ application Jan 29, 2017
mysql-db.sh java: Add MySQL query tracing lab for slow queries Sep 18, 2017
mysqlsniff.py Add Node.js MySQL demo and generalize mysqlsniff script Feb 12, 2017
netsend.py Modify netsend to use latest TP_DATA_LOC_READ_CONST macro Feb 1, 2017
nhttpslower.py node-slowhttp: Diagnosing slow outgoing HTTP requests with nhttpslower Jul 16, 2017
node-coredump.md node-coredump: Mention the core-dump module Jul 16, 2017
node-memleak.md node-memleak: Clarify command used to install heapdump module Jul 26, 2017
node-prof.md Add Node profiling exercise with perf and `--perf_basic_prof` Jul 13, 2017
node-slowdns.md node-slowdns: Diagnosing slow DNS query resolution for HTTP requests Jul 16, 2017
node-slowdown.md node-slowdown: Slowdown lab caused by bad hasher Jul 20, 2017
node-slowhttp.md node-slowhttp: Diagnosing slow outgoing HTTP requests with nhttpslower Jul 16, 2017
pargrep.cc Add pargrep, a buggy parallel grep for files Jun 8, 2017
parprimes.c Added parprimes.c for BPF contention lab Jun 20, 2016
perf-io.md Give the Java .svg a different name (#5) Sep 3, 2017
perf-opens.md Add qualification about syscount from BCC vs. perf-tools Feb 22, 2017
perf.md Update perf.md (#10) Jan 24, 2018
pg-slow.sql Add BPF one-liners lab with trace and argdist Jan 31, 2017
primes.c Fix prime calculation to account for even numbers (#4) Jun 30, 2017
server.c Add opensnoop lab with server app that fails to start Oct 15, 2016
server.js Added server.js for Node lab Jun 14, 2016
setuidsnoop.py Add solution for setuidsnoop Oct 16, 2016
setup-fedora.sh setup: Clone and build async-profiler Sep 18, 2017
wordcount.cc Added memory leak lab with BPF `memleak` Jun 16, 2016

README.md

Linux Tracing Workshops Materials

This repository contains examples and hands-on labs for various Linux tracing workshops, focusing on modern tracing tools. To perform these labs, you will need a Linux box that meets several prerequisites. You can also use an EC2 instance supplied by the instructor.


Prerequisites

  1. Linux 4.6+ (the distribution doesn't matter; tested on Ubuntu and Fedora Core)
  2. perf
  3. perf-map-agent
  4. FlameGraph
  5. perf-tools
  6. bcc
  7. OpenJDK
  8. The systemtap-sdt-dev package on Ubuntu or the systemtap-sdt-devel package on Fedora/RHEL
  9. Node.js built from source with configure --with-dtrace
  10. MySQL or MariaDB built from source with -DENABLE_DTRACE=1
  11. PostgreSQL built from source with configure --enable-dtrace
  12. MySQL Python Connector

If you're using Fedora 24+, try the experimental setup script, which can fetch you a recent kernel from mainline and install all the packages and tools you need for this workshop. Use at your own risk!


Strigo Virtual Environment

When this workshop is delivered as instructor-led training, the instructor will provision a Strigo virtual classroom (EC2 instances) for each student. To use the Strigo virtual environment:

  1. Log in to Strigo using the link provided by the instructor (you can log in with Google or create a new account, no verification required)
  2. Enter the classroom token (four characters) provided by the instructor to join the classroom
  3. Navigate to the Lab tab (fourth from the top, the icon that looks like a test tube) to get your EC2 instance started

Labs

  1. Probing Tracepoints with ftrace
  2. CPU Sampling with perf and Flame Graphs
  3. Using BPF Tools: Broken File Opens
  4. Using BPF Tools: Slow File I/O
  5. Using BPF Tools: Chasing a Memory Leak
  6. Using BPF Tools: Database and Disk Stats and Stacks
  7. Using BPF Tools: Node and JVM USDT Probes
  8. Writing BPF Tools: setuidsnoop
  9. Writing BPF Tools: dbslower
  10. Writing BPF Tools: Contention Stats and Stacks
  11. Writing BPF Tools: From BCC GitHub Issues
  12. Using perf Tools: Broken File Opens
  13. Using BPF Tools: trace and argdist One-Liners
  14. Using BPF Tools: CPU and Off-CPU Investigation
  15. Using perf Tools: Slow File I/O
  16. Using BPF Tools: Node Blocked Time Analysis
  17. Using BPF Tools: Node Garbage Collections
  18. Using BPF Tools: Node File Opens
  19. Using BPF Tools: Node Slow MySQL Queries
  20. Node Profiling with V8
  21. Node Core Dump Analysis with llnode
  22. Node Memory Leak Analysis
  23. Node Slow HTTP Requests
  24. Node Slow DNS
  25. Node Leaky Slowdown

(C) Sasha Goldshtein, 2015-2017. All rights reserved.