-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
librbd: Zipkin tracing [GSOC 2016] #10637
Closed
Closed
Changes from 27 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
5b5d443
Build support for BlkKin (LTTng + Zipkin) tracing
agshew df3fdab
cmake: add module and WITH_BLKIN option for blkin
cbodley 9568720
blkin: add header and initialization
cbodley 589dafe
msg: pass Connection to decode_message
cbodley 1fd89fa
blkin: Messenger integration
cbodley fb282d2
blkin: add traces to SimpleMessenger
cbodley 67287b8
blkin: add traces to XioMessenger
cbodley 1bc46f2
blkin: add traces to AsyncMessenger
cbodley b78200f
blkin: osd op messages carry trace information
cbodley b1143c7
blkin: set up tracing in the OSD
cbodley a87324b
blkin: set up tracing in PGs
cbodley 1bcb22a
blkin: add traces to ReplicatedBackend
cbodley b6392ab
blkin: add traces to ECBackend
cbodley 6ff1880
blkin: add traces to FileStore/Journal
cbodley 8c612fc
blkin: add config variable osd_blkin_trace_all
cbodley c1094ab
blkin: add traces to librados and Objecter
cbodley dda790f
blkin: add config variable osdc_blkin_trace_all
cbodley a1bc03c
cmake: Add blkin libs to target_link_libraries
vh4x 17509bd
Pass trace information for aio_writes within librbd.
vh4x bfc1367
Incorporate librados C changes from commit f9aea0cd33f071409c5653e0ac…
vh4x e62b40e
Modify aio_operate to take trace information.
vh4x 08b08be
Pass trace information from rbd_aio_read_traced to aio_operate_read()
vh4x 90ecc6a
librbd: Rebase template changes in librbd.
vear91 9de5eed
librbd: Pass trace information in AioObjectRequests
vear91 b7d46ac
Add the blkin tracing library as a submodule.
vear91 3eefd8c
Add tests for rbd_aio_write_traced and rbd_aio_write_traced in test_l…
vear91 d36519d
Set CEPH_FEATURE_BLKIN_TRACING to unused bit 59.
vear91 1da3bcf
Overload aio_operate instead of using an optional parameter for trace…
vear91 efa54c4
osd: encode trace as part of the partial chunk of MOSDOp
cbodley 2451abf
librados: Record events for zipkin root span initialization
vear91 c2f6bdb
librados: Remove the blkin_trace_info parameters in LIbradosTestStub.cc
vear91 9b75fe4
rbd: Call the aio_operate overload when there is trace information.
vear91 1a87bbf
rbd: Avoid storing blkin_trace_info*
vear91 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
# - Try to find blkin | ||
# Once done this will define | ||
# BLKIN_FOUND - System has blkin | ||
# BLKIN_INCLUDE_DIR - The blkin include directories | ||
# BLKIN_LIBRARIES - The libraries needed to use blkin | ||
|
||
find_package(PkgConfig) | ||
pkg_check_modules(PC_BLKIN QUIET libblkin) | ||
|
||
find_path(BLKIN_INCLUDE_DIR ztracer.hpp | ||
HINTS ${PC_BLKIN_INCLUDEDIR} ${PC_BLKIN_INCLUDE_DIRS} | ||
PATH_SUFFIXES blkin) | ||
find_library(BLKIN_LIBRARY NAMES blkin | ||
HINTS ${PC_BLKIN_LIBDIR} ${PC_BLKIN_LIBRARY_DIRS}) | ||
|
||
include(FindPackageHandleStandardArgs) | ||
# handle the QUIETLY and REQUIRED arguments and set BLKIN_FOUND to TRUE | ||
# if all listed variables are TRUE | ||
find_package_handle_standard_args(blkin DEFAULT_MSG | ||
BLKIN_LIBRARY BLKIN_INCLUDE_DIR) | ||
|
||
set(BLKIN_LIBRARIES ${BLKIN_LIBRARY} lttng-ust) | ||
mark_as_advanced(BLKIN_INCLUDE_DIR BLKIN_LIBRARIES) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,167 @@ | ||
========================= | ||
Tracing Ceph With BlkKin | ||
========================= | ||
|
||
Ceph can use Blkin, a library created by Marios Kogias and others, | ||
which enables tracking a specific request from the time it enters | ||
the system at higher levels till it is finally served by RADOS. | ||
|
||
In general, Blkin implements the Dapper_ tracing semantics | ||
in order to show the causal relationships between the different | ||
processing phases that an IO request may trigger. The goal is an | ||
end-to-end visualisation of the request's route in the system, | ||
accompanied by information concerning latencies in each processing | ||
phase. Thanks to LTTng this can happen with a minimal overhead and | ||
in realtime. The LTTng traces can then be visualized with Twitter's | ||
Zipkin_. | ||
|
||
.. _Dapper: http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf | ||
.. _Zipkin: http://twitter.github.io/zipkin/ | ||
|
||
|
||
Installing Blkin | ||
================ | ||
|
||
You can install Markos Kogias' upstream Blkin_ by hand.:: | ||
|
||
cd blkin/ | ||
make && make install | ||
|
||
or build distribution packages using DistroReadyBlkin_, which also comes with | ||
pkgconfig support. If you choose the latter, then you must generate the | ||
configure and make files first.:: | ||
|
||
cd blkin | ||
autoreconf -i | ||
|
||
.. _Blkin: https://github.com/marioskogias/blkin | ||
.. _DistroReadyBlkin: https://github.com/agshew/blkin | ||
|
||
|
||
Configuring Ceph with Blkin | ||
=========================== | ||
|
||
If you built and installed Blkin by hand, rather than building and | ||
installing packages, then set these variables before configuring | ||
Ceph.:: | ||
|
||
export BLKIN_CFLAGS=-Iblkin/ | ||
export BLKIN_LIBS=-lzipkin-cpp | ||
|
||
Since there are separate lttng and blkin changes to Ceph, you may | ||
want to configure with something like:: | ||
|
||
./configure --with-blkin --without-lttng --with-debug | ||
|
||
|
||
Testing Blkin | ||
============= | ||
|
||
It's easy to test Ceph's Blkin tracing. Let's assume you don't have | ||
Ceph already running, and you compiled Ceph with Blkin support but | ||
you did't install it. Then launch Ceph with the ``vstart.sh`` script | ||
in Ceph's src directgory so you can see the possible tracepoints.:: | ||
|
||
cd src | ||
OSD=3 MON=3 RGW=1 ./vstart.sh -n | ||
lttng list --userspace | ||
|
||
You'll see something like the following::: | ||
|
||
UST events: | ||
------------- | ||
PID: 8987 - Name: ./ceph-osd | ||
zipkin:timestamp (loglevel: TRACE_WARNING (4)) (type: tracepoint) | ||
zipkin:keyval (loglevel: TRACE_WARNING (4)) (type: tracepoint) | ||
ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) | ||
|
||
PID: 8407 - Name: ./ceph-mon | ||
zipkin:timestamp (loglevel: TRACE_WARNING (4)) (type: tracepoint) | ||
zipkin:keyval (loglevel: TRACE_WARNING (4)) (type: tracepoint) | ||
ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint) | ||
|
||
... | ||
|
||
Next, stop Ceph so that the tracepoints can be enabled.:: | ||
|
||
./stop.sh | ||
|
||
Start up an LTTng session and enable the tracepoints.:: | ||
|
||
lttng create blkin-test | ||
lttng enable-event --userspace zipkin:timestamp | ||
lttng enable-event --userspace zipkin:keyval | ||
lttng start | ||
|
||
Then start up Ceph again.:: | ||
|
||
OSD=3 MON=3 RGW=1 ./vstart.sh -n | ||
|
||
You may want to check that ceph is up.:: | ||
|
||
./ceph status | ||
|
||
Now put something in usin rados, check that it made it, get it back, and remove it.:: | ||
|
||
./rados mkpool test-blkin | ||
./rados put test-object-1 ./vstart.sh --pool=test-blkin | ||
./rados -p test-blkin ls | ||
./ceph osd map test-blkin test-object-1 | ||
./rados get test-object-1 ./vstart-copy.sh --pool=test-blkin | ||
md5sum vstart* | ||
./rados rm test-object-1 --pool=test-blkin | ||
|
||
You could also use the example in ``examples/librados/`` or ``rados bench``. | ||
|
||
Then stop the LTTng session and see what was collected.:: | ||
|
||
lttng stop | ||
lttng view | ||
|
||
You'll see something like::: | ||
|
||
[13:09:07.755054973] (+?.?????????) scruffy zipkin:timestamp: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, event = "Message allocated" } | ||
[13:09:07.755071569] (+0.000016596) scruffy zipkin:keyval: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, key = "Type", val = "MOSDOp" } | ||
[13:09:07.755074217] (+0.000002648) scruffy zipkin:keyval: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, key = "Reqid", val = "client.4126.0:1" } | ||
... | ||
|
||
|
||
Install Zipkin | ||
=============== | ||
One of the points of using Blkin is so that you can look at the traces | ||
using Zipkin. Users should run Zipkin as a tracepoints collector and | ||
also a web service, which means users need to run three services, | ||
zipkin-collector, zipkin-query and zipkin-web. | ||
|
||
Download Zipkin Package:: | ||
|
||
wget https://github.com/twitter/zipkin/archive/1.1.0.tar.gz | ||
tar zxf 1.1.0.tar.gz | ||
cd zipkin-1.1.0 | ||
bin/collector cassandra & | ||
bin/query cassandra & | ||
bin/web & | ||
|
||
Check Zipkin:: | ||
|
||
bin/test | ||
Browse http://${zipkin-web-ip}:8080 | ||
|
||
|
||
Show Ceph's Blkin Traces in Zipkin-web | ||
====================================== | ||
Blkin provides a script which translates lttng result to Zipkin | ||
(Dapper) semantics. | ||
|
||
Send lttng data to Zipkin:: | ||
|
||
python3 babeltrace_zipkin.py ${lttng-traces-dir}/${blkin-test}/ust/uid/0/64-bit/ -p ${zipkin-collector-port(9410 by default)} -s ${zipkin-collector-ip} | ||
|
||
Example:: | ||
|
||
python3 babeltrace_zipkin.py ~/lttng-traces-dir/blkin-test-20150225-160222/ust/uid/0/64-bit/ -p 9410 -s 127.0.0.1 | ||
|
||
Check Ceph traces on webpage:: | ||
|
||
Browse http://${zipkin-web-ip}:8080 | ||
Click "Find traces" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we move this under the ceph github org? I can create the repo there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/ceph/blkin and https://github.com/ceph/babeltrace-zipkin are available now