Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 393 lines (325 sloc) 16.601 kb
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
1 DTrace and Erlang/OTP
2 =====================
3
4 History
5 -------
6
7 The first implementation of DTrace probes for the Erlang virtual
8 machine was presented at the [2008 Erlang User Conference] [4]. That
9 work, based on the Erlang/OTP R12 release, was discontinued due to
10 what appears to be miscommunication with the original developers.
11
12 Several users have created Erlang port drivers, linked-in drivers, or
13 NIFs that allow Erlang code to try to activate a probe,
14 e.g. `foo_module:dtrace_probe("message goes here!")`.
15
16 Goals
17 -----
18
19 1. Annotate as much of the Erlang VM as is practical.
20 * The initial goal is to trace file I/O operations.
21 2. Support all platforms that implement DTrace: OS X, Solaris,
22 and (I hope) FreeBSD and NetBSD.
23 3. To the extent that it's practical, support SystemTap on Linux
24 via DTrace provider compatibility.
25 4. Allow Erlang code to supply annotations.
26
27 Supported platforms
28 -------------------
29
0fd4e39a » psyeugenic
2012-02-07 Update dtrace for changes in R15
30 * OS X 10.6.x / Snow Leopard. It should also work for 10.7 / Lion,
31 but I haven't personally tested it.
32 * Solaris 10. I have done limited testing on Solaris 11 and
33 OpenIndiana release 151a, and both appear to work.
34 * FreeBSD 9.0, though please see the "FreeBSD 9.0 Release Notes"
35 section below!
36 * Linux via SystemTap compatibility. Please see the file
37 `README.systemtap.md` for more details.
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
38
0c0d640e » bufflig
2012-03-21 Update README's for dtrace and systemtap
39 Just add the `--with-dynamic-trace=dtrace` option to your command when you
40 run the `configure` script. If you are using systemtap, the configure option
41 is `--with-dynamic-trace=systemtap`
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
42
0c0d640e » bufflig
2012-03-21 Update README's for dtrace and systemtap
43 Status
44 ------
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
45
0c0d640e » bufflig
2012-03-21 Update README's for dtrace and systemtap
46 As of R15B01, the dynamic trace code is included in the main OTP distribution,
47 although it's considered experimental. The main development of the dtrace code
48 still happens outside of Ericsson, but there is no need to fetch a patched
49 version of OTP to get the basic funtionality.
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
50
51 Implementation summary
52 ----------------------
53
0c0d640e » bufflig
2012-03-21 Update README's for dtrace and systemtap
54 So far, most effort has been focused on the `efile_drv.c` code,
0721ac40 » slfritchie
2011-11-17 Add DTrace support for OS X, Solaris, and Linux (via SystemTap), 1/4
55 which implements most file I/O on behalf of the Erlang virtual
56 machine. This driver also presents a big challenge: its use of an I/O
57 worker pool (enabled by using the `erl +A 8` flag, for example) makes
58 it much more difficult to trace I/O activity because each of the
59 following may be executed in a different Pthread:
60
61 * I/O initiation (Erlang code)
62 * I/O proxy process handling, e.g. read/write when file is not opened
63 in `raw` mode, operations executed by the code & file server processes.
64 (Erlang code)
65 * `efile_drv` command setup (C code)
66 * `efile_drv` command execution (C code)
67 * `efile_drv` status return (C code)
68
69 **TODO: keep this description up-to-date.**
70
71 Example output from `lib/dtrace/examples/efile_drv.d` while executing
72 `file:rename("old-name", "new-name")`:
73
74 efile_drv enter tag={3,84} user tag some-user-tag | RENAME (12) | args: old-name new-name , 0 0 (port #Port<0.59>)
75 async I/O worker tag={3,83} | RENAME (12) | efile_drv-int_entry
76 async I/O worker tag={3,83} | RENAME (12) | efile_drv-int_return
77 efile_drv return tag={3,83} user tag | RENAME (12) | errno 2
78
79 ... where the following key can help decipher the output:
80
81 * `{3,83}` is the Erlang scheduler thread number (3) and operation
82 counter number (83) assigned to this I/O operation. Together,
83 these two numbers form a unique ID for the I/O operation.
84 * `12` is the command number for the rename operation. See the
85 definition for `FILE_RENAME` in the source code file `efile_drv.c`
86 or the `BEGIN` section of the D script `lib/dtrace/examples/efile_drv.d`.
87 * `old-name` and `new-name` are the two string arguments for the
88 source and destination of the `rename(2)` system call.
89 The two integer arguments are unused; the simple formatting code
90 prints the arguments anyway, 0 and 0.
91 * The worker pool code was called on behalf of Erlang port `#Port<0.59>`.
92 * The system call failed with a POSIX errno value of 2: `ENOENT`,
93 because the path `old-name` does not exist.
94 * The `efile_drv-int_entry` and `efile_drv_int_return` probes are
95 provided in case the user is
96 interested in measuring only the latency of code executed by
97 `efile_drv` asynchronous functions by I/O worker pool threads
98 and the OS system call that they encapsulate.
99
100 So, where does the `some-user-tag` string come from?
101
102 At the moment, the user tag comes from code like the following:
103
104 put(dtrace_utag, "some-user-tag"),
105 file:rename("old-name", "new-name").
106
107 This method of tagging I/O at the Erlang level is subject to change.
108
109 Example DTrace probe specification
110 ----------------------------------
111
112 /**
113 * Fired when a message is sent from one local process to another.
114 *
115 * NOTE: The 'size' parameter is in machine-dependent words and
116 * that the actual size of any binary terms in the message
117 * are not included.
118 *
119 * @param sender the PID (string form) of the sender
120 * @param receiver the PID (string form) of the receiver
121 * @param size the size of the message being delivered (words)
122 * @param token_label for the sender's sequential trace token
123 * @param token_previous count for the sender's sequential trace token
124 * @param token_current count for the sender's sequential trace token
125 */
126 probe message__send(char *sender, char *receiver, uint32_t size,
127 int token_label, int token_previous, int token_current);
128
129 /**
130 * Fired when a message is sent from a local process to a remote process.
131 *
132 * NOTE: The 'size' parameter is in machine-dependent words and
133 * that the actual size of any binary terms in the message
134 * are not included.
135 *
136 * @param sender the PID (string form) of the sender
137 * @param node_name the Erlang node name (string form) of the receiver
138 * @param receiver the PID/name (string form) of the receiver
139 * @param size the size of the message being delivered (words)
140 * @param token_label for the sender's sequential trace token
141 * @param token_previous count for the sender's sequential trace token
142 * @param token_current count for the sender's sequential trace token
143 */
144 probe message__send__remote(char *sender, char *node_name, char *receiver,
145 uint32_t size,
146 int token_label, int token_previous, int token_current);
147
148 /**
149 * Fired when a message is queued to a local process. This probe
150 * will not fire if the sender's pid == receiver's pid.
151 *
152 * NOTE: The 'size' parameter is in machine-dependent words and
153 * that the actual size of any binary terms in the message
154 * are not included.
155 *
156 * @param receiver the PID (string form) of the receiver
157 * @param size the size of the message being delivered (words)
158 * @param queue_len length of the queue of the receiving process
159 * @param token_label for the sender's sequential trace token
160 * @param token_previous count for the sender's sequential trace token
161 * @param token_current count for the sender's sequential trace token
162 */
163 probe message__queued(char *receiver, uint32_t size, uint32_t queue_len,
164 int token_label, int token_previous, int token_current);
165
166 /**
167 * Fired when a message is 'receive'd by a local process and removed
168 * from its mailbox.
169 *
170 * NOTE: The 'size' parameter is in machine-dependent words and
171 * that the actual size of any binary terms in the message
172 * are not included.
173 *
174 * @param receiver the PID (string form) of the receiver
175 * @param size the size of the message being delivered (words)
176 * @param queue_len length of the queue of the receiving process
177 * @param token_label for the sender's sequential trace token
178 * @param token_previous count for the sender's sequential trace token
179 * @param token_current count for the sender's sequential trace token
180 */
181 probe message__receive(char *receiver, uint32_t size, uint32_t queue_len,
182 int token_label, int token_previous, int token_current);
183
184 /* ... */
185
186 /* Async driver pool */
187
188 /**
189 * Show the post-add length of the async driver thread pool member's queue.
190 *
191 * NOTE: The port name is not available: additional lock(s) must
192 * be acquired in order to get the port name safely in an SMP
193 * environment. The same is true for the aio__pool_get probe.
194 *
195 * @param port the Port (string form)
196 * @param new queue length
197 */
198 probe aio_pool__add(char *, int);
199
200 /**
201 * Show the post-get length of the async driver thread pool member's queue.
202 *
203 * @param port the Port (string form)
204 * @param new queue length
205 */
206 probe aio_pool__get(char *, int);
207
208 /* Probes for efile_drv.c */
209
210 /**
211 * Entry into the efile_drv.c file I/O driver
212 *
213 * For a list of command numbers used by this driver, see the section
214 * "Guide to probe arguments" in ../../../README.md. That section
215 * also contains explanation of the various integer and string
216 * arguments that may be present when any particular probe fires.
217 *
218 * TODO: Adding the port string, args[10], is a pain. Making that
219 * port string available to all the other efile_drv.c probes
220 * will be more pain. Is the pain worth it? If yes, then
221 * add them everywhere else and grit our teeth. If no, then
222 * rip it out.
223 *
224 * @param thread-id number of the scheduler Pthread arg0
225 * @param tag number: {thread-id, tag} uniquely names a driver operation
226 * @param user-tag string arg2
227 * @param command number arg3
228 * @param string argument 1 arg4
229 * @param string argument 2 arg5
230 * @param integer argument 1 arg6
231 * @param integer argument 2 arg7
232 * @param integer argument 3 arg8
233 * @param integer argument 4 arg9
234 * @param port the port ID of the busy port args[10]
235 */
236 probe efile_drv__entry(int, int, char *, int, char *, char *,
237 int64_t, int64_t, int64_t, int64_t, char *);
238
239 /**
240 * Entry into the driver's internal work function. Computation here
241 * is performed by a async worker pool Pthread.
242 *
243 * @param thread-id number
244 * @param tag number
245 * @param command number
246 */
247 probe efile_drv__int_entry(int, int, int);
248
249 /**
250 * Return from the driver's internal work function.
251 *
252 * @param thread-id number
253 * @param tag number
254 * @param command number
255 */
256 probe efile_drv__int_return(int, int, int);
257
258 /**
259 * Return from the efile_drv.c file I/O driver
260 *
261 * @param thread-id number arg0
262 * @param tag number arg1
263 * @param user-tag string arg2
264 * @param command number arg3
265 * @param Success? 1 is success, 0 is failure arg4
266 * @param If failure, the errno of the error. arg5
267 */
268 probe efile_drv__return(int, int, char *, int, int, int);
269
270 Guide to efile_drv.c probe arguments
271 ------------------------------------
272
273 /* Driver op code: used by efile_drv-entry arg3 */
274 /* used by efile_drv-int_entry arg3 */
275 /* used by efile_drv-int_return arg3 */
276 /* used by efile_drv-return arg3 */
277
278 #define FILE_OPEN 1 (probe arg3)
279 probe arg6 = C driver dt_i1 = flags;
280 probe arg4 = C driver dt_s1 = path;
281
282 #define FILE_READ 2 (probe arg3)
283 probe arg6 = C driver dt_i1 = fd;
284 probe arg7 = C driver dt_i2 = flags;
285 probe arg8 = C driver dt_i3 = size;
286
287 #define FILE_LSEEK 3 (probe arg3)
288 probe arg6 = C driver dt_i1 = fd;
289 probe arg7 = C driver dt_i2 = offset;
290 probe arg8 = C driver dt_i3 = origin;
291
292 #define FILE_WRITE 4 (probe arg3)
293 probe arg6 = C driver dt_i1 = fd;
294 probe arg7 = C driver dt_i2 = flags;
295 probe arg8 = C driver dt_i3 = size;
296
297 #define FILE_FSTAT 5 (probe arg3)
298 probe arg6 = C driver dt_i1 = fd;
299
300 #define FILE_PWD 6 (probe arg3)
301 none
302
303 #define FILE_READDIR 7 (probe arg3)
304 probe arg4 = C driver dt_s1 = path;
305
306 #define FILE_CHDIR 8 (probe arg3)
307 probe arg4 = C driver dt_s1 = path;
308
309 #define FILE_FSYNC 9 (probe arg3)
310 probe arg6 = C driver dt_i1 = fd;
311
312 #define FILE_MKDIR 10 (probe arg3)
313 probe arg4 = C driver dt_s1 = path;
314
315 #define FILE_DELETE 11 (probe arg3)
316 probe arg4 = C driver dt_s1 = path;
317
318 #define FILE_RENAME 12 (probe arg3)
319 probe arg4 = C driver dt_s1 = old_name;
320 probe arg5 = C driver dt_s2 = new_name;
321
322 #define FILE_RMDIR 13 (probe arg3)
323 probe arg4 = C driver dt_s1 = path;
324
325 #define FILE_TRUNCATE 14 (probe arg3)
326 probe arg6 = C driver dt_i1 = fd;
327 probe arg7 = C driver dt_i2 = flags;
328
329 #define FILE_READ_FILE 15 (probe arg3)
330 probe arg4 = C driver dt_s1 = path;
331
332 #define FILE_WRITE_INFO 16 (probe arg3)
333 probe arg6 = C driver dt_i1 = mode;
334 probe arg7 = C driver dt_i2 = uid;
335 probe arg8 = C driver dt_i3 = gid;
336
337 #define FILE_LSTAT 19 (probe arg3)
338 probe arg4 = C driver dt_s1 = path;
339
340 #define FILE_READLINK 20 (probe arg3)
341 probe arg4 = C driver dt_s1 = path;
342
343 #define FILE_LINK 21 (probe arg3)
344 probe arg4 = C driver dt_s1 = existing_path;
345 probe arg5 = C driver dt_s2 = new_path;
346
347 #define FILE_SYMLINK 22 (probe arg3)
348 probe arg4 = C driver dt_s1 = existing_path;
349 probe arg5 = C driver dt_s2 = new_path;
350
351 #define FILE_CLOSE 23 (probe arg3)
352 probe arg6 = C driver dt_i1 = fd;
353 probe arg7 = C driver dt_i2 = flags;
354
355 #define FILE_PWRITEV 24 (probe arg3)
356 probe arg6 = C driver dt_i1 = fd;
357 probe arg7 = C driver dt_i2 = flags;
358 probe arg8 = C driver dt_i3 = size;
359
360 #define FILE_PREADV 25 (probe arg3)
361 probe arg6 = C driver dt_i1 = fd;
362 probe arg7 = C driver dt_i2 = flags;
363 probe arg8 = C driver dt_i3 = size;
364
365 #define FILE_SETOPT 26 (probe arg3)
366 probe arg6 = C driver dt_i1 = opt_name;
367 probe arg7 = C driver dt_i2 = opt_specific_value;
368
369 #define FILE_IPREAD 27 (probe arg3)
370 probe arg6 = C driver dt_i1 = fd;
371 probe arg7 = C driver dt_i2 = flags;
372 probe arg8 = C driver dt_i3 = offsets[0];
373 probe arg9 = C driver dt_i4 = size;
374
375 #define FILE_ALTNAME 28 (probe arg3)
376 probe arg4 = C driver dt_s1 = path;
377
378 #define FILE_READ_LINE 29 (probe arg3)
379 probe arg6 = C driver dt_i1 = fd;
380 probe arg7 = C driver dt_i2 = flags;
381 probe arg8 = C driver dt_i3 = read_offset;
382 probe arg9 = C driver dt_i4 = read_ahead;
383
384 #define FILE_FDATASYNC 30 (probe arg3)
385 probe arg6 = C driver dt_i1 = fd;
386
387 #define FILE_FADVISE 31 (probe arg3)
388 probe arg6 = C driver dt_i1 = fd;
389 probe arg7 = C driver dt_i2 = offset;
390 probe arg8 = C driver dt_i3 = length;
391 probe arg9 = C driver dt_i4 = advise_type;
392
393 [1]: http://www.erlang.org/euc/08/
Something went wrong with that request. Please try again.