This repository has been archived by the owner on Mar 19, 2021. It is now read-only.
/
debug.tex
233 lines (184 loc) · 9.31 KB
/
debug.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
%%
%%
\chapter{What To Do When Bareos Crashes (Kaboom)}
\label{KaboomChapter}
\index[general]{Kaboom!What To Do When Bareos Crashes }
\index[general]{What To Do When Bareos Crashes (Kaboom) }
If you are running on a Linux system, and you have a set of working
configuration files, it is very unlikely that {\bf Bareos} will crash. As with
all software, however, it is inevitable that someday, it may crash,
particularly if you are running on another operating system or using a new or
unusual feature.
This chapter explains what you should do if one of the three {\bf Bareos}
daemons (Director, File, Storage) crashes. When we speak of crashing, we
mean that the daemon terminates abnormally because of an error. There are
many cases where Bareos detects errors (such as PIPE errors) and will fail
a job. These are not considered crashes. In addition, under certain
conditions, Bareos will detect a fatal in the configuration, such as
lack of permission to read/write the working directory. In that case,
Bareos will force itself to crash with a SEGFAULT. However, before
crashing, Bareos will normally display a message indicating why.
For more details, please read on.
\section{Traceback}
\index[general]{Traceback}
Each of the three Bareos daemons has a built-in exception handler which, in
case of an error, will attempt to produce a traceback. If successful the
traceback will be emailed to you.
For this to work, you need to ensure that a few things are setup correctly on
your system:
\begin{enumerate}
\item You must have a version of Bareos built with debug information turned
on and not stripped of debugging symbols.
\item You must have an installed copy of {\bf gdb} (the GNU debugger), and it
must be on {\bf Bareos's} path. On some systems such as Solaris, {\bf
gdb} may be replaced by {\bf dbx}.
\item The Bareos installed script file {\bf btraceback} must be in the same
directory as the daemon which dies, and it must be marked as executable.
\item The script file {\bf btraceback.gdb} must have the correct path to it
specified in the {\bf btraceback} file.
\item You must have a {\bf mail} program which is on {\bf Bareos's} path.
By default, this {\bf mail} program is set to {\bf bsmtp}, so it must
be correctly configured.
\item If you run either the Director or Storage daemon under a non-root
userid, you will most likely need to modify the {\bf btraceback} file
to do something like {\bf sudo} (raise to root priority) for the
call to {\bf gdb} so that it has the proper permissions to debug
Bareos.
\end{enumerate}
If all the above conditions are met, the daemon that crashes will produce a
traceback report and email it to you. If the above conditions are not true,
you can either run the debugger by hand as described below, or you may be able
to correct the problems by editing the {\bf btraceback} file. I recommend not
spending too much time on trying to get the traceback to work as it can be
very difficult.
The changes that might be needed are to add a correct path to the {\bf gdb}
program, correct the path to the {\bf btraceback.gdb} file, change the {\bf
mail} program or its path, or change your email address. The key line in the
{\bf btraceback} file is:
\footnotesize
\begin{verbatim}
gdb -quiet -batch -x /home/user/bareos/bin/btraceback.gdb \
$1 $2 2>\&1 | bsmtp -s "Bareos traceback" your-address@xxx.com
\end{verbatim}
\normalsize
Since each daemon has the same traceback code, a single btraceback file is
sufficient if you are running more than one daemon on a machine.
\section{Testing The Traceback}
\index[general]{Traceback!Testing The }
\index[general]{Testing The Traceback }
To "manually" test the traceback feature, you simply start {\bf Bareos} then
obtain the {\bf PID} of the main daemon thread (there are multiple threads).
The output produced here will look different depending on what OS and what
version of the kernel you are running.
Unfortunately, the output had to be split to fit on this page:
\footnotesize
\begin{verbatim}
[user@host]$ ps fax --columns 132 | grep bareos-dir
2103 ? S 0:00 /home/user/bareos/src/dird/bareos-dir -c
/home/user/bareos/src/dird/dird.conf
2104 ? S 0:00 \_ /home/user/bareos/src/dird/bareos-dir -c
/home/user/bareos/src/dird/dird.conf
2106 ? S 0:00 \_ /home/user/bareos/src/dird/bareos-dir -c
/home/user/bareos/src/dird/dird.conf
2105 ? S 0:00 \_ /home/user/bareos/src/dird/bareos-dir -c
/home/user/bareos/src/dird/dird.conf
\end{verbatim}
\normalsize
which in this case is 2103. Then while Bareos is running, you call the program
giving it the path to the Bareos executable and the {\bf PID}. In this case,
it is:
\footnotesize
\begin{verbatim}
./btraceback /home/user/bareos/src/dird 2103
\end{verbatim}
\normalsize
It should produce an email showing you the current state of the daemon (in
this case the Director), and then exit leaving {\bf Bareos} running as if
nothing happened. If this is not the case, you will need to correct the
problem by modifying the {\bf btraceback} script.
Typical problems might be that {\bf gdb} or {\bf dbx} for Solaris is not on
the default path. Fix this by specifying the full path to it in the {\bf
btraceback} file. Another common problem is that you haven't modified the
script so that the {\bf bsmtp} program has an appropriate smtp server or
the proper syntax for your smtp server. If you use the {\bf mail} program
and it is not on the default path, it will also fail. On some systems, it
is preferable to use {\bf Mail} rather than {\bf mail}.
\section{Getting A Traceback On Other Systems}
\index[general]{Getting A Traceback On Other Systems}
\index[general]{Systems!Getting A Traceback On Other}
It should be possible to produce a similar traceback on systems other than
Linux, either using {\bf gdb} or some other debugger. Solaris with {\bf dbx}
loaded works quite fine. On other systems, you will need to modify the {\bf
btraceback} program to invoke the correct debugger, and possibly correct the
{\bf btraceback.gdb} script to have appropriate commands for your debugger. If
anyone succeeds in making this work with another debugger, please send us a
copy of what you modified. Please keep in mind that for any debugger to
work, it will most likely need to run as root, so you may need to modify
the {\bf btraceback} script accordingly.
\label{ManuallyDebugging}
\section{Manually Running Bareos Under The Debugger}
\index[general]{Manually Running Bareos Under The Debugger}
\index[general]{Debugger!Manually Running Bareos Under The}
If for some reason you cannot get the automatic traceback, or if you want to
interactively examine the variable contents after a crash, you can run Bareos
under the debugger. Assuming you want to run the Storage daemon under the
debugger (the technique is the same for the other daemons, only the name
changes), you would do the following:
\begin{enumerate}
\item Start the Director and the File daemon. If the Storage daemon also
starts, you will need to find its PID as shown above (ps fax | grep
bareos-sd) and kill it with a command like the following:
\footnotesize
\begin{verbatim}
kill -15 PID
\end{verbatim}
\normalsize
where you replace {\bf PID} by the actual value.
\item At this point, the Director and the File daemon should be running but
the Storage daemon should not.
\item cd to the directory containing the Storage daemon
\item Start the Storage daemon under the debugger:
\footnotesize
\begin{verbatim}
gdb ./bareos-sd
\end{verbatim}
\normalsize
\item Run the Storage daemon:
\footnotesize
\begin{verbatim}
run -s -f -c ./bareos-sd.conf
\end{verbatim}
\normalsize
You may replace the {\bf ./bareos-sd.conf} with the full path to the Storage
daemon's configuration file.
\item At this point, Bareos will be fully operational.
\item In another shell command window, start the Console program and do what
is necessary to cause Bareos to die.
\item When Bareos crashes, the {\bf gdb} shell window will become active and
{\bf gdb} will show you the error that occurred.
\item To get a general traceback of all threads, issue the following command:
\footnotesize
\begin{verbatim}
thread apply all bt
\end{verbatim}
\normalsize
After that you can issue any debugging command.
\end{enumerate}
\section{Getting Debug Output from Bareos}
\index[general]{Getting Debug Output from Bareos }
Each of the daemons normally has debug compiled into the program, but
disabled. There are two ways to enable the debug output. One is to add the
{\bf -d nnn} option on the command line when starting the debugger. The {\bf
nnn} is the debug level, and generally anything between 50 and 200 is
reasonable. The higher the number, the more output is produced. The output is
written to standard output.
The second way of getting debug output is to dynamically turn it on using the
Console using the {\bf setdebug} command. The full syntax of the command is:
\footnotesize
\begin{verbatim}
setdebug level=nnn client=client-name storage=storage-name dir
\end{verbatim}
\normalsize
If none of the options are given, the command will prompt you. You can
selectively turn on/off debugging in any or all the daemons (i.e. it is not
necessary to specify all the components of the above command).