-
Notifications
You must be signed in to change notification settings - Fork 256
/
disk.tex
565 lines (480 loc) · 19.6 KB
/
disk.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
%%
%%
%\chapter{Basic Volume Management}
\label{DiskChapter}
\index[general]{Volume!Management}
\index[general]{Disk Volumes}
This chapter presents most all the features needed to do Volume management.
Most of the concepts apply equally well to both tape and disk Volumes.
However, the chapter was originally written to explain backing up to disk, so
you will see it is slanted in that direction, but all the directives
presented here apply equally well whether your volume is disk or tape.
If you have a lot of hard disk storage or you absolutely must have your
backups run within a small time window, you may want to direct Bareos to
backup to disk Volumes rather than tape Volumes. This chapter is intended to
give you some of the options that are available to you so that you can manage
either disk or tape volumes.
\section{Key Concepts and Resource Records}
\index[general]{Volume!Management!Key Concepts and Resource Records}
Getting Bareos to write to disk rather than tape in the simplest case is
rather easy. In the Storage daemon's configuration file, you simply define an
\linkResourceDirective{Sd}{Device}{Archive Device} to be a directory.
The default directory to store backups on disk is \path|/var/lib/bareos/storage|:
\footnotesize
\begin{verbatim}
Device {
Name = FileBackup
Media Type = File
Archive Device = /var/lib/bareos/storage
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
}
\end{verbatim}
\normalsize
Assuming you have the appropriate \configresource{Storage} resource in your Director's
configuration file that references the above Device resource,
\footnotesize
\begin{verbatim}
Storage {
Name = FileStorage
Address = ...
Password = ...
Device = FileBackup
Media Type = File
}
\end{verbatim}
\normalsize
Bareos will then write the archive to the file {\bf
/var/lib/bareos/storage/{\textless}volume-name{\textgreater}}
where {\textless}volume-name{\textgreater} is the
volume name of a Volume defined in the Pool. For example, if you have labeled
a Volume named {\bf Vol001}, Bareos will write to the file {\bf
/var/lib/bareos/storage/Vol001}. Although you can later move the archive file to
another directory, you should not rename it or it will become unreadable by
Bareos. This is because each archive has the filename as part of the internal
label, and the internal label must agree with the system filename before
Bareos will use it.
Although this is quite simple, there are a number of problems. The first is
that unless you specify otherwise, Bareos will always write to the same volume
until you run out of disk space. This problem is addressed below.
In addition, if you want to use concurrent jobs that write to several
different volumes at the same time, you will need to understand a number
of other details. An example of such a configuration is given
at the end of this chapter under \nameref{ConcurrentDiskJobs}.
\subsection{Pool Options to Limit the Volume Usage}
\index[general]{Pool!Options to Limit the Volume Usage}
Some of the options you have, all of which are specified in the Pool record,
are:
\begin{itemize}
\item \linkResourceDirective{Dir}{Pool}{Maximum Volume Jobs}: write only the specified number of jobs on each Volume.
\item \linkResourceDirective{Dir}{Pool}{Maximum Volume Bytes}: limit the maximum size of each Volume.
Note, if you use disk volumes you should probably limit the Volume size to some reasonable
value. If you ever have a partial
hard disk failure, you are more likely to be able to recover more data
if they are in smaller Volumes.
\item \linkResourceDirective{Dir}{Pool}{Volume Use Duration}: restrict the time between first and last data written to Volume.
\end{itemize}
Note that although you probably would not want to limit the number of bytes on
a tape as you would on a disk Volume, the other options can be very useful in
limiting the time Bareos will use a particular Volume (be it tape or disk).
For example, the above directives can allow you to ensure that you rotate
through a set of daily Volumes if you wish.
As mentioned above, each of those directives is specified in the Pool or
Pools that you use for your Volumes. In the case of \linkResourceDirective{Dir}{Pool}{Maximum Volume Jobs},
\linkResourceDirective{Dir}{Pool}{Maximum Volume Bytes} and \linkResourceDirective{Dir}{Pool}{Volume Use Duration},
you can actually
specify the desired value on a Volume by Volume basis. The value specified in
the Pool record becomes the default when labeling new Volumes. Once a Volume
has been created, it gets its own copy of the Pool defaults, and subsequently
changing the Pool will have no effect on existing Volumes. You can either
manually change the Volume values, or refresh them from the Pool defaults using
the \bcommand{update}{volume} command in the Console. As an example
of the use of one of the above, suppose your Pool resource contains:
\begin{bconfig}{Volume Use Duration}
Pool {
Name = File
Pool Type = Backup
Volume Use Duration = 23h
}
\end{bconfig}
then if you run a backup once a day (every 24 hours), Bareos will use a new
Volume for each backup, because each Volume it writes can only be used for 23 hours
after the first write. Note, setting the use duration to 23 hours is not a very
good solution for tapes unless you have someone on-site during the weekends,
because Bareos will want a new Volume and no one will be present to mount it,
so no weekend backups will be done until Monday morning.
\subsection{Automatic Volume Labeling}
\label{AutomaticLabeling}
\index[general]{Label!Automatic Volume Labeling}
\index[general]{Volume!Labeling!Automatic}
Use of the above records brings up another problem -- that of labeling your
Volumes. For automated disk backup, you can either manually label each of your
Volumes, or you can have Bareos automatically label new Volumes when they are
needed.
Please note that automatic Volume labeling can also be used with tapes, but
it is not nearly so practical since the tapes must be pre-mounted. This
requires some user interaction. Automatic labeling from templates does NOT
work with autochangers since Bareos will not access unknown slots. There
are several methods of labeling all volumes in an autochanger magazine.
For more information on this, please see the \nameref{AutochangersChapter} chapter.
Automatic Volume labeling is enabled by making a change to both the \resourcetype{Dir}{Pool}
resource and to the \resourcetype{Sd}{Device} resource shown above.
In the case of the Pool resource, you must provide Bareos with a label format
that it will use to create new names. In the simplest form, the label format
is simply the Volume name, to which Bareos will append a four digit number.
This number starts at 0001 and is incremented for each Volume the catalog
contains. Thus if you modify your Pool resource to be:
\begin{bconfig}{Label Format}
Pool {
Name = File
Pool Type = Backup
Volume Use Duration = 23h
Label Format = "Vol"
}
\end{bconfig}
Bareos will create Volume names Vol0001, Vol0002, and so on when new Volumes
are needed. Much more complex and elaborate labels can be created using
variable expansion defined in the
\ilink{Variable Expansion}{VarsChapter} chapter of this manual.
The second change that is necessary to make automatic labeling work is to give
the Storage daemon permission to automatically label Volumes. Do so by adding
\linkResourceDirective{Sd}{Device}{Label Media} = yes to the \configresource{Device} resource as follows:
\begin{bconfig}{Label Media = yes}
Device {
Name = File
Media Type = File
Archive Device = /var/lib/bareos/storage/
Random Access = yes
Automatic Mount = yes
Removable Media = no
Always Open = no
Label Media = yes
}
\end{bconfig}
See \linkResourceDirective{Dir}{Pool}{Label Format} for details about the labeling format.
\subsection{Restricting the Number of Volumes and Recycling}
\index[general]{Recycling!Restricting the Number of Volumes and Recycling}
\index[general]{Restricting the Number of Volumes and Recycling}
Automatic labeling discussed above brings up the problem of Volume management.
With the above scheme, a new Volume will be created every day. If you have not
specified Retention periods, your Catalog will continue to fill keeping track
of all the files Bareos has backed up, and this procedure will create one new
archive file (Volume) every day.
The tools Bareos gives you to help automatically manage these problems are the
following:
\begin{itemize}
\item \linkResourceDirective{Dir}{Client}{File Retention}: catalog file record retention period.
\item \linkResourceDirective{Dir}{Client}{Job Retention}: catalog job record retention period.
\item \linkResourceDirective{Dir}{Client}{Auto Prune} = yes: permit the application of the above two retention periods.
\item \linkResourceDirective{Dir}{Pool}{Volume Retention}
\item \linkResourceDirective{Dir}{Pool}{Auto Prune} = yes: permit the application of the \linkResourceDirective{Dir}{Pool}{Volume Retention} period.
\item \linkResourceDirective{Dir}{Pool}{Recycle} = yes: permit automatic recycling of Volumes whose Volume retention period has
expired.
\item \linkResourceDirective{Dir}{Pool}{Recycle Oldest Volume} = yes: prune the oldest volume in the Pool, and if all
files were pruned, recycle this volume and use it.
\item \linkResourceDirective{Dir}{Pool}{Recycle Current Volume} = yes: prune the currently mounted volume in the
Pool, and if all files were pruned, recycle this volume and use it.
\item \linkResourceDirective{Dir}{Pool}{Purge Oldest Volume} = yes: permits a forced recycling of the oldest Volume when a new one
is needed.\\
\warning{This record ignores retention periods! We highly
recommend not to use this record, but instead use \linkResourceDirective{Dir}{Pool}{Recycle Oldest Volume}.}
\item \linkResourceDirective{Dir}{Pool}{Maximum Volumes}: limit the number of Volumes that can be created.
\end{itemize}
The first three records
(\linkResourceDirective{Dir}{Client}{File Retention}, \linkResourceDirective{Dir}{Client}{Job Retention} and \linkResourceDirective{Dir}{Client}{Auto Prune})
determine the amount of time that Job and File records will remain in your
Catalog and they are discussed in detail in the
\ilink{Automatic Volume Recycling}{RecyclingChapter} chapter.
\linkResourceDirective{Dir}{Pool}{Volume Retention}, \linkResourceDirective{Dir}{Pool}{Auto Prune} and \linkResourceDirective{Dir}{Pool}{Recycle}
determine how long Bareos will keep
your Volumes before reusing them and they are also discussed in detail in the
\ilink{Automatic Volume Recycling}{RecyclingChapter} chapter.
The \linkResourceDirective{Dir}{Pool}{Maximum Volumes} record
can also be used in conjunction with the \linkResourceDirective{Dir}{Pool}{Volume Retention} period
to limit the total number of archive Volumes that
Bareos will create.
By setting an appropriate \linkResourceDirective{Dir}{Pool}{Volume Retention} period,
a Volume will be purged just before it is needed and thus Bareos can cycle
through a fixed set of Volumes. Cycling through a fixed set of Volumes can
also be done by setting
\linkResourceDirective{Dir}{Pool}{Purge Oldest Volume} = yes or \linkResourceDirective{Dir}{Pool}{Recycle Current Volume} = yes.
In this case, when Bareos needs a new Volume, it will
prune the specified volume.
\section{Concurrent Disk Jobs}
\index[general]{Concurrent Disk Jobs}
\label{ConcurrentDiskJobs}
Above, we discussed how you could have a single device named
\resourcename{Sd}{Device}{FileBackup} that writes to volumes in \fileStoragePath.
You can, in fact, run multiple concurrent jobs using the
Storage definition given with this example, and all the jobs will
simultaneously write into the Volume that is being written.
Now suppose you want to use multiple Pools, which means multiple
Volumes, or suppose you want each client to have its own Volume
and perhaps its own directory such as {\bf /home/bareos/client1}
and {\bf /home/bareos/client2} ... .
With the single Storage and Device definition above, neither of these two is possible. Why? Because
Bareos disk storage follows the same rules as tape devices. Only
one Volume can be mounted on any Device at any time. If you want
to simultaneously write multiple Volumes, you will need multiple
Device resources in your \bareosSd configuration and thus multiple
Storage resources in your \bareosDir configuration.
Okay, so now you should understand that you need multiple Device definitions
in the case of different directories or different Pools, but you also
need to know that the catalog data that Bareos keeps contains only
the Media Type and not the specific storage device. This permits a tape
for example to be re-read on any compatible tape drive. The compatibility
being determined by the
Media Type (\linkResourceDirective{Dir}{Storage}{Media Type} and \linkResourceDirective{Sd}{Device}{Media Type}).
The same applies to disk storage.
Since a volume that is written by a Device in say directory
\path|/home/bareos/backups| cannot be read by a Device with an
\linkResourceDirective{Sd}{Device}{Archive Device} = \path|/home/bareos/client1|,
you will not be able to restore all your files if you give both those devices
\linkResourceDirective{Sd}{Device}{Media Type} = File.
During the restore, Bareos will simply choose
the first available device, which may not be the correct one. If this
is confusing, just remember that the Directory has only the Media Type
and the Volume name. It does not know the \linkResourceDirective{Sd}{Device}{Archive Device} (or the
full path) that is specified in the \bareosSd. Thus you must
explicitly tie your Volumes to the correct Device by using the Media Type.
\subsection{Example for two clients, separate devices and recycling}
The following example is not very practical, but can be used to demonstrate
the proof of concept in a relatively short period of time.
The example
consists of a two clients that are backed up to a set of 12 Volumes for each client
into different directories on the Storage
machine. Each Volume is used (written) only once, and there are four Full
saves done every hour (so the whole thing cycles around after three hours).
What is key here is that each physical device on the \bareosSd
has a different Media Type. This allows the Director to choose the
correct device for restores.
The \bareosDir configuration is as follows:
\begin{bconfig}{}
Director {
Name = bareos-dir
QueryFile = "/usr/lib/bareos/scripts/query.sql"
Password = "<secret>"
}
Schedule {
Name = "FourPerHour"
Run = Level=Full hourly at 0:05
Run = Level=Full hourly at 0:20
Run = Level=Full hourly at 0:35
Run = Level=Full hourly at 0:50
}
FileSet {
Name = "Example FileSet"
Include {
Options {
compression=GZIP
signature=SHA1
}
File = /etc
}
}
Job {
Name = "RecycleExample"
Type = Backup
Level = Full
Client = client1-fd
FileSet= "Example FileSet"
Messages = Standard
Storage = FileStorage
Pool = Recycle
Schedule = FourPerHour
}
Job {
Name = "RecycleExample2"
Type = Backup
Level = Full
Client = client2-fd
FileSet= "Example FileSet"
Messages = Standard
Storage = FileStorage2
Pool = Recycle2
Schedule = FourPerHour
}
Client {
Name = client1-fd
Address = client1.example.com
Password = client1_password
}
Client {
Name = client2-fd
Address = client2.example.com
Password = client2_password
}
Storage {
Name = FileStorage
Address = bareos-sd.example.com
Password = local_storage_password
Device = RecycleDir
Media Type = File
}
Storage {
Name = FileStorage2
Address = bareos-sd.example.com
Password = local_storage_password
Device = RecycleDir2
Media Type = File1
}
Catalog {
Name = MyCatalog
...
}
Messages {
Name = Standard
...
}
Pool {
Name = Recycle
Pool Type = Backup
Label Format = "Recycle-"
Auto Prune = yes
Use Volume Once = yes
Volume Retention = 2h
Maximum Volumes = 12
Recycle = yes
}
Pool {
Name = Recycle2
Pool Type = Backup
Label Format = "Recycle2-"
Auto Prune = yes
Use Volume Once = yes
Volume Retention = 2h
Maximum Volumes = 12
Recycle = yes
}
\end{bconfig}
and the \bareosSd configuration is:
\begin{bconfig}{}
Storage {
Name = bareos-sd
Maximum Concurrent Jobs = 10
}
Director {
Name = bareos-dir
Password = local_storage_password
}
Device {
Name = RecycleDir
Media Type = File
Archive Device = /home/bareos/backups
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
}
Device {
Name = RecycleDir2
Media Type = File2
Archive Device = /home/bareos/backups2
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
}
Messages {
Name = Standard
director = bareos-dir = all
}
\end{bconfig}
With a little bit of work, you can change the above example into a weekly or
monthly cycle (take care about the amount of archive disk space used).
\subsection{Using Multiple Storage Devices}
\label{sec:MultipleStorageDevices}
\index[general]{Multiple Storage Devices}
\index[general]{Storage Device!Multiple}
Bareos treats disk volumes similar to tape volumes as much as it can.
This means that you can only have a single Volume mounted at one time on a disk as defined in your \resourcetype{Sd}{Device} resource.
If you use Bareos without \nameref{sec:DataSpooling},
multiple concurrent backup jobs can be written to a Volume using interleaving.
However, interleaving has disadvantages, see \nameref{sec:Interleaving}.
Also the \resourcetype{Sd}{Device} will be in use. If there are other jobs, requesting other Volumes,
these jobs have to wait.
On a tape (or autochanger), this is a physical limitation of the hardware.
However, when using disk storage, this is only a limitation of the software.
To enable Bareos to run concurrent jobs (on disk storage), define as many \resourcetype{Sd}{Device} as concurrent jobs should run.
All these \resourcetype{Sd}{Device}s can use the same \linkResourceDirective{Sd}{Device}{Archive Device} directory. Set \linkResourceDirective{Sd}{Device}{Maximum Concurrent Jobs} = 1 for all these devices.
\subsubsection{Example: use four storage devices pointing to the same directory}
\begin{bconfig}{\bareosDir configuration: using 4 storage devices}
Director {
Name = bareos-dir.example.com
QueryFile = "/usr/lib/bareos/scripts/query.sql"
Maximum Concurrent Jobs = 10
Password = "<secret>"
}
Storage {
Name = File
Address = bareos-sd.bareos.com
Password = "<sd-secret>"
Device = FileStorage1
Device = FileStorage2
Device = FileStorage3
Device = FileStorage4
# number of devices = Maximum Concurrent Jobs
Maximum Concurrent Jobs = 4
Media Type = File
}
[...]
\end{bconfig}
\begin{bconfig}{\bareosSd configuraton: using 4 storage devices}
Storage {
Name = bareos-sd.example.com
# any number >= 4
Maximum Concurrent Jobs = 20
}
Director {
Name = bareos-dir.example.com
Password = "<sd-secret>"
}
Device {
Name = FileStorage1
Media Type = File
Archive Device = /var/lib/bareos/storage
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
Maximum Concurrent Jobs = 1
}
Device {
Name = FileStorage2
Media Type = File
Archive Device = /var/lib/bareos/storage
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
Maximum Concurrent Jobs = 1
}
Device {
Name = FileStorage3
Media Type = File
Archive Device = /var/lib/bareos/storage
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
Maximum Concurrent Jobs = 1
}
Device {
Name = FileStorage4
Media Type = File
Archive Device = /var/lib/bareos/storage
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
Maximum Concurrent Jobs = 1
}
\end{bconfig}