This repository has been archived by the owner on Mar 19, 2021. It is now read-only.
/
ndmp.tex
1989 lines (1676 loc) · 75.4 KB
/
ndmp.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\index[general]{NDMP!Overview}
\section{NDMP Basics}
NDMP
\begin{itemize}
\item is the abbreviation for Network Data Management Protocol.
\item is a protocol that transports data between Network Attached Storages (NAS) and backup devices.
\item is widely used by storage product vendors and OS vendors like NetApp, Isilon, EMC, Oracle.
\item information is available at \url{http://www.ndmp.org/}.
\item version is currently (2016) NDMP Version 4.
\item uses TCP/IP and XDR (External Data Representation) for the communication.
\end{itemize}
The Bareos NDMP implementation is based on the NDMJOB NDMP reference implementation of Traakan, Inc., Los Altos, CA which has a BSD style license (2 clause one) with some enhancements.
% Technical details can be found in README.NDMP in the bareos sourcecode.
In NDMP, there are different components (called \bquote{agents}) involved in doing backups.
The most important agents are:
\begin{description}
\item[Data Management Agent (DMA)]
is the part that controls the NDMP backup or recover operation.
%does the administration of the backup in NDMP and commands the Primary and Secondary Storage Systems to do a backup or restore operation.
\item[Data Agent] (or Primary Storage System)
is the part that reads the data from the Filesystem during Backup and writes data to the Filesystem during recover.
%is the Storage System that will be backed up, e.g. a NAS Server that is NDMP enabled.
\item[Tape Agent] (or Secondary Storage System)
is the part that writes NDMP blocks to the Tape during backup and reads them during recover.
%is the Storage System that the backup data will be stored on.
\item[Robot Agent]
is the part that controls the media changer. It loads/unloads tapes and gets the inventory of the Changer. The use of a robot agent is optional. It is possible to run backups on a single tape drive.
\end{description}
All elements involved talk to each other via the NDMP protocol which is usually transported via TCP/IP port 10000.
The \DataManagementAgent is part of the Backup Application.
The \DataAgent is part of the (NAS)-System being backed up and recovered.
The \TapeAgent and \RobotAgent can
\begin{itemize}
\item run on the system being backed up
\item run as part of the backup application
\item or even run independently on a third system
\end{itemize}
This flexibility leads to different topologies how NDMP backups can be done.
\subsection{NDMP Topologies}
When looking at the different topologies, the location of the \RobotAgent is not specially considered,
as the data needed to control the robot is minimal compared to the backup data.
So the parts considered are
\begin{itemize}
\item the \DataManagementAgent controlling the operation
\item the \DataAgent as source of the backup data and
\item the \TapeAgent as destination of the backup data
\end{itemize}
The \DataManagementAgent always controls both \DataAgent and \TapeAgent over the Network via NDMP.
The \TapeAgent can either
\begin{itemize}
\item run on a separate system
\item run on the same system
\end{itemize}
as the Data Agent.
\subsubsection{NDMP 3-way Backup: Data Agent and Tape Agent running on different systems}
\begin{verbatim}
--+--------------- NETWORK ----+-------------------+----
| -->---->---->-->-->-->\ | //==>==>==>==>\\ |
| / | | || (2) || |
| | | | || || |
/----------\ /----------\ /----------\
| | | | | |
| DMA | DISK====>| DATA | | Tape |====>TAPE DRIVE
| | (1) | Agent | | Agent | (3)
\----------/ \----------/ \----------/
\end{verbatim}
The data path consists of three ways
\begin{itemize}
\item From Disk to Data Agent (1)
\item From Data Agent over the Network to the Tape Agent (2)
\item From Tape Agent to the Tape (3)
\end{itemize}
and is called NDMP 3-way Backup.
\subsubsection{NDMP 2-way Backup: Data Agent and Tape Agent running on the same system}
\begin{verbatim}
--+--------------- NETWORK ----+---------
| -->---->---->-->-->-->\ |
| / | |
| | | |
/----------\ /----------\
| | | Data |
| DMA | DISK====>| Agent |
| | (1) | |
\----------/ | Tape | (2)
| Agent |====>TAPE DRIVE
\----------/
\end{verbatim}
\DataAgent and \TapeAgent are both part of the same process on the system,
so the data path consists of two ways:
\begin{itemize}
\item From Disk to Data Agent (1)
\item From Tape Agent to the Tape (2)
\end{itemize}
and is called NDMP 2-way Backup, also sometimes referred as NDMP local backup.
\subsubsection{Properties of the different NDMP Backup topologies}
NDMP 3-way backup:
\begin{itemize}
\item The data can be send to a different location over the network
\item No need to attach a tape drive to the NAS system.
\item The backup speed is usually slower than with 2-way backup as the data is being sent over the network and processed multiple times.
\end{itemize}
NDMP 2-way backup:
\begin{itemize}
\item The data is directly copied from the NAS system to the Tape
\item Usually the fastest way to do a NDMP backup
\item tape drives need to be attached to the NAS System
\end{itemize}
\section{NDMP Backup in Bareos}
Bareos offers two types of NDMP integration:
\begin{description}
\item[NDMP\_NATIVE]
\item[NDMP\_BAREOS]
\end{description}
In both cases,
\begin{itemize}
\item \bareosDir acts as \DataManagementAgent.
\item The \DataAgent is part of the storage system
and must be provided by the storage vendor.
\end{itemize}
The main difference is which \TapeAgent is used.
When using NDMP\_BAREOS, the \bareosSd acts as \TapeAgent.
When using NDMP\_NATIVE, the \TapeAgent must be provided by some other systems.
Some storage vendors provide it with there storages, or offer it as an option,
e.g. Isilon with their \bquote{Isilon Backup Accelerator}.
\begin{center}
\begin{tabular}{l | c | c}
\hline
& \NdmpBareos & \NdmpNative \\
\hline
\DataManagementAgent & \bareosDir & \bareosDir \\
\TapeAgent & \bareosSd & external \\
requires external \TapeAgent & & \bcheckmark \\
backup to tape (and VTL) & \bcheckmark & \bcheckmark \\
backup to other \linkResourceDirective{Sd}{Device}{Device Type}
& \bcheckmark & \\
2-way backup & & \bcheckmark \\
3-way backup & \bcheckmark & untested \\
Full Backups & \bcheckmark & \bcheckmark \\
Differential Backups & \bcheckmark & \bcheckmark \\
Incremental Backups & \cmlink{sec:NdmpBackupLevel} (8) & \cmlink{sec:NdmpBackupLevel} (8)\\
Single File Restore & \bcheckmark & \bcheckmark \\
DAR & & \bcheckmark \\
DDAR & & \bcheckmark \\
\ilink{Copy and Migration jobs}{MigrationChapter}
& \bcheckmark & \\
\hline
\end{tabular}
\end{center}
\section{NDMP\_BAREOS}
\label{sec:NdmpBareos}
Bareos implements the \DataManagementAgent inside of the \bareosDir
and a \TapeAgent in the \bareosSd.
The \TapeAgent in the \bareosSd emulates a NDMP tape drive that has an infinite tape.
Because of the infinite tape, no \RobotAgent is required and therefore not implemented.
The blocks being written to the NDMP tape are wrapped into a normal Bareos backup stream
and then stored into the volumes managed by Bareos.
There is always a pair of storage resource definitions:
\begin{itemize}
\item a conventional Bareos storage resource and
\item a NDMP storage resource
\end{itemize}
These two are linked together.
Data that is received by the \TapeAgent inside of the \bareosSd is then stored as Bareos backup stream inside of the paired conventional Bareos storage resource.
On restore, the data is read by the conventional resource, and then recovered as NDMP stream from the NDMP resource.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\linewidth]{\idir ndmp-backup}
\caption{Relationship between Bareos and NDMP components}
\end{figure}%
\subsection{Example Setup for NDMP\_BAREOS backup}
\index[general]{NDMP!Example!NDMP\_BAREOS}
This example starts from a clean default Bareos installation.
\subsubsection{Enable NDMP on your storage appliance}
The storage appliance needs to be configured to allow NDMP connections. Therefore usually the
NDMP service needs to be enabled and configured with a username and password.
\subsubsection{Bareos Director: Configure NDMP Client Resource}
Add a Client resource to the \bareosDir configuration
and configure it to access your NDMP storage system (Primary Storage System/\DataAgent).
\begin{itemize}
\item \linkResourceDirective{Dir}{Client}{Protocol} must be either NDMPv2, NDMPv3 or NDMPv4.
\item \linkResourceDirective{Dir}{Client}{Port} is set to the NDMP Port (usually 10000).
\item \linkResourceDirective{Dir}{Client}{Username} and \linkResourceDirective{Dir}{Client}{Password} are used for the authentication against the NDMP Storage System.
\item \linkResourceDirective{Dir}{Client}{Auth Type} is either Cleartext or MD5. NDMP supports both.
\end{itemize}
In our example we connect to a Isilon storage appliance emulator:
\begin{bconfig}{}
Client {
Name = ndmp-client
Address = isilon.example.com
Port = 10000 # Default port for NDMP
Protocol = NDMPv4 # Need to specify protocol before password as protocol determines password encoding used
Auth Type = Clear # Cleartext Authentication
Username = "ndmpadmin" # username of the NDMP user on the DATA AGENT e.g. storage box being backuped.
Password = "secret" # password of the NDMP user on the DATA AGENT e.g. storage box being backuped.
}
\end{bconfig}
Verify, that you can access your Primary Storage System via Bareos:
\begin{bconsole}{verify connection to NDMP Primary Storage System}
*<input>status client=ndmp-client</input>
Data Agent isilon.example.com NDMPv4
Host info
hostname isilonsim-1
os_type Isilon OneFS
os_vers v7.1.1.5
hostid 005056ad8483ba43cc55a711cd384506e3c1
Server info
vendor Isilon
product Isilon NDMP
revision 2.2
auths (2) NDMP4_AUTH_TEXT NDMP4_AUTH_MD5
Connection types
addr_types (2) NDMP4_ADDR_TCP NDMP4_ADDR_LOCAL
Backup type info of tar format
attrs 0x7fe
set FILESYSTEM=/ifs
set FILES=
set EXCLUDE=
set PER_DIRECTORY_MATCHING=N
set HIST=f
set DIRECT=N
set LEVEL=0
set UPDATE=Y
set RECURSIVE=Y
set ENCODING=UTF-8
set ENFORCE_UNIQUE_NODE=N
set PATHNAME_SEPARATOR=/
set DMP_NAME=
set BASE_DATE=0
set NDMP_UNICODE_FH=N
Backup type info of dump format
attrs 0x7fe
set FILESYSTEM=/ifs
set FILES=
set EXCLUDE=
set PER_DIRECTORY_MATCHING=N
set HIST=f
set DIRECT=N
set LEVEL=0
set UPDATE=Y
set RECURSIVE=Y
set ENCODING=UTF-8
set ENFORCE_UNIQUE_NODE=N
set PATHNAME_SEPARATOR=/
set DMP_NAME=
set BASE_DATE=0
set NDMP_UNICODE_FH=N
File system /ifs
physdev OneFS
unsupported 0x0
type NFS
status
space 12182519808 total, 686768128 used, 11495751680 avail
inodes 17664000 total, 16997501 used
set MNTOPTS=
set MNTTIME=00:00:00 00:00:00
\end{bconsole}
This output shows that the access to the storage appliance was successful.
\subsubsection{Bareos Storage Daemon: Configure NDMP}
\label{sec:ndmp-sd-configure}
\subsubsubsection{Enabling NDMP}
To enable the NDMP \TapeAgent inside of the \bareosSd,
set \linkResourceDirective{Sd}{Storage}{NDMP Enable}=yes:
\begin{bconfig}{enable NDMP in \bareosSd}
#
# Default SD config block: enable the NDMP protocol,
# otherwise it won't listen on port 10000.
#
Storage {
Name = ....
...
NDMP Enable = yes
}
\end{bconfig}
\subsubsubsection{Add a NDMP resource}
%The NDMP service in the \bareosSd also needs access credentials.
Additionally, we need to define the access credentials for our NDMP TAPE AGENT (Secondary Storage) inside of this Storage Daemon.
%The NDMP Storage definition in the Director configuration will has the matching login information.
These are configured by adding a NDMP resource to \file{bareos-sd.conf}:
\begin{bconfig}{}
#
# This resource gives the DMA in the Director access to the Bareos SD via the NDMP protocol.
# This option is used via the NDMP protocol to open the right TAPE AGENT connection to your
# Bareos SD via the NDMP protocol. The initialization of the SD is done via the native protocol
# and is handled via the PairedStorage keyword.
#
Ndmp {
Name = bareos-dir-isilon
Username = ndmpadmin
Password = test
AuthType = Clear
}
\end{bconfig}
Username and Password can be anything, but they will have to match the settings in the \bareosDir NDMP Storage resource we configure next.
Now restart the \bareosSd.
If everything is correct, the \bareosSd starts and listens now on the usual port (9103) and additionally on port 10000 (ndmp).
\begin{commands}{}
<command>netstat</command> <parameter>-lntp | grep bareos-sd</parameter>
tcp 0 0 0.0.0.0:9103 0.0.0.0:* LISTEN 10661/bareos-sd
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 10661/bareos-sd
\end{commands}{}
\subsubsection{Bareos Director: Configure a Paired Storage}
For NDMP Backups, we always need two storages that are paired together.
The default configuration already has a Storage \resourcename{Dir}{Storage}{File} defined:
\begin{bconfig}{}
Storage {
Name = File
Address = bareos
Password = "pNZ3TvFAL/t+MyOIQo58p5B/oB79SFncdAmLXKHa9U59"
Device = FileStorage
Media Type = File
}
\end{bconfig}
We now add a paired storage to the already existing \resourcename{Dir}{}{File} storage:
\begin{bconfig}{}
#
# Same storage daemon but via NDMP protocol.
# We link via the PairedStorage config option the Bareos SD
# instance definition to a NDMP TAPE AGENT.
#
Storage {
Name = NDMPFile
Address = bareos
Port = 10000
Protocol = NDMPv4
Auth Type = Clear
Username = ndmpadmin
Password = "test"
Device = FileStorage
Media Type = File
PairedStorage = File
}
\end{bconfig}
The settings of Username and Password need to match the settings of the \bareosSd's NDMP resource we added before.
The address will be used by the storage appliance's NDMP Daemon to connect to the \bareosSd via NDMP.
Make sure that the Storage appliance can resolve the name or use an IP address.
Now save the director resource and restart the \bareosDir.
Verify that the configuration is correct:
\begin{bconsole}{verify connection to the \bareosSd}
*<input>status storage=NDMPFile</input>
Connecting to Storage daemon File at bareos:9103
bareos-sd Version: 15.2.2 (16 November 2015) x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
Daemon started 14-Jan-16 10:10. Jobs: run=0, running=0.
Heap: heap=135,168 smbytes=34,085 max_bytes=91,589 bufs=75 max_bufs=77
Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0 bwlimit=0kB/s
Running Jobs:
No Jobs running.
====
Jobs waiting to reserve a drive:
====
Terminated Jobs:
====
Device status:
Device "FileStorage" (/var/lib/bareos/storage) is not open.
==
====
Used Volume status:
====
====
*
\end{bconsole}
The output looks the same, as if a \bcommand{status storage=File} would have been called.
% for the paired storage as when the director notices the NDMP
% storage has so called paired storage it will use the native protocol to get the actual status via the
% paired storage definition.
\subsubsection{Bareos Director: Configure NDMP Fileset}
\label{sec:NdmpFileset}
To specify what files and directories from the storage appliance should be backed up,
a Fileset needs to be specified. In our example, we decided to backup \path|/ifs/home| directory.
The specified directory needs to be a filesystem or a subdirectory of a filesystem which can be accessed
by NDMP. Which filesystems are available is showed in the \bcommand{status}{client} output of the NDMP client.
\index[general]{NDMP!Environment variables}
Additionally, NDMP can be configured via NDMP environment variables. These can be specified in the Options
Block of the Fileset with the \configdirective{Meta} keyword. Which variables are available is partly depending on the
NDMP implementation of the Storage Appliance.
\begin{bconfig}{NDMP Fileset}
Fileset {
Name = "NDMP Fileset"
Include {
Options {
meta = "BUTYPE=DUMP"
meta = "USE_TBB_IF_AVAILABLE=y"
meta = "FH_REPORT_FULL_DIRENTS=y"
meta = "RESTORE_HARDLINK_BY_TABLE=y"
}
File = /ifs/home
}
}
\end{bconfig}
% meta = "ENCODING=ISO_8859_1"
\warning{Normally (\linkResourceDirective{Dir}{Client}{Protocol}=Native) Filesets get handled by the \bareosFd. When connecting directly to a NDMP Clients (\linkResourceDirective{Dir}{Client}{Protocol}=NDMP*), no \bareosFd is involved and therefore most Fileset options can't be used. Instead, parameters are handled via \configdirective{Options - Meta} from \linkResourceDirective{Dir}{FileSet}{Include}.}
\warning{Avoid using multiple \linkResourceDirective{Dir}{FileSet}{Include} \configdirective{File} directives.
The \bareosDir would try to handle them by running multiple NDMP jobs in a single Bareos job.
Even if this is working fine during backup, restore jobs will cause trouble.}
Some NDMP environment variables are set automatically by the DMA in the \bareosDir.
The following environment variables are currently set automatically:
\begin{description}
\item[FILESYSTEM] is set to the \linkResourceDirective{Dir}{FileSet}{Include} \configdirective{File} directive.
\item[HIST] = Y \\
Specifies the file history format:
\begin{description}
\item[Y] Specifies the default file history format determined by your NDMP backup settings.
\item[N] Disables file history. Without file hostory, single file restore is not possible with Bareos.
\end{description}
Some NDMP environments (eg. Isilon OneFS) allow additional parameter:
\begin{description}
\item[F] Specifies path-based file history. This is the most efficient with Bareos.
% because it don't require to temporary store information in LMDB
\item[D] Specifies directory or node file history.
\end{description}
\item[LEVEL] is set accordingly to \nameref{sec:NdmpBackupLevel}.
\item[PREFIX]
\item[TYPE] is set accordingly to BUTYPE. Default \bquote{DUMP}.
\item[UPDATE] = Y
\end{description}
\subsubsubsection{Example NDMP Fileset to backup a subset of a NDMP filesystem}
The following fileset is intended to backup all files and directories matching \path|/ifs/home/users/a*|.
It has been tested against Isilon OneFS 7.2.0.1.
See \externalReferenceIsilonNdmpEnvironmentVariables for details about the supported NDMP environment variables.
Excludes are not used in this example.
\begin{bconfig}{NDMP Fileset Isilon Include/Exclude}
Fileset {
Name = "isilon_fileset_home_a"
Include {
Options {
meta = "BUTYPE=DUMP"
meta = "USE_TBB_IF_AVAILABLE=y"
#
# EXCLUDE
#
#meta = "EXCLUDE=[b-z]*"
#
# INCLUDE
#
meta = "FILES=a*"
}
File = /ifs/home/users
}
}
\end{bconfig}
\subsubsection{Bareos Director: Configure NDMP Jobs}
To do NDMP backups and restores, some special settings need to be configured. We define special Backup and Restore jobs for NDMP.
\begin{bconfig}{NDMP backup job}
Job {
Name = "ndmp-backup-job"
Type = Backup
Protocol = NDMP_BAREOS
Level = Incremental
Client = ndmp-client
Backup Format = dump
FileSet = "NDMP Fileset"
Storage = NDMPFile
Pool = Full
Messages = Standard
}
\end{bconfig}
\begin{bconfig}{NDMP restore job}
Job {
Name = "ndmp-restore-job"
Type = Restore
Protocol = NDMP_BAREOS
Client = ndmp-client
Backup Format = dump
FileSet = "NDMP Fileset"
Storage = NDMPFile
Pool = Full
Messages = Standard
Where = /
}
\end{bconfig}
\begin{itemize}
\item \linkResourceDirective{Dir}{Job}{Backup Format}=dump is used in our example. Other Backup Formats have other advantages/disadvantages.
\end{itemize}
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\linewidth]{\idir ndmp-cfg}
\caption{NDMP configuration overview}
\label{fig:ndmp-overview}
\end{figure}%
\subsection{Run NDMP Backup}
Now we are ready to do our first NDMP backup:
\begin{bconsole}{run NDMP backup}
*<input>run job=ndmp-backup-job</input>
Using Catalog "MyCatalog"
Run Backup job
JobName: ndmp-backup-job
Level: Incremental
Client: ndmp-client
Format: dump
FileSet: NDMP Fileset
Pool: Full (From Job resource)
Storage: NDMPFile (From Job resource)
When: 2016-01-14 10:48:04
Priority: 10
OK to run? (yes/mod/no): <input>yes</input>
Job queued. JobId=1
*<input>wait jobid=1</input>
JobId=1
JobStatus=OK (T)
*<input>list joblog jobid=1</input>
2016-01-14 10:57:53 bareos-dir JobId 1: Start NDMP Backup JobId 1, Job=NDMPJob.2016-01-14_10.57.51_04
2016-01-14 10:57:53 bareos-dir JobId 1: Created new Volume "Full-0001" in catalog.
2016-01-14 10:57:53 bareos-dir JobId 1: Using Device "FileStorage" to write.
2016-01-14 10:57:53 bareos-dir JobId 1: Opening tape drive LPDA-DEJC-ENJL-AHAI-JCBD-LICP-LKHL-IEDK@/ifs/home%0 read/write
2016-01-14 10:57:53 bareos-sd JobId 1: Labeled new Volume "Full-0001" on device "FileStorage" (/var/lib/bareos/storage).
2016-01-14 10:57:53 bareos-sd JobId 1: Wrote label to prelabeled Volume "Full-0001" on device "FileStorage" (/var/lib/bareos/storage)
2016-01-14 10:57:53 bareos-dir JobId 1: Commanding tape drive to rewind
2016-01-14 10:57:53 bareos-dir JobId 1: Waiting for operation to start
2016-01-14 10:57:53 bareos-dir JobId 1: Async request NDMP4_LOG_MESSAGE
2016-01-14 10:57:53 bareos-dir JobId 1: Operation started
2016-01-14 10:57:53 bareos-dir JobId 1: Monitoring backup
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: 'Filetransfer: Transferred 5632 bytes in 0.087 seconds throughput of 63.133 KB/s'
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: 'Filetransfer: Transferred 5632 total bytes '
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: 'CPU user=0.016416 sys=0.029437 ft=0.077296 cdb=0.000000'
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: 'maxrss=14576 in=13 out=22 vol=155 inv=72'
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: '
Objects (scanned/included):
----------------------------
Regular Files: (1/1)
Sparse Files: (0/0)
Stub Files: (0/0)
Directories: (2/2)
ADS Entries: (0/0)
ADS Containers: (0/0)
Soft Links: (0/0)
Hard Links: (0/0)
Block Device: (0/0)
Char Device: (0/0)
FIFO: (0/0)
Socket: (0/0)
Whiteout: (0/0)
Unknown: (0/0)'
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: '
Dir Depth (count)
----------------------------
Total Dirs: 2
Max Depth: 1
File Size (count)
----------------------------
== 0 0
<= 8k 1
<= 64k 0
<= 1M 0
<= 20M 0
<= 100M 0
<= 1G 0
> 1G 0
-------------------------
Total Files: 1
Total Bytes: 643
Max Size: 643
Mean Size: 643'
2016-01-14 10:57:53 bareos-dir JobId 1: LOG_MESSAGE: '
File History
----------------------------
Num FH_HIST_FILE messages: 0
Num FH_HIST_DIR messages: 6
Num FH_HIST_NODE messages: 3'
2016-01-14 10:57:54 bareos-dir JobId 1: Async request NDMP4_NOTIFY_MOVER_HALTED
2016-01-14 10:57:54 bareos-dir JobId 1: DATA: bytes 2053KB MOVER: written 2079KB record 33
2016-01-14 10:57:54 bareos-dir JobId 1: Operation done, cleaning up
2016-01-14 10:57:54 bareos-dir JobId 1: Waiting for operation to halt
2016-01-14 10:57:54 bareos-dir JobId 1: Commanding tape drive to NDMP9_MTIO_EOF 2 times
2016-01-14 10:57:54 bareos-dir JobId 1: Commanding tape drive to rewind
2016-01-14 10:57:54 bareos-dir JobId 1: Closing tape drive LPDA-DEJC-ENJL-AHAI-JCBD-LICP-LKHL-IEDK@/ifs/home%0
2016-01-14 10:57:54 bareos-dir JobId 1: Operation halted, stopping
2016-01-14 10:57:54 bareos-dir JobId 1: Operation ended OKAY
2016-01-14 10:57:54 bareos-sd JobId 1: Elapsed time=00:00:01, Transfer rate=2.128 M Bytes/second
2016-01-14 10:57:54 bareos-dir JobId 1: Bareos bareos-dir 15.2.2 (16Nov15):
Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
JobId: 1
Job: ndmp-backup-job.2016-01-14_10.57.51_04
Backup Level: Full
Client: "ndmp-client"
FileSet: "NDMP Fileset" 2016-01-14 10:57:51
Pool: "Full" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "NDMPFile" (From Job resource)
Scheduled time: 14-Jan-2016 10:57:51
Start time: 14-Jan-2016 10:57:53
End time: 14-Jan-2016 10:57:54
Elapsed time: 1 sec
Priority: 10
NDMP Files Written: 3
SD Files Written: 1
NDMP Bytes Written: 2,102,784 (2.102 MB)
SD Bytes Written: 2,128,987 (2.128 MB)
Rate: 2102.8 KB/s
Volume name(s): Full-0001
Volume Session Id: 4
Volume Session Time: 1452764858
Last Volume Bytes: 2,131,177 (2.131 MB)
Termination: Backup OK
\end{bconsole}
We have successfully created our first NDMP backup.
Let us have a look what files are in our backup:
\begin{bconsole}{list the files of the backup job}
*<input>list files jobid=1</input>
/@NDMP/ifs/home%0
/ifs/home/
/ifs/home/admin/
/ifs/home/admin/.zshrc
\end{bconsole}
The real backup data is stored in the file \path|/@NDMP/ifs/home%0|
(we will refer to it as \bquote{NDMP main backup file} or \bquote{main backup file} later on).
One NDMP main backup file is created for every directory specified in the used Fileset.
The other files show the file history and are hardlinks to the backup file.
\subsection{Run NDMP Restore}
Now that we have a NDMP backup, we of course also want to restore some data from the backup.
If the backup we just did saved the Filehistory, we are able to select single files for restore.
Otherwise, we will only be able to restore the whole backup.
\subsubsection{Full Restore}
Either select all files or the main backup file (\path|/@NDMP/ifs/home%0|).
If file history is not included in the backup job, than only the main backup file is available.
\subsubsection{Restore files to original path}
\begin{bconsole}{}
*<input>restore jobid=1</input>
You have selected the following JobId: 1
Building directory tree for JobId(s) 1 ...
2 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.
cwd is: /
$ <input>mark /ifs/home/admin/.zshrc</input>
$ <input>done</input>
Bootstrap records written to /var/lib/bareos/bareos-dir.restore.1.bsr
The job will require the following
Volume(s) Storage(s) SD Device(s)
===========================================================================
Full-0001 File FileStorage
Volumes marked with "*" are online.
1 file selected to be restored.
The defined Restore Job resources are:
1: RestoreFiles
2: ndmp-restore-job
Select Restore Job (1-2): <input>2</input>
Defined Clients:
1: bareos-fd
2: ndmp-client
Select the Client (1-2): <input>2</input>
Run Restore job
JobName: ndmp-backup-job
Bootstrap: /var/lib/bareos/bareos-dir.restore.1.bsr
Where: /
Replace: Always
FileSet: NDMP Fileset
Backup Client: ndmp-client
Restore Client: ndmp-client
Format: dump
Storage: File
When: 2016-01-14 11:04:46
Catalog: MyCatalog
Priority: 10
Plugin Options: *None*
OK to run? (yes/mod/no): <input>yes</input>
Job queued. JobId=2
*<input>wait jobid=2</input>
JobId=2
JobStatus=OK (T)
*<input>list joblog jobid=2</input>
14-Jan 11:04 bareos-dir JobId 2: Start Restore Job ndmp-backup-job.2016-01-14_11.04.53_05
14-Jan 11:04 bareos-dir JobId 2: Using Device "FileStorage" to read.
14-Jan 11:04 bareos-dir JobId 2: Opening tape drive KKAE-IMLO-NHJD-GOCO-GJCO-GEHB-BODL-ADNG@/ifs/home read-only
14-Jan 11:04 bareos-dir JobId 2: Commanding tape drive to rewind
14-Jan 11:04 bareos-dir JobId 2: Waiting for operation to start
14-Jan 11:04 bareos-sd JobId 2: Ready to read from volume "Full-0001" on device "FileStorage" (/var/lib/bareos/storage).
14-Jan 11:04 bareos-sd JobId 2: Forward spacing Volume "Full-0001" to file:block 0:194.
14-Jan 11:04 bareos-dir JobId 2: Async request NDMP4_LOG_MESSAGE
14-Jan 11:04 bareos-dir JobId 2: Operation started
14-Jan 11:04 bareos-dir JobId 2: Monitoring recover
14-Jan 11:04 bareos-dir JobId 2: DATA: bytes 0KB MOVER: read 0KB record 0
14-Jan 11:04 bareos-dir JobId 2: LOG_MESSAGE: 'Filetransfer: Transferred 1048576 bytes in 0.135 seconds throughput of 7557.139 KB/s'
14-Jan 11:04 bareos-dir JobId 2: OK: /admin/.zshrc
14-Jan 11:04 bareos-dir JobId 2: LOG_MESSAGE: '
Objects:
----------------------------
Regular Files: (1)
Stub Files: (0)
Directories: (0)
ADS Entries: (0)
Soft Links: (0)
Hard Links: (0)
Block Device: (0)
Char Device: (0)
FIFO: (0)
Socket: (0)
Unknown: (0)'
14-Jan 11:04 bareos-dir JobId 2: LOG_MESSAGE: '
File Size (count)
----------------------------
== 0 0
<= 8k 1
<= 64k 0
<= 1M 0
<= 20M 0
<= 100M 0
<= 1G 0
> 1G 0
-------------------------
Total Files: 1
Total Bytes: 643
Max Size: 643
Mean Size: 643'
14-Jan 11:04 bareos-dir JobId 2: Async request NDMP4_NOTIFY_MOVER_PAUSED
14-Jan 11:04 bareos-dir JobId 2: DATA: bytes 1024KB MOVER: read 2079KB record 32
14-Jan 11:04 bareos-dir JobId 2: Mover paused, reason=NDMP9_MOVER_PAUSE_EOF
14-Jan 11:04 bareos-dir JobId 2: End of tapes
14-Jan 11:04 bareos-dir JobId 2: DATA: bytes 1024KB MOVER: read 2079KB record 32
14-Jan 11:04 bareos-dir JobId 2: Operation done, cleaning up
14-Jan 11:04 bareos-dir JobId 2: Waiting for operation to halt
14-Jan 11:04 bareos-dir JobId 2: Commanding tape drive to rewind
14-Jan 11:04 bareos-dir JobId 2: Closing tape drive KKAE-IMLO-NHJD-GOCO-GJCO-GEHB-BODL-ADNG@/ifs/home
14-Jan 11:04 bareos-dir JobId 2: Operation halted, stopping
14-Jan 11:04 bareos-dir JobId 2: Operation ended OKAY
14-Jan 11:04 bareos-dir JobId 2: LOG_FILE messages: 1 OK, 0 ERROR, total 1 of 1
14-Jan 11:04 bareos-dir JobId 2: Bareos bareos-dir 15.2.2 (16Nov15):
Build OS: x86_64-redhat-linux-gnu redhat Red Hat Enterprise Linux Server release 7.0 (Maipo)
JobId: 2
Job: ndmp-backup-job.2016-01-14_11.04.53_05
Restore Client: ndmp-client
Start time: 14-Jan-2016 11:04:55
End time: 14-Jan-2016 11:04:57
Elapsed time: 2 secs
Files Expected: 1
Files Restored: 1
Bytes Restored: 1,048,576
Rate: 524.3 KB/s
SD termination status: OK
Termination: Restore OK
\end{bconsole}
\subsubsection{Restore files to different path}
\label{sec:ndmp-where}
The restore location is determined by the \linkResourceDirective{Dir}{Job}{Where} setting of the restore job.
In NDMP, this parameter works in a special manner, the prefix can be either \bquote{relative} to the filesystem or \bquote{absolute}.
If a prefix is set in form of a directory (like \directory{/bareos-restores}), it will be a relative prefix and will be added between the filesystem and the filename. This is needed to make sure that the data is restored in a different directory, but into the same filesystem.
If the prefix is set with a leading caret (\textasciicircum), it will be an absolute prefix and will be put at the front of the restore path. This is needed if the restored data should be stored into a different filesystem.
Example:
\begin{tabular}{l | l | l}
\hline
original file name & where & restored file \\
\hline
\path|/ifs/home/admin/.zshrc| & \path|/bareos-restores| & \path|/ifs/home/bareos-restores/admin/.zshrc| \\
\path|/ifs/home/admin/.zshrc| & \textasciicircum\path|/ifs/data/bareos-restores| & \path|/ifs/data/bareos-restores/admin/.zshrc| \\
\hline
\end{tabular}
\subsection{NDMP Copy Jobs}
\index[general]{Copy!NDMP}
\index[general]{NDMP!Copy jobs}
To be able to do copy jobs, we need to have a second storage resource where we can copy the data to.
Depending on your requirements, this resource can be added to the existing \bareosSd (e.g. \resourcename{Sd}{Storage}{autochanger-0} for tape based backups) or to an additional \bareosSd.
We set up an additional \bareosSd on a host named \host{bareos-sd2.example.com}
with the default \resourcename{Sd}{Storage}{FileStorage} device.
When this is done, add a second storage resource \resourcename{Dir}{Storage}{File2} to the \file{bareos-dir.conf}:
\begin{bconfig}{Storage resource File2}
Storage {
Name = File2
Address = bareos-sd2.example.com
Password = <secretpassword>
Device = FileStorage
Media Type = File
}
\end{bconfig}
Copy Jobs copy data from one pool to another (see \nameref{MigrationChapter}).
So we need to define a pool where the copies will be written to:
Add a Pool that the copies will run to:
\begin{bconfig}{Pool resource Copy}
#
# Copy Destination Pool
#
Pool {
Name = Copy
Pool Type = Backup
Recycle = yes # Bareos can automatically recycle Volumes
AutoPrune = yes # Prune expired volumes
Volume Retention = 365 days # How long should the Full Backups be kept? (#06)
Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
Maximum Volumes = 100 # Limit number of Volumes in Pool
Label Format = "Copy-" # Volumes will be labeled "Full-<volume-id>"
Storage = File2 # Pool belongs to Storage File2
}
\end{bconfig}
Then we need to define the just defined pool as the \linkResourceDirective{Dir}{Pool}{Next Pool} of the pool
that actually holds the data to be copied.
In our case this is the \resourcename{Dir}{Pool}{Full} Pool:
\begin{bconfig}{add Next Pool setting}
#
# Full Pool definition
#
Pool {
Name = Full
[...]
Next Pool = Copy # <- this line needs to be added!
}
\end{bconfig}
Finally, we need to define a Copy Job that will select the jobs that are in the \resourcename{Dir}{Pool}{Full} pool
and copy them over to the \resourcename{Dir}{Pool}{Copy} pool
reading the data via the \resourcename{Dir}{Storage}{File} Storage
and writing the data via the \resourcename{Dir}{Storage}{File2} Storage:
\begin{bconfig}{NDMP copy job}
Job {
Name = NDMPCopy
Type = Copy
Messages = Standard
Selection Type = PoolUncopiedJobs
Pool = Full
Storage = NDMPFile
}
\end{bconfig}
After restarting the director and storage daemon, we can run the Copy job:
\begin{bconsole}{run copy job}
*<input>run job=NDMPCopy</input>
Run Copy job
JobName: NDMPCopy
Bootstrap: *None*
Pool: Full (From Job resource)
NextPool: Copy (From unknown source)
Write Storage: File2 (From Storage from Run NextPool override)
JobId: *None*
When: 2016-01-21 09:19:49