-
Notifications
You must be signed in to change notification settings - Fork 15
/
bbr-devguide.html.md.erb
581 lines (424 loc) · 22.8 KB
/
bbr-devguide.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
---
title: BOSH Backup and Restore Developer's Guide
owner: BOSH Backup & Restore Team
---
This guide describes the framework for release authors to add backup and restore functionality to their release by using BOSH Backup and Restore (BBR).
BBR is a framework for backing up and restoring BOSH deployments and BOSH Directors. BBR triggers the backup or restore process on the deployment or BOSH Director and transfers the backup artifacts to and from the deployment or BOSH Director.
The BBR framework consists of a command line interface (CLI) and a set of hooks, which call out to scripts. The framework uses BOSH to orchestrate the execution of backup and restore scripts. BBR allows release authors fine-grained control over how their release is backed up.
## <a id="mechanism"></a> Backup mechanism
Different systems require different strategies for backup and restore.
To help release authors make consistent backups of their systems, the BBR framework provides hooks to
lock scripts that bring a job to a consistent state before backup begins.
The following diagram illustrates an example BBR script execution sequence.
![This diagram is described in detail immediately below.](images/script-execution-order.png)
The diagram shows a deployment that has two instance groups.
Group foo contains instance foo/0, which contains jobs 1 and 2.
Group bar contains instances bar/0 and bar/1, each of which contains job 1.
* Stage 1 starts: all pre-backup-lock scripts run for all jobs in all instance groups. Stage 1 ends.
* Stage 2 starts: all backup scripts run for all jobs in all instance groups. Stage 2 ends.
* Stage 3 starts: all post-backup-lock scripts run for all jobs in all instance groups. Stage 3 ends.
There is no fixed order in which BBR calls the scripts within each stage.
Because the framework is focused on the interface, hooks, backup destination, and intra-deployment
orchestration, it is mechanism-agnostic.
Release authors can back up their release however they want to, as long as the scripts they write
have the correct name and directory structure.
To ensure that backup and restore scripts do not get out of sync with the releases themselves,
release authors are responsible for naming, creating, testing, and troubleshooting their own backup
and restore scripts.
<p class="note">
<span class="note__title"><strong>Note</strong></span>
For all releases, these backup and restore jobs might be co-located with
those of other releases. Therefore, release authors must give these jobs unique and descriptive
names to avoid name collisions.
</p>
## <a id='script-implementation'></a>Script organization
BBR sets out a contract with release authors to call designated backup and restore
scripts under the `/var/vcap/jobs/JOB-NAME/bin/bbr/SCRIPT-NAME` directory:
* Backup script names
* `pre-backup-lock`
* `backup`
* `post-backup-unlock`
* Restore script names
* `pre-restore-lock`
* `restore`
* `post-restore-unlock`
* Metadata script name (if needed)
* `metadata`
For example, the directory structure of a templated BBR job looks like the below:
* `/var/vcap/jobs/bbr-uaadb/bin/bbr/backup`
* `/var/vcap/jobs/bbr-uaadb/bin/bbr/post-restore-unlock`
Release authors must implement scripts as part of the BBR contract.
Package and distribute the scripts as part of your BOSH release.
The [Exemplar Backup and Restore Release](https://github.com/cloudfoundry-incubator/exemplar-backup-and-restore-release) in GitHub provides examples
of how release authors can structure their jobs to implement the contract with BBR.
The BBR CLI can remotely locate and run the scripts, even if they are located on different VMs.
For example, the lock/unlock scripts can be located on release VMs
while, due to disk-space constraints, the backup/restore scripts are co-located on a separate backup-restore VM.
Scripts are executed in a specific order.
For the backup workflow, the order is `pre-backup-lock`, `backup`, and then `post-backup-unlock`.
For the restore workflow, the order is `pre-restore-lock`, `restore`, and then `post-restore-unlock`.
All scripts should be executable scripts.
ERB tags can be used for templating.
These scripts are executed similarly to other release job scripts, such as `start`, `stop`, or `drain`, and you can use the job's package dependencies.
BBR checks exit codes of the scripts when orchestrating.
Exit code `0` indicates success, and any other exit code indicates a failure.
### <a id="directory"></a> Directory structure of a BOSH release with BBR scripts
The following is an example directory structure of a BOSH release that includes BBR scripts.
```
acme-release
├── README.md
├── config
│ ├── blobs.yml
│ └── final.yml
├── jobs
│ ├── bbr-acmedb
│ │ ├── monit
│ │ ├── spec
│ │ └── templates
│ │ ├── backup.sh.erb
│ │ ├── config.json.erb
│ │ └── restore.sh.erb
│ └── bbr-lock-unlock-acme
│ ├── monit
│ ├── spec
│ └── templates
│ ├── post-backup-unlock.sh.erb
│ ├── pre-backup-lock.sh.erb
│ └── metadata.sh.erb (optional)
├── packages
└── src
```
### <a id='script-ordering'></a>Script ordering across jobs
During the backup workflow, all `pre-backup-lock` scripts are invoked before the `backup` scripts on all of the jobs in the deployment.
All `backup` scripts are invoked before the `post-backup-unlock` scripts on all of the jobs in the deployment.
<p class="note">
<span class="note__title"><strong>Note</strong></span>
Both the <code>pre-backup-lock</code> and <code>post-backup-unlock</code> are called in parallel while respecting the locking order constraints.
</p>
During the restore workflow, all `pre-restore-lock` scripts are invoked before the `restore` scripts on all of the jobs in the deployment.
All `restore` scripts are invoked before the `post-restore-unlock` scripts on all of the jobs in the deployment.
By default `pre-backup-lock` and `pre-restore-lock` scripts from different jobs are invoked in an arbitrary order.
You can specify an order for lock scripts from specific jobs in the metadata script.
The `post-backup-unlock` and `post-restore-unlock` scripts from different jobs are invoked in an arbitrary order
when no locking dependency is specified, or the opposite order of lock scripts as stated in metadata scripts.
When a job specifies locking dependency to a job that does not exist in the current deployment, BBR ignores that dependency.
For more information, see [Metadata script](#metadata) below.
### <a id='logs'></a> Logs
The `stdout` and `stderr` streams are captured and sent to the operator who invokes the backup and restore.
<p class="note">
<span class="note__title"><strong>Note</strong></span>
Release authors should avoid printing sensitive information to <code>stdout</code> or <code>stderr</code>.
BBR prints any output from the scripts in case of failure. Ensure your script does not print any sensitive data, such as credentials.
In particular, if you are using <code>set -x</code> in your script, add <code>set +x</code> before you use any credentials.
</p>
## <a id='backup-workflow'></a>Backup workflow
### <a id='pre-backup-lock'></a>pre-backup-lock
The release job can have a `pre-backup-lock` script that stops any processes that could make changes to the components being backed up.
This script must allow the job to lock so that backups are consistent across a cluster.
For example, in a Cloud Foundry deployment, the `pre-backup-lock` script stops Cloud Controller processes
that might make changes to its blobstore and database. This ensures the blobstore and the Cloud Controller database are consistent with each other.
#### Job configuration
To add a `pre-backup-lock` script to a job:
<ol>
<li>
Create a script with any name in the templates directory of your job.
</li>
<li>
In the templates section of the release job spec file, add the script name and <code>bin/bbr/pre-backup-lock</code> as a key-value pair. For example:
<pre class="terminal">
---
name: lock-unlock-acme
templates:
pre-backup-lock.sh.erb: bin/bbr/pre-backup-lock
</pre>
</li>
<li>
If you want your <code>bin/bbr/pre-backup-lock</code> scripts to run in a specific order, define that order using the optional metadata script. In the sample directory structure illustrated above, this script is called <code>metadata.sh.erb</code>.<br><br>
a. The metadata script specifies that the current job must be locked before some other job(s) in the deployment.
When run, this script must print a YAML file to stdout. For example:
<pre class="terminal">
#!/usr/bin/env bash
echo "---
backup_should_be_locked_before:
- job_name: lock-unlock-acme
release: acme-release" </pre>
b. Add an entry to the templates section of the release job spec file:
<pre class="terminal">
templates:
...
metadata.sh.erb: bin/bbr/metadata
</li>
</pre>
See <a href="#metadata">Metadata script</a> below for information about the properties you can use in this script.
</ol>
### <a id='backup'></a>Backup
The release job can have a backup script that dumps the backup of the job's database to the directory specified by `$BBR_ARTIFACT_DIRECTORY`. For example, when backing up MySQL, this script can invoke the `mysqldump` binary for the MySQL adapter.
There must be at least one job in the deployment providing a backup script. If there is no backup script, calling `backup` or `pre-backup-check` for the deployment fails.
#### Job configuration
To add a `backup` script to a release job:
1. Create a script with any name in the templates directory of a release job.
2. In the templates section of the release job spec file, add the script name and <code>bin/bbr/backup</code> as a key-value pair. For example:
```
---
name: backup-restore-acme
templates:
backup.erb: bin/bbr/backup
```
### <a id='post-backup-unlock'></a>post-backup-unlock
The backup and restore job can have a `post-backup-unlock` script that undoes the operations performed by `pre-backup-lock`.
#### Job configuration
To add a `post-backup-unlock` script to a release job:
1. Create a script with any name in the templates directory of a release job.
2. In the templates section of the release job spec file, add the script name and <code>bin/bbr/post-backup-unlock</code> as a key-value pair. For example:
```
---
name: lock-unlock-acme
templates:
post-backup-unlock.erb: bin/bbr/post-backup-unlock
```
### <a id='error-handling'></a>Error handling during backup
If errors occur during the backup workflow, cleanup tasks are executed to put the system back in a working state.
A normal workflow for backing up a deployment does the below:
<% mermaid_diagram do %>
graph TB
start(Start) --> check-deployment(Check that the deployment exists)
check-deployment -- Yes --> make-local-artifact(Make a local artifact dir)
check-deployment -- No --> exit(Exit)
make-local-artifact -- Success --> pre-backup(Run pre-backup lock scripts)
make-local-artifact -- Failure --> exit
pre-backup -- Success --> backup(Run backup scripts)
pre-backup -- Failure --> post-backup(Run post-backup-unlock scripts)
backup --> post-backup
post-backup -- backup was successful --> drain(Drains artifact from instance)
post-backup -- backup failed --> cleanup(Removes backup from the instance)
drain --> cleanup
cleanup --> exit
<% end %>
## <a id='restore-workflow'></a>Restore workflow
### <a id='pre-restore-lock'></a>pre-restore-lock
The release job can have a `pre-restore-lock` script that stops any processes that could make changes to the components being restored. This script must allow the job to lock so that restorations are consistent across a cluster.
#### Job configuration
To add a `pre-restore-lock` script to a job:
1. Create a script with any name in the templates directory of your job.
1. In the templates section of the release job spec file, add the script name and <code>bin/bbr/pre-restore-lock</code> as a key-value pair. For example:
```
---
name: lock-unlock-acme
templates:
pre-restore-lock.sh.erb: bin/bbr/pre-restore-lock
```
1. If you want your <code>bin/bbr/pre-restore-lock</code> scripts to run in a specific order, define that order using the optional metadata script. In the sample directory structure illustrated above, this script is called <code>metadata.sh.erb</code>.<br><br>
The `pre-restore-lock` scripts are called before any `restore` scripts have been called. Success indicates the job is ready to be restored.<br><br>
a. The metadata script specifies that the current job must be locked before some other job(s) in the deployment. When run, this script must print a YAML file to stdout. For example:
<pre class="terminal">
#!/usr/bin/env bash
echo "---
restore\_should\_be\_locked\_before:
job_name: lock-unlock-acme
release: acme-release"
</pre>
b. Add an entry to the templates section of the release job spec file:
```
templates:
...
metadata.sh.erb: bin/bbr/metadata
```
See <a href="#metadata">Metadata script</a> below for information about the properties you can use in this script.
### <a id='restore'></a>Restore
If a release has a `backup` script, it should also have a `restore` script. The `restore` script expects a backup artifact to be provided in `$BBR_ARTIFACT_DIRECTORY`. For example, when restoring MySQL, the script invokes `mysql` to restore from a mysqldump file.
#### Job configuration
To add a `restore` script to a release job:
1. Create a script with any name in the templates directory of a release job.
2. In the templates section of the release job spec file, add the script name and <code>bin/bbr/restore</code> as a key-value pair. For example:
```yaml
---
name: backup-restore-acme
templates:
restore.erb: bin/bbr/restore
```
### <a id='post-restore-unlock'></a>post-restore-unlock
The release job can have a `post-restore-unlock` script that resumes normal service operation. `post-restore-unlock` should be idempotent because it can be called multiple times even if `pre-restore-lock` has not been not called.
#### Job configuration
To add a `post-restore-unlock` script to a release:
1. Create a script with any name in the templates directory of your job.
2. In the templates section of the release job spec file, add the script name and <code>bin/bbr/post-restore-unlock</code> as a key-value pair. For example:
```
---
name: lock-unlock-acme
templates:
post-restore-unlock.sh.erb: bin/bbr/post-restore-unlock
```
### <a id='error-handling-restore'></a>Error handling during restore
If errors occur during the restore workflow, cleanup tasks are executed to put the system back in a working state.
A normal workflow for restoring a deployment does the below:
<% mermaid_diagram do %>
graph TB
start(Start) --> validate-artifact(Check that the artifact is present and the checksums match)
validate-artifact --> check-deployment-exists(Check that the deployment exists)
check-deployment-exists -- Yes --> check-deployment-matches-backup(Check the structure of the backup matches the destination deployment)
check-deployment-exists -- No --> exit(Exit)
check-deployment-matches-backup -- Yes --> copy-to-remote(Copy the backup artifacts to the relevant VMs)
check-deployment-matches-backup -- No --> cleanup(Cleanup, close ssh connections)
copy-to-remote -- Success --> pre-restore-lock(Run pre-restore-lock scripts)
copy-to-remote -- Failure --> cleanup
pre-restore-lock -- Success --> restore(Run restore scripts)
pre-restore-lock -- Failure --> post-restore-unlock(Run post-restore-unlock scripts)
restore --> post-restore-unlock
post-restore-unlock --> cleanup
cleanup --> exit
<% end %>
## <a id='metadata'></a>(Optional) Metadata script
The `metadata` script is an optional script that `bbr` executes before
any other scripts to get more information about the jobs, for example,
locking dependencies and whether the BBR job is enabled or not.
The script is expected to print a `yaml` on standard out with more information
about the job for backup and restore.
### Job configuration
To add a `metadata` script to a release job:
1. Create a script with any name in the templates directory of a release job.
2. In the templates section of the release job spec file, add the script name and <code>bin/bbr/metadata</code> as a key-value pair. For example:
```yaml
---
name: backup-restore-acme
templates:
metadata.erb: bin/bbr/metadata
```
### (Optional) Properties
#### backup\_should\_be\_locked\_before
Use this property to specify the locking dependencies for the current job during backup.
The jobs are specified as an array with their release names.
For example:
<pre class="terminal">
#!/usr/bin/env bash
echo "---
backup_should_be_locked_before:
- job_name: lock-unlock-acme
release: acme-release" </pre>
#### restore\_should\_be\_locked\_before
If you want your <code>bin/bbr/pre-restore-lock</code> scripts to run in a
specific order, define that order using the metadata script, as above, but with
the `restore_should_be_locked_before` key.
For example:
<pre class="terminal">
#!/usr/bin/env bash
echo "---
restore_should_be_locked_before:
- job_name: lock-unlock-acme
release: acme-release" </pre>
#### skip\_bbr\_scripts
Use this property to disable a BBR job.
When the metadata script is templated with this property set to true, the BBR CLI
does not run the BBR scripts in this job.
For example:
<pre class="terminal">
#!/usr/bin/env bash
echo "---
skip_bbr_scripts: true" </pre>
## <a id='testing'></a>Testing
The functional testing pattern recommended for BBR scripts is:
<ol>
<li>
Create data for which the release is responsible.
</li>
<li>
Back up the release.
</li>
<li>
Delete data.
</li>
<li>
Restore the release.
</li>
<li>
Validate that restored data are correct.
</li>
</ol>
Where scripts are implemented for releases that are part of larger deployments, you should perform end-to-end testing that validates consistency across releases.
## <a id="bbr-utilities"></a>Backup and restore utilities
If your release stores state in a Postgres or MySQL database, deploy the `database-backup-restorer` job from the [backup-and-restore-sdk-release](https://github.com/cloudfoundry-incubator/backup-and-restore-sdk-release) GitHub repository.
Ensure your job templates a `config.json` as follows:
```json
{
"username": "db user",
"password": "db password",
"host": "db host",
"port": "db port",
"adapter": "bbr supported adapter (e.g. mysql or postgres)",
"database": "database name"
}
```
Ensure your `backup` and `restore` scripts call the appropriate `database-backup-restorer` binaries as follows:
**Backup**
```bash
/var/vcap/jobs/database-backup-restorer/bin/backup /path/to/config.json
cp <output_file> $ARTIFACT_DIRECTORY
```
**Restore**
```bash
cp $ARTIFACT_DIRECTORY <output_file>
/var/vcap/jobs/database-backup-restorer/bin/restore /path/to/config.json
```
## <a id="bbr-cf"></a>BBR and Cloud Foundry databases
<p class="note important">
<span class="note__title"><strong>Important</strong></span>
Release authors must ensure the following.
<br><strong>For CF Releases</strong>, Your scripts must respect the <code>release_level_backup</code> job property so the scripts run if set to <code>true</code>, but do nothing if set to <code>false</code>.
</p>
BBR provides a pattern for Cloud Foundry release authors that ensures their scripts abide by the BBR Framework contract.
A Cloud Foundry operator adds a `backup-restore` instance to their Cloud Foundry deployment. By default, release-specific database `backup` and `restore` job scripts are co-located on this instance along with the `database-backup-restorer` job from the [backup-and-restore-sdk-release](https://github.com/cloudfoundry-incubator/backup-and-restore-sdk-release) GitHub repository.
If your release requires that you stop your component's processes during backup and restore, you also need to provide `unlock` and `lock` scripts. You can co-locate these scripts on either the `backup-restore` instance or your component's instance. You should co-locate these scripts on the component's instance if you want to interact directly with `monit` or your running job.
More information about [`unlock`](#post-backup-unlock) and [`lock`](#pre-backup-lock) scripts are below.
### Naming recommendations
When Cloud Foundry is deployed, multiple backup and restore scripts are co-located on the
`backup-restore` instance, separated by job name.
For this reason, the job names for each release should have a consistent format
so that there is no confusion between BBR jobs.
The recommendations for job names are as follows:
* Use the prefix `bbr-` to show that the job is bbr-related
* Include the name of the component that is being backed up
For example, you have a release named `acme-release` that has a
database (db) that must be backed up.
The recommended BBR job name for this database is `bbr-acmedb`.
This follows the format `bbr-RELEASE-NAMEdb`.
### <a id="bbr-cf-example"></a>Example
Acme Release is a Cloud Foundry component that has no persistent disk on its component VM.
The release is an API that stores its data in a MySQL database deployed on a different
instance group in this CF deployment.
The release authors have chosen to create two new jobs to be incorporated into
their existing release; one job includes the `backup` and `restore` scripts in
its templates directory, and the other job includes the `pre-backup-lock` and
`post-backup-unlock` scripts.
This is because the operator co-locates the former job on the `backup-restore`
VM and the latter job on the Acme Release VM.
Here is how the backup and restore job should be placed:
<% mermaid_diagram do %>
graph LR
subgraph Backup Restore VM
database-backup-restorer
bbr-acmedb
end
subgraph Acme VM
acme
bbr-lock-unlock-acme
end
subgraph Acme Release
jobs/acme-job-->acme
jobs/bbr-lock-unlock-acme-->bbr-lock-unlock-acme
jobs/bbr-acmedb-->bbr-acmedb
end
subgraph BBR SDK Release
jobs/database-backup-restorer-->database-backup-restorer
end
classDef lavender fill:#8ca5ce,stroke:#333
classDef turquoise fill:#8bc1ce,stroke:#333
class jobs/acme-job lavender
class jobs/bbr-lock-unlock-acme lavender
class jobs/bbr-acmedb lavender
class jobs/database-backup-restorer turquoise
<% end %>
## <a id='drats'></a>Acceptance tests
The Disaster Recovery Acceptance Test Suite (DRATs) runs against a Cloud
Foundry deployment to ensure that backup and restore works as expected.
For more information and instructions for how to run DRATs, see
[disaster-recovery-acceptance-tests](https://github.com/cloudfoundry-incubator/disaster-recovery-acceptance-tests)
in GitHub.