Skip to content
This repository has been archived by the owner on Aug 25, 2023. It is now read-only.

Commit

Permalink
Merge 4f91383 into b7ffe98
Browse files Browse the repository at this point in the history
  • Loading branch information
@marcin-kolda
marcin-kolda authored Jul 9, 2018
2 parents b7ffe98 + 4f91383 commit 7c8a84f
Show file tree
Hide file tree
Showing 7 changed files with 206 additions and 59 deletions.
27 changes: 5 additions & 22 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -158,28 +158,11 @@ To check the content for given backup __Y__ in Big Query:
1. Select table and check _Schema_, _Details_ or _Preview_ tab.

## How to restore data from backups?
There are several options to restore data, available from _\<your-project-id>_.__appspot.com__ (dropdown tab _Actions_)
* __Restore whole dataset__ (_\<your-project-id>.appspot.com_/__ui/restoreDataset__). Parameters:
* Source project id: id of project where dataset is placed originally,
* Source dataset id: original dataset id,
* Target dataset id (optional): id of temporary dataset that will be used (and created if does not exist) as container for restored table. Remember that this will be a temporary dataset with expiration time set to 7 days. __Note that passed dataset could already exists - it should be in the same location as backup__.
If _target dataset id_ is not passed, then _source dataset id_ value will be used as a target dataset id in restoration project
* Max partition days (optional): number of days from partitioned tables will be restored (eg. 30 means that partitions from last 30 days will be restored),
* __Restore tables from list of backups__ (_\<your-project-id>.appspot.com_/__ui/restoreList__). Parameters:
* Target dataset id (optional): id of temporary dataset that will be used (and created if does not exist) as container for restored table. Remember that this will be a temporary dataset with expiration time set to 7 days. __Note that passed dataset could already exists - it should be in the same location as backup__.
If _target dataset id_ is not passed, then source dataset id value of each backup will be used as a target dataset id in restoration project.
In case of restoring backups from different datasets multiple target datasets will be created.
* Backup list: set of backups in __JSON__ format, each of them is designated by the url safe key of backup entity available from [Datastore](https://console.cloud.google.com/datastore). Example:
```json
[
{
"backupUrlSafeKey" : "ahFlfmRldi1wcm9qZWN0LPPicXIlCxIFVGFibGUYgICAkOaLgAgMCxIGQmFja3VwGICAgICAgJJJJA"
},
{
"backupUrlSafeKey" : "ahFlfmRldi1wcm9qZWN0LWJicXIlCxIFVGFibGUYgICAkJOlgAgMCxIGQmFja3VwGICAgICAgIAKDA"
}
]
```
There are several options to restore data, available from _\<your-project-id>_.__appspot.com__
* __Restore whole dataset__
* __Restore single table__
* __Restore tables from custom list of backups__

#### Checking status of restoration process
Restore process is asynchronous. To check status of process, follow links returned in response:

Expand Down
15 changes: 10 additions & 5 deletions src/main.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,28 @@ def render_response(self, _template, **context):

class MainPage(BaseHandler):
def get(self):
domain = Environment.get_domain(configuration.backup_project_id)
self.render_response('index.html', domain=domain)
self.render_response('index.html')


class RestoreDatasetUIHandler(BaseHandler):
def get(self):
self.render_response('restoreDataset.html')
self.render_response('restoreDataset.html',
restoration_project_id=
configuration.restoration_project_id)


class RestoreListUIHandler(BaseHandler):
def get(self):
self.render_response('restoreList.html')
self.render_response('restoreList.html',
restoration_project_id=
configuration.restoration_project_id)


class RestoreTableUIHandler(BaseHandler):
def get(self):
self.render_response('restoreTable.html')
self.render_response('restoreTable.html',
restoration_project_id=
configuration.restoration_project_id)


app = webapp2.WSGIApplication([
Expand Down
16 changes: 11 additions & 5 deletions templates/base.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,24 @@

<div class="collapse navbar-collapse" id="navbarsExampleDefault">
<ul class="navbar-nav mr-auto">
<li class="nav-item active">
<li class="nav-item">
<a class="nav-link" href="/">Home <span class="sr-only">(current)</span></a>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" id="dropdown01"
<a class="nav-link dropdown-toggle" id="dropdown_actions"
data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Actions</a>
<div class="dropdown-menu" aria-labelledby="dropdown01">
<a class="dropdown-item" href="/ui/restoreDataset">Restore Dataset</a>
<a class="dropdown-item" href="/ui/restoreList">Restore custom backup list</a>
<div class="dropdown-menu" aria-labelledby="dropdown_actions">
<a class="dropdown-item" href="/ui/restoreDataset">Restore dataset</a>
<a class="dropdown-item" href="/ui/restoreTable">Restore table</a>
</div>
</li>
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" id="dropdown_advanced"
data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">Advanced</a>
<div class="dropdown-menu" aria-labelledby="dropdown_advanced">
<a class="dropdown-item" href="/ui/restoreList">Restore custom backup list</a>
</div>
</li>
</ul>
</div>
</nav>
Expand Down
19 changes: 14 additions & 5 deletions templates/index.html
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@
{% extends "base.html" %}
{% block body %}
<h1>Backup Big Query (BBQ)</h1>
<br/>

<h4>What would you like to restore?</h4>

<ul>
<li>
<a href="/ui/restoreDataset">Dataset</a>
</li>
<li>
<a href="/ui/restoreTable">Table</a>
</li>
</ul>

Full documentation is available on <a href="https://github.com/ocadotechnology/bbq/blob/master/README.md" target="_blank">BBQ Github page</a>

<h4>Expand <strong>
<word style="color:red">Actions</word>
</strong>
dropdown to list possible actions.
</h4>
{% endblock %}


Expand Down
59 changes: 53 additions & 6 deletions templates/restoreDataset.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,59 @@ <h2>Restore dataset</h2>


<form name="restoreDatasetForm" onsubmit="return OnSubmitForm();">
Source project id<span style="color: red;">*</span>:<br> <input type="text" name="projectId" value="" required><br>
Source dataset id<span style="color: red;">*</span>:<br> <input type="text" name="datasetId" value="" required><br>
Target dataset id:<br> <input type="text" name="targetDatasetId" value=""><br>
Max partition days:<br> <input type="number" min="1" name="max_partition_days" value=""><br>
<br>
<input type="submit" value="Submit">
<div class="form-group row">
<label for="sourceProjectId" class="col-sm-2 col-form-label">Source project id</label>
<div class="col-sm-8">
<input type="text" class="form-control" id="sourceProjectId" name="projectId"
placeholder="project id" required/>
<small id="sourceProjectIdHelp" class="form-text text-muted">Project id where dataset is
placed originally
</small>
</div>
</div>
<div class="form-group row">
<label for="sourceDatasetId" class="col-sm-2 col-form-label">Source dataset id</label>
<div class="col-sm-8">
<input type="text" class="form-control" id="sourceDatasetId" name="datasetId"
placeholder="dataset id">
<small id="sourceDatasetIdHelp" class="form-text text-muted">Dataset id where tables were
placed originally
</small>
</div>
</div>
<div class="form-group row">
<label for="targetDatasetId" class="col-sm-2 col-form-label">Target dataset id</label>
<div class="input-group col-sm-8">
<div class="input-group-prepend">
<span class="input-group-text" id="projectAddOn">{{ restoration_project_id }}.</span>
</div>
<input type="text" class="form-control" aria-describedby="projectAddOn" id="targetDatasetId"
name="targetDatasetId" placeholder="source_project_id___source_dataset_id">
<small id="targetDatasetIdHelp" class="form-text text-muted">Optional. Temporary dataset id
that will be used (and created if does not exist) as container for restored table. Remember
that this will be a temporary dataset with expiration time set to 7 days. Note that passed
dataset could already exists - it should be in the same location as backup. If target
dataset id is not passed, then source dataset id value will be used as a target dataset id
in restoration project. In case of restoring backups from different datasets multiple target
datasets will be created.
</small>
</div>
</div>
<div class="form-group row">
<label for="maxPartitionsDays" class="col-sm-2 col-form-label">Max partition days</label>
<div class="col-sm-4">
<input type="number" class="form-control" id="maxPartitionsDays" name="max_partition_days">
<small id="maxPartitionsDaysHelp" class="form-text text-muted">Optional. Number of days from
partitioned tables will be restored (eg. 30 means that partitions from last 30 days from now
will be restored)
</small>
</div>
</div>
<div class="form-group row">
<div class="col-sm-4">
<button type="submit" class="btn btn-primary">Restore dataset</button>
</div>
</div>
</form>

<br><strong>Response:</strong> <br>
Expand Down
45 changes: 38 additions & 7 deletions templates/restoreList.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,51 @@ <h2>Restore custom backup list</h2>
}
</script>


<form name="restoreListForm" onsubmit="return OnSubmitForm();">
Target dataset id (optional):<br> <input type="text" name="targetDatasetId" value=""><br>
Backup list<span style="color: red;">*</span>:<br><textarea name="requestBody" placeholder='
[
<div class="form-group row">
<label for="targetDatasetId" class="col-sm-2 col-form-label">Target dataset id</label>
<div class="input-group col-sm-8">
<div class="input-group-prepend">
<span class="input-group-text" id="projectAddOn">{{ restoration_project_id }}.</span>
</div>
<input type="text" class="form-control" aria-describedby="projectAddOn" id="targetDatasetId"
name="targetDatasetId" placeholder="source_project_id___source_dataset_id">
<small id="targetDatasetIdHelp" class="form-text text-muted">Optional. Temporary dataset id
that will be used (and created if does not exist) as container for restored table. Remember
that this will be a temporary dataset with expiration time set to 7 days. Note that passed
dataset could already exists - it should be in the same location as backup. If target
dataset id is not passed, then source dataset id value will be used as a target dataset id
in restoration project
</small>
</div>
</div>
<div class="form-group row">
<label for="requestBody" class="col-sm-2 col-form-label">Backups list</label>
<div class="input-group col-sm-8">
<textarea class="form-control" id="requestBody" name="requestBody" rows="20" cols="100" required>[
{
"backupUrlSafeKey" : "ahFlfmRldi1wcm9qZWN0LPPicXIlCxIFVGFibGUYgICAkOaLgAgMCxIGQmFja3VwGICAgICAgJJJJA"
},
{
"backupUrlSafeKey" : "ahFlfmRldi1wcm9qZWN0LWJicXIlCxIFVGFibGUYgICAkJOlgAgMCxIGQmFja3VwGICAgICAgIAKDA"
}
]' rows="50" cols="100" required></textarea>
<br>
<input type="submit" value="Submit">
]
</textarea>
</div>
</div>
<div class="form-group row">
<label class="col-sm-2 col-form-label"></label>
<div class="input-group col-sm-8">
<small id="requestBodyHelp" class="form-text text-muted">backupUrlSafeKey is a "URL-safe key" value taken
from <a href="https://console.cloud.google.com/datastore/entities/query?kind=Backup">Cloud Datastore Backup entities</a>
</small>
</div>
</div>
<div class="form-group row">
<div class="col-sm-4">
<button type="submit" class="btn btn-primary">Restore list of backups</button>
</div>
</div>
</form>

<br><strong>Response:</strong> <br>
Expand Down
84 changes: 75 additions & 9 deletions templates/restoreTable.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,82 @@ <h2>Restore table</h2>
}
</script>

<p class="lead">
This form restores single non-partitioned table or single partition. In case of restoring multiple
partitions from partitioned table, you need to submit this form for every partition which you want
to restore.
</p>

<form name="restoreTableForm" onsubmit="return OnSubmitForm();">
Source project id<span style="color: red;">*</span>:<br><input type="text" name="projectId" value="" required><br>
Source dataset id<span style="color: red;">*</span>:<br><input type="text" name="datasetId" value="" required><br>
Source table id<span style="color: red;">*</span>:<br><input type="text" name="tableId" value="" required><br>
Source table partition id:<br><input type="number" min="9999999" max="100000000"
name="partitionId" value=""><br>
Target dataset id:<br><input type="text" name="targetDatasetId" value=""><br>
Restoration date:<br><input type="date" name="restorationDate" value=""><br>
<br>
<input type="submit" value="Submit">
<div class="form-group row">
<label for="sourceProjectId" class="col-sm-2 col-form-label">Source project id</label>
<div class="col-sm-8">
<input type="text" class="form-control" id="sourceProjectId" name="projectId"
placeholder="project id" required/>
<small id="sourceProjectIdHelp" class="form-text text-muted">Project id where dataset is
placed originally
</small>
</div>
</div>
<div class="form-group row">
<label for="sourceDatasetId" class="col-sm-2 col-form-label">Source dataset id</label>
<div class="col-sm-8">
<input type="text" class="form-control" id="sourceDatasetId" name="datasetId"
placeholder="dataset id">
<small id="sourceDatasetIdHelp" class="form-text text-muted">Dataset id where tables were
placed originally
</small>
</div>
</div>
<div class="form-group row">
<label for="sourceTableId" class="col-sm-2 col-form-label">Source table id</label>
<div class="col-sm-8">
<input type="text" class="form-control" id="sourceTableId" name="tableId"
placeholder="table id">
<small id="sourceTableIdHelp" class="form-text text-muted">Table id of original table</small>
</div>
</div>
<div class="form-group row">
<label for="sourcePartitionId" class="col-sm-2 col-form-label">Source partition id</label>
<div class="col-sm-8">
<input type="number" class="form-control" id="sourcePartitionId" name="partitionId"
placeholder="partition id">
<small id="sourcePartitionIdHelp" class="form-text text-muted">Valid and required for
partitioned tables only. Partition id of original table. If specified, only this single
partition will be restored
</small>
</div>
</div>
<div class="form-group row">
<label for="restorationDate" class="col-sm-2 col-form-label">Restoration date</label>
<div class="col-sm-4">
<input type="date" class="form-control" id="restorationDate" name="restorationDate">
<small id="restorationDateHelp" class="form-text text-muted">Optional. BBQ will restore the
most recent version of backup on given date. Default value is today</small>
</div>
</div>
<div class="form-group row">
<label for="targetDatasetId" class="col-sm-2 col-form-label">Target dataset id</label>
<div class="input-group col-sm-8">
<div class="input-group-prepend">
<span class="input-group-text" id="projectAddOn">{{ restoration_project_id }}.</span>
</div>
<input type="text" class="form-control" aria-describedby="projectAddOn" id="targetDatasetId"
name="targetDatasetId" placeholder="source_project_id___source_dataset_id">
<small id="targetDatasetIdHelp" class="form-text text-muted">Optional. Temporary dataset id
that will be used (and created if does not exist) as container for restored table. Remember
that this will be a temporary dataset with expiration time set to 7 days. Note that passed
dataset could already exists - it should be in the same location as backup. If target
dataset id is not passed, then source dataset id value will be used as a target dataset id
in restoration project
</small>
</div>
</div>
<div class="form-group row">
<div class="col-sm-4">
<button type="submit" class="btn btn-primary">Restore table</button>
</div>
</div>
</form>

<br><strong>Response:</strong> <br>
Expand Down

0 comments on commit 7c8a84f

Please sign in to comment.