Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

web: Implement api/web server for restic (WIP/PoC) #1963

Open
wants to merge 13 commits into
base: master
from

Conversation

Projects
None yet
7 participants
@kitone
Copy link
Contributor

kitone commented Aug 19, 2018

What is the purpose of this change? What does it change?

It's an early status / proof of concept, but I want to have feedback as soon a possible.

To work and prototype fast (and maybe ugly ;), the views use bootstrap and fancytree, the treeview is lazy load using the /api.

You can access to the webserver with :
(need to be in the restic project path for templates,js,css)
./restic web
http://localhost:8080/web/

json api respond to :

http://localhost:8080/api/snapshots/
http://localhost:8080/api/snapshots/{snapshot_short_id}/nodes/
http://localhost:8080/api/snapshots/{snapshot_short_id}/nodes/{subtree}

My point is not de build a web ui to manage all restic functions, but the idea is to be able to navigate, stats, browse and do quick restore / download from a web page.

I think having endpoint api build in restic would make this easier (also for other building an ui).

@fd0, somes questions :

  • First of all, does it make sense ? :)
  • Can we have the stats on snapshot without having to run / always compute (restic stats) ? Is this save in the repository ?
  • Why a Node doesn't have an id/key to identify them ? It would be easer to ask for a node by key/id and rebuild the content then download the content from a webpage ?

Best regards,
kitone

Was the change discussed in an issue or in the forum before?

Found issue #60 : Implement web server to view backups

Checklist

  • I have read the Contribution Guidelines
  • I have added tests for all changes in this PR
  • I have added documentation for the changes (in the manual)
  • There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
  • I have run gofmt on the code in all commits
  • All commit messages are formatted in the same style as the other commits in the repo
  • I'm done, this Pull Request is ready for review
// }
},
postProcess: function(event, data) {
data.result = convertData(data.response);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'convertData' is not defined.

$tdList = $(node.tr).find(">td");

$tdList.eq(2).text(node.data.user);
$tdList.eq(3).text(node.data.group);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'$tdList' is not defined.


$tdList = $(node.tr).find(">td");

$tdList.eq(2).text(node.data.user);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'$tdList' is not defined.

renderColumns: function(event, data) {
var node = data.node;

$tdList = $(node.tr).find(">td");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'$tdList' is not defined.

@@ -0,0 +1,73 @@
function readyFn(jQuery) {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'jQuery' is defined but never used.

$tdList.eq(2).text(node.data.user);
$tdList.eq(3).text(node.data.group);
// mtime
$tdList.eq(4).text(human_date(node.data.mtime));

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

Identifier 'human_date' is not in camel case.
'$tdList' is not defined.
'human_date' is not defined.

var snapshot_id = node.tree.data["snapshot-id"];
var node_key = node.key;
data.result = {
url: "/api/snapshots/" + snapshot_id + "/nodes/" + node_key

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

Identifier 'snapshot_id' is not in camel case.
Identifier 'node_key' is not in camel case.

lazyLoad: function(event, data) {
var node = data.node;
var snapshot_id = node.tree.data["snapshot-id"];
var node_key = node.key;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

Identifier 'node_key' is not in camel case.

},
lazyLoad: function(event, data) {
var node = data.node;
var snapshot_id = node.tree.data["snapshot-id"];

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

Identifier 'snapshot_id' is not in camel case.

$tdList = $(data.node.tr).find(">td");

$tdList.eq(2).text(n.user);
$tdList.eq(3).text(n.group);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 19, 2018

'$tdList' is not defined.

@gbolo

This comment has been minimized.

Copy link

gbolo commented Aug 19, 2018

Hi @kitone

The POC appears to be functional (but not buildable form the make file). Nice to see a GUI for snapshots. Is the intention to also be able to download these files? Good job so far!

I would like to see the following features:

  • implemented a client-side search for files
  • TLS for http server
  • Some authentication (like basic auth and/or oauth2)
var link = document.createElement("a");
link.text = "Download";
link.href = `/web/snapshots/${snapshot_id}/download?path=${path}`;
$tdList.eq(6).append(link);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Aug 20, 2018

'$tdList' is not defined.

@kitone

This comment has been minimized.

Copy link
Contributor Author

kitone commented Aug 20, 2018

Hi @gbolo,

Just add a way to download a file.

@fd0

This comment has been minimized.

Copy link
Member

fd0 commented Sep 2, 2018

Hi @kitone! Thank you very much for your contribution!

I'm sorry about the rather long delay, restic is a fun project that I'm doing in my spare time, and I the last few weeks have been very busy real-life wise.

First of all, does it make sense ? :)

Very much, at least for the use case for restoring/accessing files. The backup use case is very different and needs further thought, I'd like to not do that right now

So as long as the web UI you're building is restricted to restoring/accessing files you have my full support :)

Can we have the stats on snapshot without having to run / always compute (restic stats) ? Is this save in the repository ?

That depends on the data that is to be queried: the restore size probably won't change, but is not saved anywhere, so for now it needs to be re-computed.

Why a Node doesn't have an id/key to identify them ? It would be easer to ask for a node by key/id and rebuild the content then download the content from a webpage

Did you find the design document yet? It describes the data we store in the repo, after reading it you should have a good understanding of what's available. Only the data structures that have been saved to the repo have an ID, and since we don't save every single node by itself, only trees (collections of nodes) have IDs.

I'll try to have a look at the code shortly. Please let me know if there's anything you need! And don't hesitate to bug me here several times, GitHub issues get complicated/forgotten/convoluted very fast. I'm trying to keep up, but the project is so big already that I sometimes get lost and forget things. ;)

$(msg).addClass("alert alert-danger");
$(msg).text(data.responseText);
submit_button.attr("disabled", false);
submit_button.val("Restore");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

$(msg).removeClass("alert-success");
$(msg).addClass("alert alert-danger");
$(msg).text(data.responseText);
submit_button.attr("disabled", false);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

$(msg).addClass("alert alert-success");
$(msg).text(response);
submit_button.attr("disabled", false);
submit_button.val("Restore");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

$(msg).removeClass("alert-danger");
$(msg).addClass("alert alert-success");
$(msg).text(response);
submit_button.attr("disabled", false);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

$.ajax({
type: "POST",
url: restore_api,
data: restore_data

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'restore_data' is not in camel case.

// Submit the form using AJAX.
$.ajax({
type: "POST",
url: restore_api,

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'restore_api' is not in camel case.

});
var submit_button = $("input[name=submit]");
submit_button.attr("disabled", true);
submit_button.val("In Progress...");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

files: files_to_restore()
});
var submit_button = $("input[name=submit]");
submit_button.attr("disabled", true);

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

target: $("input[name=restore_target]").val(),
files: files_to_restore()
});
var submit_button = $("input[name=submit]");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'submit_button' is not in camel case.

var restore_api = $(this).attr("action");
var restore_data = JSON.stringify({
target: $("input[name=restore_target]").val(),
files: files_to_restore()

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 2, 2018

Identifier 'files_to_restore' is not in camel case.

@kitone

This comment has been minimized.

Copy link
Contributor Author

kitone commented Sep 2, 2018

Hi! Don't worry, I just come back from vacation three day ago! Take your time, I know what it is.

I just push some commit with cleanup and the ability to restore selected files locally.

<td>${v.username}</td>
<td>${v.hostname}</td>
<td>${arrayToDiv(v.paths)}</td>
<td>${arrayToDiv(v.tags)}</td>

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'arrayToDiv' is not defined.

<td>${humanDate(v.time)}</td>
<td>${v.username}</td>
<td>${v.hostname}</td>
<td>${arrayToDiv(v.paths)}</td>

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'arrayToDiv' is not defined.

<td>
<a href="/web/snapshots/${v.short_id}/nodes/">${v.short_id}<a>
</td>
<td>${humanDate(v.time)}</td>

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'humanDate' is not defined.

$("#table-snapshots tbody").append(
`<tr>
<td>
<a href="/web/snapshots/${v.short_id}/nodes/">${v.short_id}<a>

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'short_id' is not in camel case.

// console.log(data);
$.each(data, (k, v) => {
$("#table-snapshots tbody").append(
`<tr>

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'template literal syntax' is only available in ES6 (use 'esversion: 6').

submitButton.val("Restore");
})
.fail(data => {
const msg = $("#form-messages");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'const' is available in ES6 (use 'esversion: 6') or Mozilla JS extensions (use moz).

submitButton.attr("disabled", false);
submitButton.val("Restore");
})
.fail(data => {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'arrow function syntax (=>)' is only available in ES6 (use 'esversion: 6').

data: restoreData
})
.done(response => {
const msg = $("#form-messages");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'const' is available in ES6 (use 'esversion: 6') or Mozilla JS extensions (use moz).

url: restoreApi,
data: restoreData
})
.done(response => {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'arrow function syntax (=>)' is only available in ES6 (use 'esversion: 6').

target: $("input[name=restore_target]").val(),
files: files_to_restore()
});
const submitButton = $("input[name=submit]");

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'const' is available in ES6 (use 'esversion: 6') or Mozilla JS extensions (use moz).

const restoreApi = $(this).attr("action");
const restoreData = JSON.stringify({
target: $("input[name=restore_target]").val(),
files: files_to_restore()

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'files_to_restore' is not in camel case.

@@ -0,0 +1,54 @@
/*jshint esversion: 6 */
function files_to_restore() {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'files_to_restore' is not in camel case.

td.eq(4).text(humanDate(n.mtime));
// size
if (n.hasOwnProperty("size")) {
td.eq(5).text(humanFileSize(n.size));

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'humanFileSize' is not defined.

td.eq(2).text(n.user);
td.eq(3).text(n.group);
// mtime
td.eq(4).text(humanDate(n.mtime));

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'humanDate' is not defined.

}
return c;
});
return nodes;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'nodes' is not defined.

/*jshint esversion: 6 */
function convertData(data) {
// work with the childs "nodes"
nodes = data.nodes;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'nodes' is not defined.

return `${pathFromRoot(node.parent)}/${node.data.name}`;
}

function readyFn(jQuery) {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'jQuery' is defined but never used.

const snapshot_id = data.tree.data["snapshotId"];
const link = document.createElement("a");
link.text = "Download";
link.href = `/web/snapshots/${snapshot_id}/download?path=${path}`;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'snapshot_id' is not in camel case.

}
// actions
if (data.node.type !== "dir") {
const snapshot_id = data.tree.data["snapshotId"];

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'snapshot_id' is not in camel case.
['snapshotId'] is better written in dot notation.

const snapshot_id = node.tree.data["snapshot-id"];
const node_key = node.data.subtree;
data.result = {
url: `/api/snapshots/${snapshot_id}/nodes/${node_key}`

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'snapshot_id' is not in camel case.
Identifier 'node_key' is not in camel case.

lazyLoad(event, data) {
const node = data.node;
const snapshot_id = node.tree.data["snapshot-id"];
const node_key = node.data.subtree;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'node_key' is not in camel case.

},
lazyLoad(event, data) {
const node = data.node;
const snapshot_id = node.tree.data["snapshot-id"];

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'snapshot_id' is not in camel case.

// delete unused attributes
delete c.atime;
delete c.ctime;
delete c.device_id;

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

Identifier 'device_id' is not in camel case.

return moment(datetime).format("YYYY/MM/DD HH:MM:SS");
}

function arrayToDiv(arr) {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'arrayToDiv' is defined but never used.

return Math.max(bytes, 0.1).toFixed(1) + byteUnits[i];
}

function humanDate(datetime) {

This comment has been minimized.

Copy link
@houndci-bot

houndci-bot Sep 4, 2018

'humanDate' is defined but never used.

Show resolved Hide resolved templates/web/static/base.js

@kitone kitone force-pushed the kitone:web branch from 6ef0d5d to dd7b4a8 Sep 4, 2018

@McFlat

This comment has been minimized.

Copy link

McFlat commented Oct 28, 2018

I'd recommend a whole other application called something along the lines of restic-web written in go or nodejs and react or angular for the frontend, to be able to do everything you can do from the commandline interface.

Really it's too much to add a web command to restic but if thats what you want, then provide it as a plugin, the plugin being another repo altogether that handles the web stuff for restic, build it with golang or nodejs for the backend and reactjs of angularjs for the frontend.

Just my two cents, please don't use jQuery, as the application infrastructure grows you will see the need for something like angularjs or reactjs or backbonejs for that matter.

@mholt

This comment has been minimized.

Copy link
Contributor

mholt commented Oct 28, 2018

I do agree that while this is nice, this would probably be best as a separate repo/project, not integrated into restic directly. Restic's global --json flag works on almost all the commands now (the last few are in progress) which make automating restic just as easy, without the added complexity of a REST API server (and integrated UI).

@nioncode

This comment has been minimized.

Copy link

nioncode commented Dec 22, 2018

@fd0 Did you already decide on how you'd like to integrate a 'browse' functionality for restic? I.e. either add it to restic's core or implement it as a separate application?

I think it would be great to have it in restic's core similar to the fuse mount to make it easy for users on Windows to browse their repository and restore selected files / folders from an earlier snapshot. I don't know how useful the api parts of this PR are when almost all commands allow returning JSON by now, but the web server would be a great addition at least. If you'd chime in to give a direction where you'd like this PR to go, then I might join @kitone to get this merged.

@nioncode

This comment has been minimized.

Copy link

nioncode commented Jan 1, 2019

After closing #2125, I think an API provided by restic (after being started by restic serve or something) would be great. Then users could query this JSON API and integrate all sorts of things on top (e.g. Browser UI, dokan / fuse mount, native ui, …).
I think the focus at least for now should be to have a working browse + restore functionality, with pruning etc. coming up in a next version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.