Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Backblaze B2 as backend #125

Closed
tamalsaha opened this issue Jul 3, 2017 · 12 comments

Comments

@tamalsaha
Copy link
Contributor

commented Jul 3, 2017

If there are enough user request, we should add this.

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2018

First thing I checked for when I was point at stash 😄

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 2, 2018

@whereisaaron, will you help us test this? I don't have a B2 account. The code is mostly there.

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2018

Yes, love too @tamalsaha . Can likely test this week and next, as I am testing some new automated cluster deployments (k8s 1.8). Point me at a test image ideally.
B2 accounts are free for the first 1 Gb and no credit card required to sign up. I think you just need to verify a phone number. But I'll be happy to test too. https://www.backblaze.com/b2/sign-up.html

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 3, 2018

Can you try the new 0.6.0 release? Please check the instructions here: https://appscode.com/products/stash/0.6.0/guides/backends/

Please file issues, if you see any problems.

@tamalsaha tamalsaha closed this Jan 3, 2018

@tamalsaha tamalsaha added this to the 0.6.0 milestone Jan 3, 2018

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

Hi @tamalsaha, I am testing 0.6.1 with B2 on a k8s 1.8 cluster with RBAC.

Unfortunately the RBAC permission in the official/stable Helm chart don't see sufficient. See operator log below.

I0104 23:54:30.310281       1 logs.go:19] registry.ping url=https://registry-1.docker.io/v2/
I0104 23:54:30.522037       1 logs.go:19] registry.manifest.get url=https://registry-1.docker.io/v2/appscode/stash/manifests/0.6.1 repository=appscode/stash reference=0.6.1
W0104 23:54:30.742410       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0104 23:54:30.745102       1 logs.go:19] registry.ping url=https://registry-1.docker.io/v2/
I0104 23:54:30.866475       1 logs.go:19] registry.manifest.get url=https://registry-1.docker.io/v2/appscode/kubectl/manifests/1.8.0 repository=appscode/kubectl reference=1.8.0
F0104 23:54:34.104993       1 run.go:73] clusterroles.rbac.authorization.k8s.io "stash-sidecar" is forbidden: User "system:serviceaccount:backup:stash-backup" cannot get clusterroles.rbac.authorization.k8s.io at the cluster scope

This is what the Helm chart deploys for RBAC. Can you spot what is missing? Seems like it is after clusterroles.rbac.authorization.k8s.io but dosn't grant that to itself?

# Source: stash/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: stash-backup
  labels:
    chart: "stash-0.2.0"
    app: "stash"
    heritage: "Tiller"
    release: "stash-backup"
---
# Source: stash/templates/cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: stash-backup
  labels:
    chart: "stash-0.2.0"
    app: "stash"
    heritage: "Tiller"
    release: "stash-backup"
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - "*"
- apiGroups:
  - extensions
  resources:
  - thirdpartyresources
  verbs:
  - "*"
- apiGroups:
  - stash.appscode.com
  resources: ["*"]
  verbs: ["*"]
- apiGroups:
  - extensions
  resources:
  - replicasets
  - daemonsets
  verbs: ["get", "list", "watch", "patch"]
- apiGroups:
  - apps
  resources:
  - deployments
  verbs: ["get", "list", "watch", "patch"]
- apiGroups: [""]
  resources:
  - namespaces
  - replicationcontrollers
  verbs: ["get", "list", "watch", "patch"]
- apiGroups: [""]
  resources:
  - secrets
  verbs: ["get"]
- apiGroups: [""]
  resources:
  - events
  verbs: ["create"]
- apiGroups: [""]
  resources:
  - pods
  verbs: ["list", delete"]
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs: ["get", "create", "delete", "patch"]
---
# Source: stash/templates/cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: stash-backup
  labels:
    chart: "stash-0.2.0"
    app: "stash"
    heritage: "Tiller"
    release: "stash-backup"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: stash-backup
subjects:
- kind: ServiceAccount
  name: stash-backup
  namespace: backup
@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 5, 2018

The official /stable chart has not been published after the release. Can you please try the chart in this repo? https://github.com/appscode/stash/tree/master/chart/stable/stash

@tamalsaha tamalsaha reopened this Jan 5, 2018

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

Will do. That one does look better.

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 5, 2018

helm/charts#3235

fyi.

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

Hi @tamalsaha, the RBAC in that Helm chart worked great thanks.

The B2 backend also worked fine for the demo backup. Yay!

I'll use it more the next week and try some restores. I guess you can close this one as a success and I'll report any issues I hit. Thanks for your work to add B2.

A minor gripe is few unwanted line-breaks in the logging, that's going to get all separated and messed up in CloudWatch / Elastisearch, but possibly this is the restic client's problem? It needs a non-interactive output mode!

I0105 03:07:55.736401       1 backup.go:150] Found restic stash-demo
I0105 03:07:55.740346       1 backup.go:158] Found repository secret b2-secret
[golang-sh]$ /bin/restic snapshots --json --cache-dir /tmp/restic-cache
unable to open config file: Stat: b2_download_file_by_name: 404: bucket my-cluster-backup does not have file: demo/deployment/stash-demo/config
Is there a repository at the following location?
b2:my-cluster-backup:/demo/deployment/stash-demo
[golang-sh]$ /bin/restic init --cache-dir /tmp/restic-cache
created restic backend 034eb17d56 at b2:my-cluster-backup:/demo/deployment/stash-demo

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
I0105 03:08:09.102481       1 scheduler.go:92] Starting Stash backup
I0105 03:08:09.102775       1 reflector.go:202] Starting reflector *v1alpha1.Restic (5m0s) from github.com/appscode/stash/pkg/backup/scheduler.go:94
I0105 03:08:09.102806       1 reflector.go:240] Listing and watching *v1alpha1.Restic from github.com/appscode/stash/pkg/backup/scheduler.go:94
I0105 03:08:09.202752       1 restic.go:140] Sync/Add/Update for Restic stash-demo
I0105 03:11:09.000363       1 scheduler.go:131] Acquired lock for Restic default/stash-demo
[golang-sh]$ /bin/restic snapshots --json --cache-dir /tmp/restic-cache
null
[golang-sh]$ /bin/restic backup /source/data --force --hostname stash-demo --cache-dir /tmp/restic-cache
scan [/source/data]
[0:00] 26 directories, 30 files, 441.151 KiB
scanned 26 directories, 30 files in 0:00
[0:07] 100.00%  0B/s  441.151 KiB / 441.151 KiB  56 / 56 items  0 errors  ETA 0:00

duration: 0:07, 0.06MiB/s
snapshot 81d1d9d9 saved
I0105 03:11:28.138414       1 recorder.go:79] Event created:  stash-demo.1506cb546d2d637c
[golang-sh]$ /bin/restic forget --keep-last 5 --prune --cache-dir /tmp/restic-cache
snapshots for (host [stash-demo], paths [/source/data]):

keep 1 snapshots:
ID        Date                 Host        Tags        Directory
----------------------------------------------------------------------
81d1d9d9  2018-01-05 03:11:20  stash-demo              /source/data
----------------------------------------------------------------------
1 snapshots

The backup snapshot file in B2:
image

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 5, 2018

I think we can just pass restic output to /dev/null. To show showing restic commands, we need to set ShowCMD = false https://github.com/appscode/stash/blob/321d741480afa422db8c9a07cfbbd863d51bca18/pkg/cli/restic.go#L31

@whereisaaron

This comment has been minimized.

Copy link
Contributor

commented Jan 5, 2018

It's a hard choice @tamalsaha , because I do like seeing the stats about files, size, and upload time. And if an error occurs you'd want to see the stderr output from restic. Maybe the command line echo and restic stdout output could be tied to the next up '-v' log level.

@tamalsaha

This comment has been minimized.

Copy link
Contributor Author

commented Jan 5, 2018

yeah! Btw, please file issues / features requests for any stash stuff as you try it. We would love to get any feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.