Skip to content

Commit

Permalink
Updating bucketclass and backingstore md docsAddition of diagnostics …
Browse files Browse the repository at this point in the history
…option to CLIRevert "Addition of diagnostics option to CLI"This reverts commit 0b85c16.
  • Loading branch information
jeniawhite committed Oct 31, 2019
1 parent 4409106 commit fb4150e
Show file tree
Hide file tree
Showing 2 changed files with 229 additions and 5 deletions.
119 changes: 115 additions & 4 deletions doc/backing-store-crd.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ BackingStore CRD represents a storage target to be used as underlying storage fo
These storage targets are used to store deduped+compressed+encrypted chunks of data (encryption keys are stored separatly).
Backing-stores are referred to by name when defining [BucketClass](bucket-class-crd.md).

Multiple types of backing-stores are supported: aws-s3, s3-compatible, google-cloud-storage, azure-blob, obc, pvc.
Multiple types of backing-stores are currently supported: aws-s3, s3-compatible, google-cloud-storage, azure-blob.
Backing-store type pv-pool is not yet supported by the operator. Instead, the web management console must be used to administer pv-pool backing-stores.
Adding support for a new type of backing-store is rather easy as it requires just GET/PUT key-value store, see [Backing-stores supported by NooBaa](https://github.com/noobaa/noobaa-core/tree/master/src/agent/block_store_services).


Expand All @@ -17,11 +18,23 @@ Adding support for a new type of backing-store is rather easy as it requires jus

# Reconcile

#### OBC type
#### AWS-S3 type

The operator will create a claim and the appropriate provisioner will create a new bucket or connect to existing one depending on the obc options. Once the claim is ready its details will be used to configure a cloud resource in NooBaa.
Create a cloud storage agent within the NooBaa brain and use S3 API for storing encrypted chunks of data in the AWS cloud.

#### PVC type
#### S3-COMPATIBLE type

Create a cloud storage agent within the NooBaa brain and use S3 API for storing encrypted chunks of data in any S3 API compatible endpoint.

#### GOOGLE-CLOUD-STORAGE type

Create a cloud storage agent within the NooBaa brain and use Google Cloud Storage API for storing encrypted chunks of data in Google Cloud Storage.

#### AZURE-BLOB type

Create a cloud storage agent within the NooBaa brain and use BLOB API for storing encrypted chunks of data in Azure cloud.

#### PV-POOL type (Not yet implemented)

Create a NooBaa storage agent StatefulSet with PVC mounted in each pod. Each agent will connect to the NooBaa brain and provide the PV filesystem storage to be used for storing encrypted chunks of data.

Expand Down Expand Up @@ -80,3 +93,101 @@ status:
lastTime: "2019-06-04T13:05:35.473Z"
troubleshooting: "https://github.com/noobaa/noobaa-core/wiki/Backing-store-finalizer-troubleshooting"
```
# Example
Here are some examples of the cli usage and BackingStore CRs for the different backing-store types:
AWS-S3-TYPE:
```shell
nb -n noobaa backingstore create aws-s3 bs --access-key KEY --secret-key SECRET --target-bucket BUCKET
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
finalizers:
- noobaa.io/finalizer
labels:
app: noobaa
name: bs
namespace: noobaa
spec:
awsS3:
secret:
name: backing-store-aws-s3-bs
namespace: noobaa
targetBucket: BUCKET
type: aws-s3
```
AZURE-BLOB-TYPE:
```shell
nb -n noobaa backingstore create azure-blob bs --account-key KEY --account-name NAME --target-blob-container bs
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
finalizers:
- noobaa.io/finalizer
labels:
app: noobaa
name: bs
namespace: noobaa
spec:
azureBlob:
secret:
name: backing-store-azure-blob-bs
namespace: noobaa
targetBlobContainer: bs
type: azure-blob
```
GOOGLE-CLOUD-STORAGE-TYPE:
```shell
nb -n noobaa backingstore create google-cloud-storage bs --private-key-json-file key.json --target-bucket bs
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
finalizers:
- noobaa.io/finalizer
labels:
app: noobaa
name: bs
namespace: noobaa
spec:
googleCloudStorage:
secret:
name: backing-store-google-cloud-storage-bs
namespace: noobaa
targetBucket: bs
type: google-cloud-storage
```
S3-COMPATIBLE-TYPE:
```shell
nb -n noobaa backingstore create s3-compatible bs --endpoint ENDPOINT --signature-version v4 --access-key KEY --secret-key SECRET --target-bucket bs
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BackingStore
metadata:
finalizers:
- noobaa.io/finalizer
labels:
app: noobaa
name: bs
namespace: noobaa
spec:
s3Compatible:
endpoint: ENDPOINT
secret:
name: backing-store-s3-compatible-bs
namespace: noobaa
signatureVersion: v4
targetBucket: bs
type: s3-compatible
```
115 changes: 114 additions & 1 deletion doc/bucket-class-crd.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@ NooBaaBucket CRD represents a class for buckets that defines policies for data p

Data placement capabilities are built as a multi-layer structure, here are the layers bottom-up:
- Spread Layer - list of backing-stores, aggregates the storage of multiple stores.
- Mirroring Layer - list of spread-layers, async-mirroring to all mirrors, with locality optimization.
- Mirroring Layer - list of spread-layers, async-mirroring to all mirrors, with locality optimization (mirroring requires at least two backing-stores).
- Tiering Layer - list of mirroring-layers, push cold data to next tier.

Constraints:
A backing-store name may appear in more than one bucket-class but may not appear more than once in a single bucket-class.
The operator cli currently only supports a single tier placement-policy for a bucket-class.
YAML must be used to create a bucket-class with a placement-policy that has multiple tiers.

For more information on using bucket-classes from S3 see [S3 Account](s3-account.md).

# Definitions
Expand Down Expand Up @@ -72,3 +77,111 @@ status:
lastTime: "2019-06-04T13:05:35.473Z"
troubleshooting: "https://github.com/noobaa/noobaa-core/wiki/Bucket-class-finalizer-troubleshooting"
```
# Example
Here are some examples of the cli/YAML usage and BucketClass CRs for the different bucket-class configurations:
Single tier, single backing-store, placement Spread:
```shell
nb -n noobaa bucketclass create bc --backingstores bs --placement Spread
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
labels:
app: noobaa
name: bc
namespace: noobaa
spec:
placementPolicy:
tiers:
- backingStores:
- bs
placement: Spread
```
Single tier, two backing-stores, placement Spread:
```shell
nb -n noobaa bucketclass create bc --backingstores bs1,bs2 --placement Spread
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
labels:
app: noobaa
name: bc
namespace: noobaa
spec:
placementPolicy:
tiers:
- backingStores:
- bs1
- bs2
placement: Spread
```
Single tier, two backing-store, placement Mirror:
```shell
nb -n jenia bucketclass create bc --backingstores bs1,bs2 --placement Mirror
```
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
labels:
app: noobaa
name: bc
namespace: noobaa
spec:
placementPolicy:
tiers:
- backingStores:
- bs1
- bs2
placement: Mirror
```
Two tiers (not yet supported via operator cli), single backing-store per tier, placement Spread in tiers:
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
labels:
app: noobaa
name: bc
namespace: noobaa
spec:
placementPolicy:
tiers:
- backingStores:
- bs1
placement: Spread
- backingStores:
- bs2
placement: Spread
```
Two tiers (not yet supported via operator cli), two backing-store per tier, placement Spread in first tier and Mirror in second tier:
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
labels:
app: noobaa
name: bc
namespace: noobaa
spec:
placementPolicy:
tiers:
- backingStores:
- bs1
- bs2
placement: Spread
- backingStores:
- bs3
- bs4
placement: Mirror
```

0 comments on commit fb4150e

Please sign in to comment.