Skip to content

Commit

Permalink
add support for gcs
Browse files Browse the repository at this point in the history
  • Loading branch information
Simon3 authored and maorfr committed Mar 16, 2019
1 parent fc65d87 commit 4c7e401
Show file tree
Hide file tree
Showing 10 changed files with 393 additions and 12 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,7 @@ test/
# dep/glide
vendor/
bin/

# IntelliJ IDEA
.idea/
skbn.iml
155 changes: 149 additions & 6 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions Gopkg.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,14 @@
name = "github.com/aws/aws-sdk-go"
version = "1.16.7"

[[override]]
name = "golang.org/x/oauth2"
revision = "e64efc72b421e893cbf63f17ba2221e7d6d0b0f3"

[[constraint]]
name = "cloud.google.com/go"
version = "0.37.0"

[[constraint]]
name = "github.com/djherbis/buffer"
version = "1.0.0"
Expand Down
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Skbn currently supports the following providers:
* AWS S3
* Minio S3
* Azure Blob Storage
* Google Cloud Storage

## Install

Expand Down Expand Up @@ -67,6 +68,14 @@ skbn cp \
--dst k8s://<namespace>/<podName>/<containerName>/<path>
```

### Copy files from Kubernetes to Google Cloud Storage

```
skbn cp \
--src k8s://<namespace>/<podName>/<containerName>/<path> \
--dst gcs://<bucket>/<path>
```

## Advanced usage

### Copy files from source to destination in parallel
Expand Down Expand Up @@ -166,6 +175,11 @@ In addition, the `AWS_REGION` environment variable should be set (default is `eu

Skbn uses `AZURE_STORAGE_ACCOUNT` and `AZURE_STORAGE_ACCESS_KEY` environment variables for authentication.

### Google Cloud Storage

Skbn uses Google [Application Default Credentials](https://cloud.google.com/docs/authentication/production).
Basically, it will first look for the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. If it is not defined, it will look for the default service account, or throw an error if none is configured.

## Examples

1. [In-cluster example](/examples/in-cluster)
Expand Down
2 changes: 1 addition & 1 deletion cmd/skbn.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"log"
"os"

"github.com/maorfr/skbn/pkg/skbn"
"skbn/pkg/skbn"

"github.com/spf13/cobra"
)
Expand Down
6 changes: 5 additions & 1 deletion examples/code/example.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@ package main

import (
"log"
"time"

"github.com/maorfr/skbn/pkg/skbn"
"skbn/pkg/skbn"
)

func main() {
Expand All @@ -12,7 +13,10 @@ func main() {
parallel := 0 // all at once
bufferSize := 1.0 // 1GB of in memory buffer size

start := time.Now()
if err := skbn.Copy(src, dst, parallel, bufferSize); err != nil {
log.Fatal(err)
}
elapsed := time.Since(start)
log.Printf("Copy execution time: %s", elapsed)
}

0 comments on commit 4c7e401

Please sign in to comment.