Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.go text eol=lf
7 changes: 7 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
118 changes: 118 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
name: Build
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
build-all-clis:
name: Build Only (No Release)
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v5

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: 'go.mod'

- name: Alioss CLI Build for Linux
env:
GOOS: linux
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Alioss CLI for Linux"
go build -o "alioss-cli-linux-amd64" ./alioss
sha1sum "alioss-cli-linux-amd64"

- name: Alioss CLI Build for Windows
env:
GOOS: windows
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Alioss CLI for Windows"
go build -o "alioss-cli-windows-amd64.exe" ./alioss
sha1sum "alioss-cli-windows-amd64.exe"

- name: Azurebs CLI Build for Linux
env:
GOOS: linux
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Azurebs CLI for Linux"
go build -o "azurebs-cli-linux-amd64" ./azurebs
sha1sum "azurebs-cli-linux-amd64"

- name: Azurebs CLI Build for Windows
env:
GOOS: windows
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Azurebs CLI for Windows"
go build -o "azurebs-cli-windows-amd64.exe" ./azurebs
sha1sum "azurebs-cli-windows-amd64.exe"

- name: Dav CLI Build for Linux
env:
GOOS: linux
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Dav CLI for Linux"
go build -o "dav-cli-linux-amd64" ./dav/main
sha1sum "dav-cli-linux-amd64"

- name: Dav CLI Build for Windows
env:
GOOS: windows
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Dav CLI for Windows"
go build -o "dav-cli-windows-amd64.exe" ./dav/main
sha1sum "dav-cli-windows-amd64.exe"

- name: GCS CLI Build for Linux
env:
GOOS: linux
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Gcs CLI for Linux"
go build -o "gcs-cli-linux-amd64" ./gcs
sha1sum "gcs-cli-linux-amd64"

- name: GCS CLI Build for Windows
env:
GOOS: windows
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building Gcs CLI for Windows"
go build -o "gcs-cli-windows-amd64.exe" ./gcs
sha1sum "gcs-cli-windows-amd64.exe"

- name: S3 CLI Build for Linux
env:
GOOS: linux
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building S3 CLI for Linux"
go build -o "s3-cli-linux-amd64" ./s3
sha1sum "s3-cli-linux-amd64"

- name: S3 CLI Build for Windows
env:
GOOS: windows
GOARCH: amd64
CGO_ENABLED: 0
run: |
echo "Building S3 CLI for Windows"
go build -o "s3-cli-windows-amd64.exe" ./s3
sha1sum "s3-cli-windows-amd64.exe"
58 changes: 58 additions & 0 deletions .github/workflows/unit-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name : Unit Test

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
tests:
name: Run Unit Tests
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v5
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
# TODO: Re-enable linting after fixing existing issues on alioss
# - name: Lint code
# uses: golangci/golangci-lint-action@v8
- name: Cache Go modules
uses: actions/cache@v4
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: |
${{ runner.os }}-go-

- name: alioss unit tests
run: |
export CGO_ENABLED=0
go version
go run github.com/onsi/ginkgo/v2/ginkgo --skip-package=integration ./alioss/...

- name: azurebs unit tests
run: |
export CGO_ENABLED=0
go version
go run github.com/onsi/ginkgo/v2/ginkgo --skip-package=integration ./azurebs/...

- name: dav unit tests
run: |
export CGO_ENABLED=0
go version
go test -v ./dav/...

- name: gcs unit tests
run: |
export CGO_ENABLED=0
go version
go run github.com/onsi/ginkgo/v2/ginkgo --skip-package=integration ./gcs/...

- name: s3 unit tests
run: |
export CGO_ENABLED=0
go version
go run github.com/onsi/ginkgo/v2/ginkgo --skip-package=integration ./s3/...
29 changes: 29 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Binaries for programs and plugins
*.exe
*.dll
*.so
*.dylib
*.out
*.bin

# Go coverage files
*.coverprofile
*.cov

# Output folders
/bin/
/build/
/dist/

# Test binaries
*.test

# Dependency / tooling caches
/vendor/
/coverage/
/tmp/
/.cache/

# IDE/editor
.vscode/
.idea/
12 changes: 12 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
version: "2"

linters:
default: standard

settings:
errcheck:
check-blank: true # assignment to blank identifier: `_ := someFunc()`.

formatters:
enable:
- goimports
10 changes: 10 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
Copyright (c) 2022 Cloud Foundry Contributors. All Rights Reserved.

Other copyright notices for portions of this software are listed in the LICENSE file.

This product is licensed to you under the Apache License, Version 2.0 (the "License").
You may not use this product except in compliance with the License.

This product may include a number of subcomponents with separate copyright notices
and license terms. Your use of these subcomponents is subject to the terms and
conditions of the subcomponent's license, as noted in the LICENSE file.
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Storage CLI
This repository consolidates five independent blob-storage CLIs, one per provider, into a single codebase. Each provider has its own dedicated directory (azurebs/, s3/, gcs/, alioss/, dav/), containing an independent main package and implementation. The tools are intentionally maintained as separate binaries, preserving each provider’s native SDK, command-line flags, and operational semantics. Each CLI exposes similar high-level operations (e.g., put, get, delete).

Key points

- Each provider builds independently.

- Client setup, config, and options are contained within the provider’s folder.

- All tools support the same core commands (such as put, get, and delete) for a familiar workflow, while each provider defines its own flags, parameters, and execution flow that align with its native SDK and terminology.

- Central issue tracking, shared CI, and aligned release process without merging implementations.


## Providers
- [Alioss](./alioss/README.md)
- [Azurebs](./azurebs/README.md)
- [Dav](./dav/README.md)
- [Gcs](./gcs/README.md)
- [S3](./s3/README.md)


## Build
Use following command to build it locally

```shell
go build -o <provider-folder-name>/<build-name> <provider-folder-name>/main.go
```
e.g. `go build -o alioss/alioss-cli alioss/main.go`


## Notes
These commit IDs represent the last migration checkpoint from each provider's original repository, marking the final commit that was copied during the consolidation process.

- alioss -> c303a62679ff467ba5012cc1a7ecfb7b6be47ea0
- azurebs -> 18667d2a0b5237c38d053238906b4500cfb82ce8
- dav -> c64e57857539d0173d46e79093c2e998ec71ab63
- gcs -> d4ab2040f37415a559942feb7e264c6b28950f77
- s3 -> 7ac9468ba8567eaf79828f30007c5a44066ef50f
60 changes: 60 additions & 0 deletions alioss/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Ali Storage CLI

The Ali Storage CLI is for uploading, fetching and deleting content to and from an Ali OSS.
It is highly inspired by the https://github.com/cloudfoundry/bosh-s3cli.

## Usage

Given a JSON config file (`config.json`)...

``` json
{
"access_key_id": "<string> (required)",
"access_key_secret": "<string> (required)",
"endpoint": "<string> (required)",
"bucket_name": "<string> (required)"
}
```

``` bash
# Command: "put"
# Upload a blob to the blobstore.
./alioss-cli -c config.json put <path/to/file> <remote-blob>

# Command: "get"
# Fetch a blob from the blobstore.
# Destination file will be overwritten if exists.
./alioss-cli -c config.json get <remote-blob> <path/to/file>

# Command: "delete"
# Remove a blob from the blobstore.
./alioss-cli -c config.json delete <remote-blob>

# Command: "exists"
# Checks if blob exists in the blobstore.
./alioss-cli -c config.json exists <remote-blob>

# Command: "sign"
# Create a self-signed url for a blob in the blobstore.
./alioss-cli -c config.json sign <remote-blob> <get|put> <seconds-to-expiration>
```

### Using signed urls with curl
``` bash
# Uploading a blob:
curl -X PUT -T path/to/file <signed url>

# Downloading a blob:
curl -X GET <signed url>
```
## Running integration tests

To run the integration tests:
- Export the following variables into your environment:
``` bash
export ACCESS_KEY_ID=<your Alibaba access key id>
export ACCESS_KEY_SECRET=<your Alibaba access key secret>
export ENDPOINT=<your Alibaba OSS endpoint>
export BUCKET_NAME=<your Alibaba OSS bucket>
```
- go build && go test ./integration/...
Loading