Skip to content

Commit

Permalink
Improve README documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
orlangure committed Feb 15, 2020
1 parent 47f559a commit 0c2fd3f
Showing 1 changed file with 62 additions and 8 deletions.
70 changes: 62 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,67 @@
# gompress
CLI tool to compress the contents of AWS S3 bucket.

Until better documentation is available, download the code, build it using Go
(1.13), and run the binary providing all the flags that the binary expects
(source and destination).
Gompress takes files from the original location, compresses them on the fly
using gzip, and uploads to their destination, keeping the original names (only
adding `.gz`).

`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables must be
set, and they should belong to a user with sufficient permissions to list files
in s3 bucket, download them, and upload them to the location provided in
configuration.
Gompress never keeps large files completely in memory, and never writes them to
disk. It streams data from source into destination, adding a compression layer
in the middle.

Pre-compiled binaries, docker image and better docs are coming.
It is recommended to become familiar with [AWS S3
pricing](https://aws.amazon.com/s3/pricing/) before using this utility. Moving
files around costs money. To reduce the cost, use EC2 instances in the same
region as the buckets!

## Usage

### Running in a docker container:

```
docker run --rm \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
orlangure/gompress \
-src-region us-east-1 \
-src-bucket uncompressed-files \
-src-prefix some-folder/and-sub-folder \
-dst-region us-east-1 \
-dst-bucket compressed-files \
-dst-prefix a-folder \
-keep
```

AWS credentials with sufficient permissions to list, get, create and delete
objects are required. AWS key and secret should be provided using [environment
variables](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file).

When `-keep` flag is provided, original files remain in their original
location. Otherwise, when `-keep` flag is not set, original files are removed
once their compressed versions are successfully uploaded to the destination
path.

### Building from source (Go 1.13 is required):

```
$ git clone https://github.com/orlangure/gompress
$ cd gompress
$ go install .
$ gompress
Usage of gompress:
-dst-bucket string
target s3 bucket name
-dst-prefix string
new files will be prefixed with this value
-dst-region string
target region (default "us-east-1")
-keep
set to keep original files (remove by default)
-src-bucket string
source s3 bucket name
-src-prefix string
source file prefix
-src-region string
source region (default "us-east-1")
```

0 comments on commit 0c2fd3f

Please sign in to comment.