gocacheprog-s3

gocacheprog-s3 is a "cacheprog" for Go that uses S3.

Usage

As of writing (Apr 2024), GOCACHEPROG support requires a custom build of the Go toolchain. As such, we provide a Docker image with both a custom build of the Go toolchain and the gocacheprog-s3 binary baked in.

ghcr.io/nfi-hashicorp/gocacheprog-s3:latest-go1.22.2

Docker example

% docker run --platform=linux/amd64 -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e AWS_REGION -e GOCACHEPROG="gocacheprog-s3 -v=2 mycoolbucket" ghcr.io/nfi-hashicorp/gocacheprog-s3:go1.22.2-latest go install github.com/nfi-hashicorp/gocacheprog-s3@latest
...
disk stats: 
413 gets: 0 hits, 413 misses, 0 errors, 0s total dur
901 puts: 0 errors, 0s total dur
s3 stats: 
414 gets: 1 hits, 413 misses, 0 errors, 0s total dur; total 0.00 MB; avg 0.00 MB/s
902 puts: 0 errors, 1m17.3s total dur; total 127.59 MB; avg 1.65 MB/s

And then again to see the speed up.

% docker run --platform=linux/amd64 -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_SESSION_TOKEN -e AWS_REGION -e GOCACHEPROG="gocacheprog-s3 -v=2 mycoolbucket" ghcr.io/nfi-hashicorp/gocacheprog-s3:go1.22.2-latest go install github.com/nfi-hashicorp/gocacheprog-s3@latest
...
disk stats: 
642 gets: 0 hits, 642 misses, 0 errors, 0s total dur
642 puts: 0 errors, 0s total dur
s3 stats: 
643 gets: 642 hits, 1 misses, 0 errors, 34.8s total dur; total 127.53 MB; avg 3.66 MB/s
2 puts: 0 errors, 300ms total dur; total 0.00 MB; avg 0.00 MB/s

Local dev example

% GOCACHEPROG="./gocacheprog-s3 -v=2 -local-cache-dir=go-cache $BUCKET"
% aws s3 rm --recursive s3://$BUCKET/go-cache > /dev/null 2>&1
% rm -rf go-cache
% go build .
disk stats: 
413 gets: 0 hits, 413 misses, 0 errors, 0s total dur
893 puts: 0 errors, 0s total dur
s3 stats: 
414 gets: 1 hits, 413 misses, 0 errors, 0s total dur; total 0.00 MB; avg 0.00 MB/s
894 puts: 0 errors, 1m18.7s total dur; total 140.99 MB; avg 1.79 MB/s
total time:  1m21s

And then it goes much faster the second time.

% rm -rf go-cache
% go build .     
disk stats: 
641 gets: 0 hits, 641 misses, 0 errors, 0s total dur
641 puts: 0 errors, 0s total dur
s3 stats: 
642 gets: 641 hits, 1 misses, 0 errors, 35.8s total dur; total 140.42 MB; avg 3.92 MB/s
2 puts: 0 errors, 300ms total dur; total 0.00 MB; avg 0.01 MB/s
total time:  12s

Credits

Derived from https://github.com/bradfitz/go-tool-cache and or-shachar/go-tool-cache.