An open-source Node.js implementation of a server handling the S3 protocol
JavaScript Perl Java C Ruby Shell Other
Latest commit 28c9dad Jan 19, 2017 @LaurenSpiegel LaurenSpiegel committed on GitHub Merge pull request #532 from scality/Forward-rel/6.4-to-master
Forward rel/6.4 to master
Permalink
Failed to load latest commit information.
bin FT: Add account-level listing Jan 12, 2017
conf DOCKER: using Environment Variables Nov 28, 2016
lib COMPAT: deleteCORS return 204 status Jan 18, 2017
localData Set up for file backend. Jun 10, 2016
localMetadata Set up for file backend. Jun 10, 2016
res Add files via upload Jun 8, 2016
tests COMPAT: deleteCORS return 204 status Jan 18, 2017
.babelrc feat(babel): Upgrade to babel6 Jun 1, 2016
.eslintrc Remove mentions of the forbidden word Jun 1, 2016
.gitignore Initial commit Aug 31, 2015
CONTRIBUTING.md Add Contribution rules Jun 8, 2016
DESIGN.md DOC: Add website API's to design doc Dec 30, 2016
DOCKER.md Adding in production section Dec 2, 2016
Dockerfile Dockerfile: Make image even lighter Dec 3, 2016
DockerfileMem Dockerfile: Make image even lighter Dec 3, 2016
Healthchecks.md ft: add stats to healthcheck route Nov 22, 2016
LICENSE Fix typo in LICENSE file Jun 8, 2016
README.md FT: Implement bucket website serving Dec 23, 2016
TESTING.md Remove mentions of the forbidden word Jun 1, 2016
circle.yml Merge remote-tracking branch 'origin/rel/6.3' into port/fwdport6.3-ma… Jan 17, 2017
config.json Add listenOn field to config Jan 17, 2017
constants.js Restore max limit on listing Dec 15, 2016
docker-entrypoint.sh DOCKER: using Environment Variables Nov 28, 2016
index.js cleanup: remove unneeded babel ignores Sep 30, 2016
init.js FIX: Log caught errors properly Nov 17, 2016
package.json FT: Implement bucket website serving Dec 23, 2016
tester.js tests: Rework functional testing for improved test visibility Dec 2, 2016
utapiServer.js cleanup: remove unneeded babel ignores Sep 30, 2016
wait_for_local_port.bash tests: Rework functional testing for improved test visibility Dec 2, 2016

README.md

S3 Server

S3 Server logo

CircleCI Scality CI

Learn more at s3.scality.com

Contributing

In order to contribute, please follow the Contributing Guidelines.

Installation

Dependencies

Building and running the S3 Server requires node.js 4.5 and npm v2 . Up-to-date versions can be found at Nodesource.

Clone source code

git clone https://github.com/scality/S3.git

Install js dependencies

Go to the ./S3 folder,

npm install

Run it with a file backend

npm start

This starts an S3 server on port 8000. The default access key is accessKey1 with a secret key of verySecretKey1.

By default the metadata files will be saved in the localMetadata directory and the data files will be saved in the localData directory within the ./S3 directory on your machine. These directories have been pre-created within the repository. If you would like to save the data or metadata in different locations of your choice, you must specify them with absolute paths. So, when starting the server:

mkdir -m 700 $(pwd)/myFavoriteDataPath
mkdir -m 700 $(pwd)/myFavoriteMetadataPath
export S3DATAPATH="$(pwd)/myFavoriteDataPath"
export S3METADATAPATH="$(pwd)/myFavoriteMetadataPath"
npm start

Run it with an in-memory backend

npm run mem_backend

This starts an S3 server on port 8000. The default access key is accessKey1 with a secret key of verySecretKey1.

Run it for continuous integration testing or in production with Docker

DOCKER.md

Testing

You can run the unit tests with the following command:

npm test

You can run the linter with:

npm run lint

Running functional tests locally:

The test suite requires additional tools, s3cmd and Redis installed in the environment the tests are running in.

  • Install s3cmd
  • Install redis and start Redis.
  • Add localCache section to your config.json:
"localCache": {
    "host": REDIS_HOST,
    "port": REDIS_PORT
}

where REDIS_HOST is your Redis instance IP address ("127.0.0.1" if your Redis is running locally) and REDIS_PORT is your Redis instance port (6379 by default)

  • Add the following to the etc/hosts file on your machine:
127.0.0.1 bucketwebsitetester.s3-website-us-east-1.amazonaws.com
  • Start the S3 server in memory and run the functional tests:
npm run mem_backend
npm run ft_test

Configuration

If you want to specify an endpoint (other than localhost), you need to add it to your config.json:

"regions": {

     "localregion": ["localhost"],
     "specifiedregion": ["myhostname.com"]
},

Note that our S3server supports both:

However, hosted-style requests will not hit the server if you are using an ip address for your host. So, make sure you are using path-style requests in that case. For instance, if you are using the AWS SDK for JavaScript, you would instantiate your client like this:

const s3 = new aws.S3({
   endpoint: 'http://127.0.0.1:8000',
   s3ForcePathStyle: true,
});

Getting started: List of applications that have been tested with S3 Server

GUI

Cyberduck

Cloud Explorer

Command Line Tools

s3curl

https://github.com/scality/S3/blob/master/tests/functional/s3curl/s3curl.pl

aws-cli

~/.aws/credentials on Linux, OS X, or Unix or C:\Users\USERNAME\.aws\credentials on Windows

[default]
aws_access_key_id = accessKey1
aws_secret_access_key = verySecretKey1

See all buckets:

aws s3 ls --endpoint-url=http://localhost:8000

s3cmd

If using s3cmd as a client to S3 be aware that v4 signature format is buggy in s3cmd versions < 1.6.1.

~/.s3cfg on Linux, OS X, or Unix or C:\Users\USERNAME\.s3cfg on Windows

[default]
access_key = accessKey1
secret_key = verySecretKey1
host_base = localhost:8000
host_bucket = %(bucket).localhost:8000
signature_v2 = False
use_https = False

See all buckets:

s3cmd ls

rclone

~/.rclone.conf on Linux, OS X, or Unix or C:\Users\USERNAME\.rclone.conf on Windows

[remote]
type = s3
env_auth = false
access_key_id = accessKey1
secret_access_key = verySecretKey1
region = other-v2-signature
endpoint = http://localhost:8000
location_constraint =
acl = private
server_side_encryption =
storage_class =

See all buckets:

rclone lsd remote:

JavaScript

AWS JavaScript SDK

const AWS = require('aws-sdk');

const s3 = new AWS.S3({
    accessKeyId: 'accessKey1',
    secretAccessKey: 'verySecretKey1',
    endpoint: 'localhost:8000',
    sslEnabled: false,
    s3ForcePathStyle: true,
});

JAVA

AWS JAVA SDK

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.services.s3.model.Bucket;

public class S3 {

    public static void main(String[] args) {

        AWSCredentials credentials = new BasicAWSCredentials("accessKey1",
        "verySecretKey1");

        // Create a client connection based on credentials
        AmazonS3 s3client = new AmazonS3Client(credentials);
        s3client.setEndpoint("http://localhost:8000");
        // Using path-style requests
        // (deprecated) s3client.setS3ClientOptions(new S3ClientOptions().withPathStyleAccess(true));
        s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());

        // Create bucket
        String bucketName = "javabucket";
        s3client.createBucket(bucketName);

        // List off all buckets
        for (Bucket bucket : s3client.listBuckets()) {
            System.out.println(" - " + bucket.getName());
        }
    }
}

Ruby

AWS SDK for Ruby - Version 2

require 'aws-sdk'

s3 = Aws::S3::Client.new(
  :access_key_id => 'accessKey1',
  :secret_access_key => 'verySecretKey1',
  :endpoint => 'http://localhost:8000',
  :force_path_style => true
)

resp = s3.list_buckets

fog

require "fog"

connection = Fog::Storage.new(
{
    :provider => "AWS",
    :aws_access_key_id => 'accessKey1',
    :aws_secret_access_key => 'verySecretKey1',
    :endpoint => 'http://localhost:8000',
    :path_style => true,
    :scheme => 'http',
})

Python

boto2

import boto
from boto.s3.connection import S3Connection, OrdinaryCallingFormat


connection = S3Connection(
    aws_access_key_id='accessKey1',
    aws_secret_access_key='verySecretKey1',
    is_secure=False,
    port=8000,
    calling_format=OrdinaryCallingFormat(),
    host='localhost'
)

connection.create_bucket('mybucket')

boto3

import boto3
client = boto3.client(
    's3',
    aws_access_key_id='accessKey1',
    aws_secret_access_key='verySecretKey1',
    endpoint_url='http://localhost:8000'
)

lists = client.list_buckets()

PHP

Should use v3 over v2 because v2 would create virtual-hosted style URLs while v3 generates path-style URLs.

AWS PHP SDK v3

use Aws\S3\S3Client;

$client = S3Client::factory([
    'region'  => 'us-east-1',
    'version'   => 'latest',
    'endpoint' => 'http://localhost:8000',
    'credentials' => [
         'key'    => 'accessKey1',
         'secret' => 'verySecretKey1'
    ]
]);

$client->createBucket(array(
    'Bucket' => 'bucketphp',
));