Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 243 lines (164 sloc) 10.995 kB
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
1 # node.js blob service gateway.
5b3c29c add copyright headers
SonicWang authored
2
dd7a8d2 s3 interface is back
SonicWang authored
3 Copyright (c) 2011-2012 VMware, Inc.
5b3c29c add copyright headers
SonicWang authored
4
dd7a8d2 s3 interface is back
SonicWang authored
5 The blob service provides an S3-compatible HTTP endpoint to an underlying storage provider. A driver model is used for different providers. Currently the available drivers include S3 (Amazon web services) or a local file system (FS) driver.
cea513e Initial commit
SonicWang authored
6
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
7 ## Authors
8 - Sonic Wang (wangs@vmware.com)
cea513e Initial commit
SonicWang authored
9
10 ## Features
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
11 - RESTful web service
dd7a8d2 s3 interface is back
SonicWang authored
12 - S3 compatibility
13 - plugin model (currently support local fs, s3)
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
14 - streaming in/out blobs
15 - basic blob operations: create/delete/get/copy
dd7a8d2 s3 interface is back
SonicWang authored
16 - create/list/delete buckets
452d7a5 update to latest vblob version
SonicWang authored
17 - enumerate objects with prefix/delimiter/marker/max-keys
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
18 - user defined meta data
cea513e Initial commit
SonicWang authored
19
452d7a5 update to latest vblob version
SonicWang authored
20 ## FS driver
21 For scalability, the file system driver supports multiple instances of the gateway operating on the same files. Metadata and blobs live in separate physical files in different directories. New files do not replace existing files until they have been fully uploaded and persisted. Old files are cleaned up using garbage collection processes which run in the background.
22
23 The file system driver was designed to provide consistency without the use of any database. It depends on the file system for reference-counting hard links and atomic 'mv' operations.
24
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
25 ## API Documentation
452d7a5 update to latest vblob version
SonicWang authored
26 - For details of supported API features see the [doc](doc) directory
cea513e Initial commit
SonicWang authored
27
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
28 ## Dependencies
aad4913 update readme file
SonicWang authored
29 The following 3rd party npm modules have been included in the node_modules subdirectory
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
30
aad4913 update readme file
SonicWang authored
31 - express: web framework
32 - winston: logging module
33 - sax: xml parser
34 - vows: test framework
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
35
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
36 ## Installing node.js
452d7a5 update to latest vblob version
SonicWang authored
37 The blob service depends on [node.js](http://nodejs.org/). We are working with the v0.6 branch and v0.6.10 tag.
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
38
39 To match the same branch and tag from github:
40
452d7a5 update to latest vblob version
SonicWang authored
41 $> git clone -b v0.6 https://github.com/joyent/node.git <target-directory>
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
42 $> cd <target-directory>
452d7a5 update to latest vblob version
SonicWang authored
43 $> git checkout v0.6.10
cea513e Initial commit
SonicWang authored
44
452d7a5 update to latest vblob version
SonicWang authored
45 To build node.js
cea513e Initial commit
SonicWang authored
46
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
47 ./configure
48 make
49 make install
cea513e Initial commit
SonicWang authored
50
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
51 ## Installing npm
52 Most node.js applications require modules distributed via npm. Instructions for installing npm can be found at [npmjs.org](http://npmjs.org/).
cea513e Initial commit
SonicWang authored
53
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
54 ## Deploying the blob service from source
cea513e Initial commit
SonicWang authored
55
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
56 $> git clone <project-url> <target-directory>
57 $> cd <target-directory>
452d7a5 update to latest vblob version
SonicWang authored
58 $> cp config.json.default config.json
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
59
60 now edit `config.json`
cea513e Initial commit
SonicWang authored
61
452d7a5 update to latest vblob version
SonicWang authored
62 ## Configuration via config.json
63 The gateway and its drivers can be configured via config.json which is read at startup. This file is structured as an array of drivers each with a name, type, and options, followed by some global options which include the name of the current driver. It is possible to configure multiple drivers of the same same type in the same file, but only one of these will be activated on startup.
cea513e Initial commit
SonicWang authored
64
452d7a5 update to latest vblob version
SonicWang authored
65 Drivers are assumed to live under `./drivers/<type>` -- currently only `fs` and `s3` driver types are included.
dd7a8d2 s3 interface is back
SonicWang authored
66
452d7a5 update to latest vblob version
SonicWang authored
67 The FS driver stores all blobs in directories and files under a "root" location in the local file system. The default location if this value is unspecified is `./fs_root`
68
69 The S3 driver requires a valid key and secret from your Amazon S3 storage account. All operations are simply passed through the gateway and handled by S3.
70
71 NOTE: if no config.json file is found, the gateway will use the settings from `config.json.default`. The following listing shows all currently available config.json options for the FS and S3 drivers.
cea513e Initial commit
SonicWang authored
72
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
73 {
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
74 "drivers": [
75 {
76 "<fs-driver-name>": {
77 "type": "fs",
78 "option": {
452d7a5 update to latest vblob version
SonicWang authored
79 "root": "<pathname for storing blobs - default is `./fs_root` >",
80 "node_exepath": "<path to node executable - default is `node` >",
81 "collector" : <enable gc and enumeration - default is `true`, NOTE: must be limited to one instance >,
a12a88f fixing pivotal tracker ticket https://www.pivotaltracker.com/projects…
SonicWang authored
82 "tmp_path" : "<path to a global tmp folder for storing runtime temp files, default is `/tmp` >",
452d7a5 update to latest vblob version
SonicWang authored
83 "gc_exepath": "<path to gc js file, default is `drivers/fs/fs_gc.js` >",
84 "gc_interval": <ms per gc execution, default is `600,000` (10 min) >,
85 "gcfc_exepath": "<path to lightweight gc js file, default is `drivers/fs/fs_gcfc.js` >",
86 "gcfc_interval": <ms per lightweiht gc execution, default is `1,500` (1.5 sec) >,
87 "gctmp_exepath": "<path to tmp folder gc js file, default is `drivers/fs/fs_gctmp.js` >",
88 "gctmp_interval": <ms per tmp folder gc execution, default is 3,600,000 (1 hr) >,
89 "ec_exepath": "<path to ec js file, default is `drivers/fs/fs_ec.js` >",
90 "ec_interval": <ms per ec execution, default is `1,500` (1.5 sec) >,
91 "quota": <maximum number of bytes allowed to store, default is 100MB >,
92 "obj_limit" : <maximum number of blobs allowed to store, default is 10,000 >
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
93 }
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
94 }
dd7a8d2 s3 interface is back
SonicWang authored
95 },
96 {
97 "<s3-driver-name>": {
98 "type": "s3",
99 "option": {
100 "key": "<s3 key>",
101 "secret": "<s3 secret>"
102 }
103 }
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
104 }
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
105 ],
106 "port": "<port>",
107 "current_driver": "<driver-name>",
108 "logtype": "winston",
109 "logfile": "<pathname for the log>",
452d7a5 update to latest vblob version
SonicWang authored
110 "keyID": "<id for front-end authentication>",
111 "secretID": "<secret for front-end authentication>",
112 "auth": "<`basic`, `digest` or `s3`; other values mean disabled>",
113 "debug": <set to true to enable verbose output of all requests in the console>,
114 "account_file" : <pathname for the cloudfoundry binding file, default is "./account.json">,
115 "account_api" : <set to true to enable cloudfoundry service apis, default is true>
b3d979b documentation of REST and driver APIs; update readme accordingly
SonicWang authored
116 }
117
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
118 `current driver` is used to specify the driver to be used by the service at startup. Only 1 driver can be in use at any time. If no default is specified the first driver will be used.
cea513e Initial commit
SonicWang authored
119
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
120 `logfile` specifies the path to the log file. The `logtype` has to be [winston](https://github.com/indexzero/winston).
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
121
dd7a8d2 s3 interface is back
SonicWang authored
122 `keyID` and `secretID` and `auth` are used to control front-end authentication. If either the key or id is not present or if auth is not set to a proper auth type then authentication is disabled. Currently the following types are supported: "basic", "digest", and "s3". "basic" and "digest" implementation follows rfc2617.
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
123
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
124 `debug` is used to log request and response headers to the console for debugging purposes. Its value is treated as boolean.
cea513e Initial commit
SonicWang authored
125
dd7a8d2 s3 interface is back
SonicWang authored
126 `account_file` is the place where vblob instance stores the CloudFoundry service binding credentials. After a vblob service instance is provisioned in CloundFoundry, user can binding multiple CF apps to the instance. This file indicates vblob where to load and update the binding crendentials so that the apps can properly authenticate with the vblob instance. `account_api` controls whether the CF bind/unbind API is on/off in the instance. For a single node deployment, this is always set to true. For a multi-node deployment, only one node should turn on this switch.
127
cea513e Initial commit
SonicWang authored
128 ## Usage
129
130 node server.js [-f path_to_config_file]
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
131
452d7a5 update to latest vblob version
SonicWang authored
132 Note that `-f config-path` is optional. The gateway will look for `./config.json` or `./config.json.default`
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
133
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
134 ## Testing
452d7a5 update to latest vblob version
SonicWang authored
135 To run the common (non-driver-specific) unit tests, first make sure server.js is configured and running with one of the drivers, then
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
136
137 cd test
452d7a5 update to latest vblob version
SonicWang authored
138 ../node_modules/vows/bin/vows common_test/test*.js --spec
139
140 To run the fs-driver tests from the same test folder do:
cea513e Initial commit
SonicWang authored
141
452d7a5 update to latest vblob version
SonicWang authored
142 ../node_modules/vows/bin/vows fs_test/test*.js --spec
143
144 NOTE: To avoid invoking vows via ../node_modules, install vows globally using `npm install -g vows`
aad4913 update readme file
SonicWang authored
145
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
146 ## Manual usage with curl
aad4913 update readme file
SonicWang authored
147 The following curl commands assume:
f2ca2d3 vblob gw with no-mongo implementaion, with read/write isolation, with…
SonicWang authored
148
452d7a5 update to latest vblob version
SonicWang authored
149 - authentication is NOT enabled.
150 - the node.js process is running on localhost and listening on port 9981.
cea513e Initial commit
SonicWang authored
151
dd7a8d2 s3 interface is back
SonicWang authored
152 ### Listing all buckets
cea513e Initial commit
SonicWang authored
153
452d7a5 update to latest vblob version
SonicWang authored
154 curl http://localhost:9981 -v
cea513e Initial commit
SonicWang authored
155
dd7a8d2 s3 interface is back
SonicWang authored
156 ### Listing objects in a bucket
cea513e Initial commit
SonicWang authored
157
452d7a5 update to latest vblob version
SonicWang authored
158 curl http://localhost:9981/container1 -v
cea513e Initial commit
SonicWang authored
159
452d7a5 update to latest vblob version
SonicWang authored
160 Four parameters are supported: prefix; delimiter; marker; max-keys. E.g.:
cea513e Initial commit
SonicWang authored
161
452d7a5 update to latest vblob version
SonicWang authored
162 curl "http://localhost:9981/container1/?prefix=A/&delimiter=/" -v
cea513e Initial commit
SonicWang authored
163
452d7a5 update to latest vblob version
SonicWang authored
164 The above query will list virtual folders in 'container1' starting with 'A/' and using delimiter '/'.
cea513e Initial commit
SonicWang authored
165
dd7a8d2 s3 interface is back
SonicWang authored
166 ### Create a bucket
cea513e Initial commit
SonicWang authored
167
452d7a5 update to latest vblob version
SonicWang authored
168 curl http://localhost:9981/container1 -X PUT -v
cea513e Initial commit
SonicWang authored
169
dd7a8d2 s3 interface is back
SonicWang authored
170 ### Delete a bucket
cea513e Initial commit
SonicWang authored
171
452d7a5 update to latest vblob version
SonicWang authored
172 curl http://localhost:9981/container1 -X DELETE -v
cea513e Initial commit
SonicWang authored
173
174 ### Uploading a file
175
452d7a5 update to latest vblob version
SonicWang authored
176 curl http://localhost:9981/container1/file1.txt -X PUT -T file1.txt -v
cea513e Initial commit
SonicWang authored
177
452d7a5 update to latest vblob version
SonicWang authored
178 User-defined metadata is supported. All user meta keys start with prefix `x-amz-meta-`. E.g.
cea513e Initial commit
SonicWang authored
179
452d7a5 update to latest vblob version
SonicWang authored
180 curl http://localhost:9981/container1/file1.txt -X PUT -T file1.txt -H "x-amz-meta-comment:hello_world"
cea513e Initial commit
SonicWang authored
181
182 ### Copying a file
183
452d7a5 update to latest vblob version
SonicWang authored
184 curl http://localhost:9981/container1/file1.txt -X PUT -H "x-amz-copy-source:/container2/file2.txt"
cea513e Initial commit
SonicWang authored
185
452d7a5 update to latest vblob version
SonicWang authored
186 The above request copy file2.txt in container2 to file1.txt in container1. This operation will copy meta data as well.
cea513e Initial commit
SonicWang authored
187
188 ### Deleting a file
189
452d7a5 update to latest vblob version
SonicWang authored
190 curl http://localhost:9981/container1/file1.txt -X DELETE -v
cea513e Initial commit
SonicWang authored
191
192 ### Reading a file
193
452d7a5 update to latest vblob version
SonicWang authored
194 curl http://localhost:9981/container1/file1.txt -v
cea513e Initial commit
SonicWang authored
195
452d7a5 update to latest vblob version
SonicWang authored
196 Standard `range` headers are supported for single range reads. E.g.
cea513e Initial commit
SonicWang authored
197
452d7a5 update to latest vblob version
SonicWang authored
198 curl http://localhost:9981/container1/file1.txt -H "range:bytes=123-892" -v
cea513e Initial commit
SonicWang authored
199
dd7a8d2 s3 interface is back
SonicWang authored
200 ## Using s3-curl with gateway authentication
201
202 Instead of using curl with authentication disabled, the gateway can also be accessed in an authenticated fashion via s3-curl.pl which is a utility for making signed requests to s3 available on [aws](http://aws.amazon.com/code/128).
203
204 Credentials for authentication via s3-curl are stored in `.s3curl` as follows
205
206 %awsSecretAccessKeys = (
207
208 # real s3 account
209 s3 => {
210 id => '<s3-id>',
211 key => '<s3-key',
212 },
213
214 gateway => {
215 id => '<gateway-id>',
216 key => '<gateway-key>',
217 },
218 );
219
220 Requests are then performed by specifying which credentials to use on the command line. Parameters for `curl` go after the `--`. E.g.
221
452d7a5 update to latest vblob version
SonicWang authored
222 ./s3curl.pl --id gateway -- -X DELETE -v http://localhost:9981/BucketToDelete
dd7a8d2 s3 interface is back
SonicWang authored
223
224 A small modification is required to add the endpoint where the gateway is running to the list of endpoints in the perl script. E.g. if you are running on localhost, you would add localhost to the @endpoints array as follows:
225
226 # begin customizing here
227 my @endpoints = ( 's3.amazonaws.com',
228 's3-us-west-1.amazonaws.com',
229 's3-eu-west-1.amazonaws.com',
230 's3-ap-southeast-1.amazonaws.com',
231 's3-ap-northeast-1.amazonaws.com',
232 'localhost' );
233
cea513e Initial commit
SonicWang authored
234 ## Server Tuning
235
310ef06 add binding/unbinding/get-binding for cloundfoundry service; add s3 c…
SonicWang authored
236 When the gateway is handling a large number of concurrent requests, it may open too many file descriptors. It is suggested to increase the file descriptor limit. E.g.: in unix-type systems:
cea513e Initial commit
SonicWang authored
237
452d7a5 update to latest vblob version
SonicWang authored
238 ulimit -n 8192
6bdd54f Update README to include Bug filing info per Deepika
Chloe Jackson authored
239
240 ## File a Bug
241
0a38aa1 @mreider Update README.md
mreider authored
242 Use the Github Issues feature in the appropriate repository to file a bug against Cloud Foundry Open Source and its components,
Something went wrong with that request. Please try again.