Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 343 lines (261 sloc) 13.13 kB
f8f436e @mludvig Added comprehensive README file
mludvig authored
1 S3cmd tool for Amazon Simple Storage Service (S3)
2 =================================================
3
4 Author:
5 Michal Ludvig <michal@logix.cz>
6
7 S3tools / S3cmd project homepage:
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
8 http://s3tools.org
f8f436e @mludvig Added comprehensive README file
mludvig authored
9
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
10 S3tools / S3cmd mailing lists:
11 * Announcements of new releases:
12 s3tools-announce@lists.sourceforge.net
13
14 * General questions and discussion about usage
15 s3tools-general@lists.sourceforge.net
16
17 * Bug reports
18 s3tools-bugs@lists.sourceforge.net
c3bc0a8 @mludvig Added more documentation.
mludvig authored
19
f8f436e @mludvig Added comprehensive README file
mludvig authored
20 Amazon S3 homepage:
21 http://aws.amazon.com/s3
22
c3bc0a8 @mludvig Added more documentation.
mludvig authored
23 !!!
24 !!! Please consult INSTALL file for installation instructions!
25 !!!
26
f8f436e @mludvig Added comprehensive README file
mludvig authored
27 What is Amazon S3
28 -----------------
29 Amazon S3 provides a managed internet-accessible storage
30 service where anyone can store any amount of data and
31 retrieve it later again. Maximum amount of data in one
32 "object" is 5GB, maximum number of objects is not limited.
33
34 S3 is a paid service operated by the well known Amazon.com
35 internet book shop. Before storing anything into S3 you
36 must sign up for an "AWS" account (where AWS = Amazon Web
37 Services) to obtain a pair of identifiers: Access Key and
38 Secret Key. You will need to give these keys to S3cmd.
39 Think of them as if they were a username and password for
40 your S3 account.
41
42 Pricing explained
43 -----------------
c5a458d @mludvig 2007-08-13 Michal Ludvig <michal@logix.cz>
mludvig authored
44 At the time of this writing the costs of using S3 are (in USD):
45
46 $0.15 per GB per month of storage space used
47
48 plus
49
50 $0.10 per GB - all data uploaded
51
52 plus
53
54 $0.18 per GB - first 10 TB / month data downloaded
55 $0.16 per GB - next 40 TB / month data downloaded
56 $0.13 per GB - data downloaded / month over 50 TB
57
58 plus
59
60 $0.01 per 1,000 PUT or LIST requests
61 $0.01 per 10,000 GET and all other requests
f8f436e @mludvig Added comprehensive README file
mludvig authored
62
63 If for instance on 1st of January you upload 2GB of
64 photos in JPEG from your holiday in New Zealand, at the
65 end of January you will be charged $0.30 for using 2GB of
c5a458d @mludvig 2007-08-13 Michal Ludvig <michal@logix.cz>
mludvig authored
66 storage space for a month, $0.20 for uploading 2GB
67 of data, and a few cents for requests.
68 That comes to slightly over $0.50 for a complete backup
69 of your precious holiday pictures.
f8f436e @mludvig Added comprehensive README file
mludvig authored
70
71 In February you don't touch it. Your data are still on S3
72 servers so you pay $0.30 for those two gigabytes, but not
73 a single cent will be charged for any transfer. That comes
74 to $0.30 as an ongoing cost of your backup. Not too bad.
75
76 In March you allow anonymous read access to some of your
77 pictures and your friends download, say, 500MB of them.
78 As the files are owned by you, you are responsible for the
79 costs incurred. That means at the end of March you'll be
c5a458d @mludvig 2007-08-13 Michal Ludvig <michal@logix.cz>
mludvig authored
80 charged $0.30 for storage plus $0.09 for the download traffic
f8f436e @mludvig Added comprehensive README file
mludvig authored
81 generated by your friends.
82
83 There is no minimum monthly contract or a setup fee. What
84 you use is what you pay for. At the beginning my bill used
85 to be like US$0.03 or even nil.
86
87 That's the pricing model of Amazon S3 in a nutshell. Check
88 Amazon S3 homepage at http://aws.amazon.com/s3 for more
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
89 details.
f8f436e @mludvig Added comprehensive README file
mludvig authored
90
91 Needless to say that all these money are charged by Amazon
92 itself, there is obviously no payment for using S3cmd :-)
93
94 Amazon S3 basics
95 ----------------
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
96 Files stored in S3 are called "objects" and their names are
97 officially called "keys". Since this is sometimes confusing
98 for the users we often refer to the objects as "files" or
99 "remote files". Each object belongs to exactly one "bucket".
f8f436e @mludvig Added comprehensive README file
mludvig authored
100
101 To describe objects in S3 storage we invented a URI-like
102 schema in the following form:
103
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
104 s3://BUCKET
105 or
f8f436e @mludvig Added comprehensive README file
mludvig authored
106 s3://BUCKET/OBJECT
107
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
108 Buckets
109 -------
110 Buckets are sort of like directories or folders with some
111 restrictions:
112 1) each user can only have 100 buckets at the most,
113 2) bucket names must be unique amongst all users of S3,
114 3) buckets can not be nested into a deeper hierarchy and
115 4) a name of a bucket can only consist of basic alphanumeric
116 characters plus dot (.) and dash (-). No spaces, no accented
117 or UTF-8 letters, etc.
118
119 It is a good idea to use DNS-compatible bucket names. That
120 for instance means you should not use upper case characters.
121 While DNS compliance is not strictly required some features
122 described below are not available for DNS-incompatible named
123 buckets. One more step further is using a fully qualified
124 domain name (FQDN) for a bucket - that has even more benefits.
125
126 * For example "s3://--My-Bucket--" is not DNS compatible.
127 * On the other hand "s3://my-bucket" is DNS compatible but
128 is not FQDN.
129 * Finally "s3://my-bucket.s3tools.org" is DNS compatible
130 and FQDN provided you own the s3tools.org domain and can
131 create the domain record for "my-bucket.s3tools.org".
132
133 Look for "Virtual Hosts" later in this text for more details
134 regarding FQDN named buckets.
135
136 Objects (files stored in Amazon S3)
137 -----------------------------------
138 Unlike for buckets there are almost no restrictions on object
139 names. These can be any UTF-8 strings of up to 1024 bytes long.
140 Interestingly enough the object name can contain forward
141 slash character (/) thus a "my/funny/picture.jpg" is a valid
142 object name. Note that there are not directories nor
143 buckets called "my" and "funny" - it is really a single object
144 name called "my/funny/picture.jpg" and S3 does not care at
145 all that it _looks_ like a directory structure.
146
147 The full URI of such an image could be, for example:
f8f436e @mludvig Added comprehensive README file
mludvig authored
148
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
149 s3://my-bucket/my/funny/picture.jpg
150
151 Public vs Private files
152 -----------------------
153 The files stored in S3 can be either Private or Public. The
154 Private ones are readable only by the user who uploaded them
155 while the Public ones can be read by anyone. Additionally the
156 Public files can be accessed using HTTP protocol, not only
157 using s3cmd or a similar tool.
158
159 The ACL (Access Control List) of a file can be set at the
160 time of upload using --acl-public or --acl-private options
161 with 's3cmd put' or 's3cmd sync' commands (see below).
162
163 Alternatively the ACL can be altered for existing remote files
164 with 's3cmd setacl --acl-public' (or --acl-private) command.
165
166 Simple s3cmd HowTo
f8f436e @mludvig Added comprehensive README file
mludvig authored
167 ------------------
168 1) Register for Amazon AWS / S3
169 Go to http://aws.amazon.com/s3, click the "Sign up
170 for web service" button in the right column and work
171 through the registration. You will have to supply
172 your Credit Card details in order to allow Amazon
173 charge you for S3 usage.
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
174 At the end you should have your Access and Secret Keys
f8f436e @mludvig Added comprehensive README file
mludvig authored
175
176 2) Run "s3cmd --configure"
177 You will be asked for the two keys - copy and paste
178 them from your confirmation email or from your Amazon
179 account page. Be careful when copying them! They are
180 case sensitive and must be entered accurately or you'll
181 keep getting errors about invalid signatures or similar.
182
183 3) Run "s3cmd ls" to list all your buckets.
184 As you just started using S3 there are no buckets owned by
185 you as of now. So the output will be empty.
186
187 4) Make a bucket with "s3cmd mb s3://my-new-bucket-name"
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
188 As mentioned above the bucket names must be unique amongst
f8f436e @mludvig Added comprehensive README file
mludvig authored
189 _all_ users of S3. That means the simple names like "test"
190 or "asdf" are already taken and you must make up something
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
191 more original. To demonstrate as many features as possible
192 let's create a FQDN-named bucket s3://public.s3tools.org:
f8f436e @mludvig Added comprehensive README file
mludvig authored
193
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
194 ~$ s3cmd mb s3://public.s3tools.org
195 Bucket 's3://public.s3tools.org' created
f8f436e @mludvig Added comprehensive README file
mludvig authored
196
197 5) List your buckets again with "s3cmd ls"
198 Now you should see your freshly created bucket
199
200 ~$ s3cmd ls
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
201 2009-01-28 12:34 s3://public.s3tools.org
f8f436e @mludvig Added comprehensive README file
mludvig authored
202
203 6) List the contents of the bucket
204
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
205 ~$ s3cmd ls s3://public.s3tools.org
f8f436e @mludvig Added comprehensive README file
mludvig authored
206 ~$
207
208 It's empty, indeed.
209
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
210 7) Upload a single file into the bucket:
211
212 ~$ s3cmd put some-file.xml s3://public.s3tools.org/somefile.xml
213 some-file.xml -> s3://public.s3tools.org/somefile.xml [1 of 1]
214 123456 of 123456 100% in 2s 51.75 kB/s done
215
216 Upload a two directory tree into the bucket's virtual 'directory':
217
218 ~$ s3cmd put --recursive dir1 dir2 s3://public.s3tools.org/somewhere/
219 File 'dir1/file1-1.txt' stored as 's3://public.s3tools.org/somewhere/dir1/file1-1.txt' [1 of 5]
220 File 'dir1/file1-2.txt' stored as 's3://public.s3tools.org/somewhere/dir1/file1-2.txt' [2 of 5]
221 File 'dir1/file1-3.log' stored as 's3://public.s3tools.org/somewhere/dir1/file1-3.log' [3 of 5]
222 File 'dir2/file2-1.bin' stored as 's3://public.s3tools.org/somewhere/dir2/file2-1.bin' [4 of 5]
223 File 'dir2/file2-2.txt' stored as 's3://public.s3tools.org/somewhere/dir2/file2-2.txt' [5 of 5]
224
225 As you can see we didn't have to create the /somewhere
226 'directory'. In fact it's only a filename prefix, not
227 a real directory and it doesn't have to be created in
228 any way beforehand.
229
230 8) Now list the bucket contents again:
f8f436e @mludvig Added comprehensive README file
mludvig authored
231
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
232 ~$ s3cmd ls s3://public.s3tools.org
233 DIR s3://public.s3tools.org/somewhere/
234 2009-02-10 05:10 123456 s3://public.s3tools.org/somefile.xml
f8f436e @mludvig Added comprehensive README file
mludvig authored
235
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
236 Use --recursive (or -r) to list all the remote files:
f8f436e @mludvig Added comprehensive README file
mludvig authored
237
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
238 ~$ s3cmd ls s3://public.s3tools.org
239 2009-02-10 05:10 123456 s3://public.s3tools.org/somefile.xml
240 2009-02-10 05:13 18 s3://public.s3tools.org/somewhere/dir1/file1-1.txt
241 2009-02-10 05:13 8 s3://public.s3tools.org/somewhere/dir1/file1-2.txt
242 2009-02-10 05:13 16 s3://public.s3tools.org/somewhere/dir1/file1-3.log
243 2009-02-10 05:13 11 s3://public.s3tools.org/somewhere/dir2/file2-1.bin
244 2009-02-10 05:13 8 s3://public.s3tools.org/somewhere/dir2/file2-2.txt
f8f436e @mludvig Added comprehensive README file
mludvig authored
245
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
246 9) Retrieve one of the files back and verify that it hasn't been
247 corrupted:
f8f436e @mludvig Added comprehensive README file
mludvig authored
248
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
249 ~$ s3cmd get s3://public.s3tools.org/somefile.xml some-file-2.xml
250 s3://public.s3tools.org/somefile.xml -> some-file-2.xml [1 of 1]
251 123456 of 123456 100% in 3s 35.75 kB/s done
f8f436e @mludvig Added comprehensive README file
mludvig authored
252
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
253 ~$ md5sum some-file.xml some-file-2.xml
254 39bcb6992e461b269b95b3bda303addf some-file.xml
255 39bcb6992e461b269b95b3bda303addf some-file-2.xml
f8f436e @mludvig Added comprehensive README file
mludvig authored
256
257 Checksums of the original file matches the one of the
258 retrieved one. Looks like it worked :-)
259
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
260 To retrieve a whole 'directory tree' from S3 use recursive get:
f8f436e @mludvig Added comprehensive README file
mludvig authored
261
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
262 ~$ s3cmd get --recursive s3://public.s3tools.org/somewhere
263 File s3://public.s3tools.org/somewhere/dir1/file1-1.txt saved as './somewhere/dir1/file1-1.txt'
264 File s3://public.s3tools.org/somewhere/dir1/file1-2.txt saved as './somewhere/dir1/file1-2.txt'
265 File s3://public.s3tools.org/somewhere/dir1/file1-3.log saved as './somewhere/dir1/file1-3.log'
266 File s3://public.s3tools.org/somewhere/dir2/file2-1.bin saved as './somewhere/dir2/file2-1.bin'
267 File s3://public.s3tools.org/somewhere/dir2/file2-2.txt saved as './somewhere/dir2/file2-2.txt'
f8f436e @mludvig Added comprehensive README file
mludvig authored
268
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
269 Since the destination directory wasn't specified s3cmd
270 saved the directory structure in a current working
271 directory ('.').
f8f436e @mludvig Added comprehensive README file
mludvig authored
272
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
273 There is an important difference between:
274 get s3://public.s3tools.org/somewhere
275 and
276 get s3://public.s3tools.org/somewhere/
277 (note the trailing slash)
278 S3cmd always uses the last path part, ie the word
279 after the last slash, for naming files.
280
281 In the case of s3://.../somewhere the last path part
282 is 'somewhere' and therefore the recursive get names
283 the local files as somewhere/dir1, somewhere/dir2, etc.
f8f436e @mludvig Added comprehensive README file
mludvig authored
284
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
285 On the other hand in s3://.../somewhere/ the last path
286 part is empty and s3cmd will only create 'dir1' and 'dir2'
287 without the 'somewhere/' prefix:
288
289 ~$ s3cmd get --recursive s3://public.s3tools.org/somewhere /tmp
290 File s3://public.s3tools.org/somewhere/dir1/file1-1.txt saved as '/tmp/dir1/file1-1.txt'
291 File s3://public.s3tools.org/somewhere/dir1/file1-2.txt saved as '/tmp/dir1/file1-2.txt'
292 File s3://public.s3tools.org/somewhere/dir1/file1-3.log saved as '/tmp/dir1/file1-3.log'
293 File s3://public.s3tools.org/somewhere/dir2/file2-1.bin saved as '/tmp/dir2/file2-1.bin'
294
295 See? It's /tmp/dir1 and not /tmp/somewhere/dir1 as it
296 was in the previous example.
297
298 10) Clean up - delete the remote files and remove the bucket:
299
300 Remove everything under s3://public.s3tools.org/somewhere/
301
302 ~$ s3cmd del --recursive s3://public.s3tools.org/somewhere/
303 File s3://public.s3tools.org/somewhere/dir1/file1-1.txt deleted
304 File s3://public.s3tools.org/somewhere/dir1/file1-2.txt deleted
305 ...
306
307 Now try to remove the bucket:
308
309 ~$ s3cmd rb s3://public.s3tools.org
310 ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty
311
312 Ouch, we forgot about s3://public.s3tools.org/somefile.xml
313 We can force the bucket removal anyway:
314
315 ~$ s3cmd rb --force s3://public.s3tools.org/
316 WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time...
317 File s3://public.s3tools.org/somefile.xml deleted
318 Bucket 's3://public.s3tools.org/' removed
f8f436e @mludvig Added comprehensive README file
mludvig authored
319
320 Hints
321 -----
322 The basic usage is as simple as described in the previous
323 section.
324
325 You can increase the level of verbosity with -v option and
326 if you're really keen to know what the program does under
327 its bonet run it with -d to see all 'debugging' output.
328
329 After configuring it with --configure all available options
330 are spitted into your ~/.s3cfg file. It's a text file ready
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
331 to be modified in your favourite text editor.
f8f436e @mludvig Added comprehensive README file
mludvig authored
332
333 For more information refer to:
4927c90 @mludvig * README: Updated for 0.9.9
mludvig authored
334 * S3cmd / S3tools homepage at http://s3tools.org
f8f436e @mludvig Added comprehensive README file
mludvig authored
335 * Amazon S3 homepage at http://aws.amazon.com/s3
336
337 Enjoy!
338
339 Michal Ludvig
340 * michal@logix.cz
341 * http://www.logix.cz/michal
342
Something went wrong with that request. Please try again.