This repository has been archived by the owner on Sep 23, 2020. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
updating the README link for Cumulus that was in the introductory email
- Loading branch information
Showing
1 changed file
with
7 additions
and
185 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,193 +1,15 @@ | ||
| cumulus : An Amazon s3 look-alike | ||
| ================================= | ||
| This is an old README for the initial branch development of Cumulus. | ||
|
|
||
| Cumulus is an open source implementation of the Amazon S3 REST API. It | ||
| is packaged with the Nimbus (open source cloud computing software for | ||
| science) however it can be used without nimbus as well. Cumulus allows | ||
| you to server files to users via a known and adopted REST API. You | ||
| clients will be able to access your data storaging service with the | ||
| Amazon S3 clients they already use. | ||
| See the current documentation here: | ||
|
|
||
| Installation | ||
| ============ | ||
| http://github.com/nimbusproject/nimbus/tree/master/cumulus/docs/ | ||
|
|
||
| Cumulus is easy to install. Most users will be able to follow the 'Quick | ||
| Start' guide to achieve a successful installation, but more details are | ||
| provided here for didactic purposes. | ||
| And here: | ||
|
|
||
| Required software: | ||
| ------------------ | ||
| - Python 2.5 | ||
| - twisted.web | ||
| - boto | ||
| http://www.nimbusproject.org/docs/2.5/faq.html#cumulus | ||
|
|
||
| Check prereqs | ||
| ------------- | ||
| The following shows what software is needed and how to check for it on your | ||
| system: | ||
| And here: | ||
|
|
||
| 1) python 2.5 | ||
| http://www.nimbusproject.org/docs/2.5/admin/reference.html#cumulus | ||
|
|
||
| % python --version | ||
|
|
||
| 2) twisted web | ||
|
|
||
| % python -c "from twisted.web import server, resource" | ||
|
|
||
| 3) sqlite | ||
|
|
||
| % which sqlite3 | ||
| ... | ||
| % python -c "import sqlite3" | ||
|
|
||
|
|
||
| Configuration | ||
| ------------- | ||
| From the distribution base directory run the program | ||
| ./install [<target directory>]. If no target is specified it will install | ||
| to the same directory. This program will create an environment setup | ||
| script call 'env.sh'. At times this script may need to be sourced to | ||
| have more convenient access to commands. install.sh also creates | ||
| a configuration file at ~/.nimbus/cumulus.ini | ||
|
|
||
| When the server is run it expects to find the file cumulus.ini in either: | ||
|
|
||
| 1) /etc/nimbus/cumulus.ini | ||
| 2) ~/.nimbus/cumulus.ini | ||
| 3) the same directory from which the program was launched | ||
| 4) file pointed to by the environment variable CUMULUS_SETTINGS_FILE | ||
|
|
||
| This file is generated when the install.sh script is run, but it can | ||
| be modified by the admin later. | ||
|
|
||
| Repository Location | ||
| ------------------- | ||
|
|
||
| In the current implementation user files are stored on a locally mounted | ||
| file system. The reliability and performance of cumulus will thus be | ||
| limited by the reliabilty and performance of that file system. Because | ||
| of this cumulus administrators will often want to specify a location for | ||
| the repository. | ||
|
|
||
| Within the cumulus.ini file three is the [posix]:directory directive. | ||
| This is the directory in which all of the files in the CB | ||
| repository will be stored. The names of the files in that directory | ||
| will be obfuscated based on the bucket/key name. In order to discover | ||
| what file belongs to what bucket/key you must use go through the security | ||
| module. If the authz module is in use there are a series of tools under | ||
| the bin directory which start with cloudfs-* that can help with this. | ||
| In most cases there will be no need for a system administrator to use | ||
| these tools and they are provided for expert usage for problimatic | ||
| situations. | ||
|
|
||
| User Management | ||
| =============== | ||
|
|
||
| There are three tools that are included with cumulus that allow an | ||
| administrator to create, remove, and see a listing of the current users. | ||
| The tools are under the bin/ directory: | ||
|
|
||
| cumulus-add-user.sh | ||
| cumulus-list-users.sh | ||
| cumulus-remove-user.sh | ||
|
|
||
| Each tool has a good usage description via --help and do what would be | ||
| expected of them based on their names. List user allows an admin to | ||
| query the system for user information. The --report options can be used | ||
| in conjunction with the -b option for scripts. | ||
|
|
||
| As an exmple of cumulus-list-users.sh, lets review a situation where an | ||
| an admin recalls | ||
| that a user has a friendly name that starts with a 'b' but they do not | ||
| know the entire name. The admin needs to get the S3 ID and password | ||
| for that user. For that job cumulus-list-users.sh would be used in the | ||
| following way: | ||
|
|
||
| % ./bin/cumulus-list-users.sh n\* | ||
| friendly : nimbusadmin@nimbusproject.org | ||
| ID : eqe0YoRAs2GT1sDvPZKAU | ||
| password : S9Ii7QqcCQxDecrezMn6o5frSFvXhThYWmCE4S7nAf | ||
| quota : None | ||
| canonical_id : 048db304-6b4c-11df-897b-001de0a80259 | ||
|
|
||
| perhaps that is too much noise for the admin and they only want to see | ||
| ID and password in a comma separate format: | ||
|
|
||
| % ./bin/cumulus-list-users.sh -r ID,password -b b\* | ||
| eqe0YoRAs2GT1sDvPZKAU,S9Ii7QqcCQxDecrezMn6o5frSFvXhThYWmCE4S7nAf | ||
|
|
||
| Running the server: | ||
| =================== | ||
|
|
||
| Running the cumulus server is very easy: | ||
|
|
||
| % ./bin/cumulus.sh | ||
|
|
||
| If you wish to make a daemon process out of it you can do so around this | ||
| program. The program nimbusctl (provided with a full Nimbus distribution) | ||
| can be looked to as an example. Customization to the server are done in | ||
| the cumulus.ini file (as explained above). | ||
|
|
||
| HTTPS | ||
| ----- | ||
|
|
||
| In order to use HTTP the admin must have access to a match certificate and | ||
| key file. Such files are produced by Nimbus on installation and they are | ||
| stored at: $NIMBUS_HOME/var/{hostkey, hostcert}.pem. To enable https | ||
| in cumulus the following section of cumulus.ini must be properly altered: | ||
|
|
||
| [https] | ||
| enabled=False | ||
| key=<path to key file> | ||
| cert=<path to certificate file> | ||
|
|
||
|
|
||
| Clients | ||
| ======= | ||
|
|
||
| Since cumulus is protocol compliant with the S3 rest protocol all S3 | ||
| clients should be able to be used with cumulus. We test with both | ||
| the commandline tool s3cmd and the python API boto. | ||
|
|
||
| Using the s3cmd client | ||
| ---------------------- | ||
| Once you have the s3cmd successfully installed and configured you must | ||
| modify the file: $HOME/.s3cfg in order to direct it at this server. | ||
| Make sure the following key value pairs reflect the following changes: | ||
|
|
||
| host_base = <hostname of service> | ||
| host_bucket = <hostname of server> | ||
| use_https = False | ||
|
|
||
| A sample s3cfg file can be found in the install directory in a file named | ||
| dot_s3cfg. | ||
|
|
||
|
|
||
| Using boto | ||
| ---------- | ||
| To use boto it is important to disable virtual host based buckets and | ||
| to point the client at the right server. here is example code that | ||
| will instantiate a boto S3Connection for use with CB: | ||
|
|
||
| cf = OrdinaryCallingFormat() | ||
| hostname = "somehost.com" | ||
|
|
||
| conn = S3Connection(id, pw, host=hostname, port=80, is_secure=False, calling_format=cf) | ||
|
|
||
|
|
||
| From there the S3Connection object can be used like any other. | ||
|
|
||
| Deficiencies | ||
| ============ | ||
|
|
||
| The following features of S3 are not currently implemented in cumulus | ||
|
|
||
| - Versioning | ||
| - Location | ||
| - Logging | ||
| - Object POST | ||
| - Object COPY | ||
| - torrent | ||
|
|
||
| When using s3cmd it seems that all buckets must start with a capital | ||
| letter. |