elclients.html

m4_include(/mcs/m4/worksp.lib.m4)
_NIMBUS_HEADER(EC2 Client Notes)
_NIMBUS_HEADER2(n,n,y,n,n,n,n)
_NIMBUS_CENTER3_COLUMN
_NIMBUS_IS_DEPRECATED


<h2>EC2 Client Notes</h2>

<p>This section explains how to use 3rd-party EC2 and S3 clients with the Nimbus EC2 frontends, both SOAP and Query.</p>

<ul>
    <li><p><a href="#s3boto">Uploading VM Images to Cumulus with Boto</a></p></li>
    <li><p><a href="#s3cmd">Uploading VM Images to Cumulus with s3cmd</a></p></li>
    <li><p><a href="#boto">Using the EC2 Query frontend from Python with Boto</a></p></li>
    <li><p><a href="#botoex">A Full Example with Boto</a></p></li>
    <li><p><a href="#ec2-api-tools">Using the EC2 SOAP frontend from the console</a></p></li>
    <li><p><a href="#spot">Using spot instances</a></p></li>
</ul>

<a name="s3boto"> </a>
<h2>Uploading VM Images to Cumulus with Boto _NAMELINK(s3boto)</h2>
<p>
The python library boto can be used to upload images to 
<a href="faq.html#cumulus">Cumulus</a>.  Once a VM is staged into Cumulus
it can be submitted for execution.  
</p>
<p>
In order for Nimbus to efficiently find VM images in Cumulus a naming convention
is used.  Three pieces of information are needed to follow this convention:
</p>
<ol>
    <li>The repository bucket name</li>By default this is
    <tt class="literal">Repo</tt>
    <li>The image name prefix</li>By default this is 
    <tt class="literal">VMS</tt>
    <li>Your canonical ID</li>The site admin must provide this to you at 
    the time of your account creation.  It can also be found in your 
    cloud clients configuration file (<a href="clouds/cloudquickstart.html">More Info</a>)
</ol>
<p>
That information is used to form a Cumulus url in the following way:
</p>
<pre class="panel">
cumulus://&lt;cumulus hostname&gt;/&lt;bucket name&gt;/&lt;prefix&gt;/&lt;canonical ID&gt;/&lt;image name&gt;
</pre>
<p>
Once you have the proper url formed you can use that information and 
boto to upload the image into the storage repository.
</p>
<pre class="panel">
from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection

s3id="&lt;Cumulus ID&gt;"
s3pw="&lt;Cumulus password&gt;"
host="&lt;Cumulus hostname&gt;"
port=&lt;Cumulus port&gt;
canonical_id="&lt;Canonical ID&gt;"

cf = OrdinaryCallingFormat()
s3conn = S3Connection(s3id, s3pw, host=host, port=port, is_secure=False, calling_format=cf)

bucket = s3conn.get_bucket("Repo")
k = Key(bucket)
k.key = "VMS/" + canonical_id + "/" + image_name
k.set_contents_from_filename(path_to_image)
</pre>
<p>
At this point your image will be ready for submission using the name <i>image_name</i>
</p>
<div class="note"><p class="note-title">Image Names</p>
<p>
You may upload an image to any Cumulus name you wish and still submit it
for run.  This is described below.  However, the above naming convention
is strongly recommended in order to have a smooth and natural experience.
</p>
</div>
<a name="s3cmd"> </a>
<h2>Uploading VM Images to Cumulus with s3cmd _NAMELINK(s3cmd)</h2>
<p>
There is more information <a href="admin/reference.html#s3cmd">here.</a>
</p>
<a name="boto"> </a>
<h2>Using the EC2 Query frontend from Python with Boto _NAMELINK(ec2-api-tools)</h2>

<p>
    Nimbus supports the EC2 Query interface which is used
    by the excellent Python client, <a href="http://code.google.com/p/boto/">boto</a>.
    In order to use this interface, you must have Query credentials from the cloud
    administrator. These are composed of an <i>access identifier</i> (usually the hash
    of your DN) and a shared <i>secret key</i>.
</p>
<p>
    You also need the URL of the Query interface on the Nimbus service node. See the
    <a href="_NIMBUS_WEBSITE/nimbus_cloud">University of Chicago cloud</a> for example.
<p>
Before you can launch a VM you must first upload it to Cumulus.  That process
is described <a href="#s3boto">here</a>
</p>

<div class="panel"><pre>
import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn =  boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE",
                         port=SERVICE_PORT_NUMBER, region=region)

# then do something with the connection
conn.run_instances("image_name")
</pre></div>

<p>Please refer to the boto documentation for more details on usage.</p>
<div class="note"><p class="note-title">Image Names</p>
<p>Image names can be a full cumulus:// url, or simply the image name 
portion of the naming convention cumulus://&lt;cumulus hostname&gt;/&lt;bucket name&gt;/&lt;prefix&gt;/&lt;canonical ID&gt;/&lt;image name&gt;
</p>
</div>

<a name="botoex"> </a>
<h2>A Full Example with Boto _NAMELINK(botoex)</h2>
<div class="panel"><pre>
import boto
from boto.s3.key import Key
import time
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.connection import S3Connection
from boto.ec2.connection import EC2Connection
from boto.ec2.regioninfo import RegionInfo
    
access_id="&lt;Cumulus ID&gt;"
access_secret="&lt;Cumulus password&gt;"
host="&lt;Cumulus hostname&gt;"
port=&lt;Service port: 8444&gt;
cumulusport=&lt;Cumulus port: 8888&gt;
canonical_id="&lt;Canonical ID&gt;"

region = RegionInfo(name="nimbus", endpoint=host)
ec2conn = boto.connect_ec2(access_id, access_secret, region=region, port=port)
cf = OrdinaryCallingFormat()
s3conn = S3Connection(access_id, access_secret, host=host, port=cumulusport, is_secure=False, calling_format=cf)

# upload the file according to the Nimbus naming convention
bucket = s3conn.get_bucket("Repo")
k = boto.s3.key.Key(bucket)
k.key = "VMS/%s/%s" % (canonical_id, "my_image_name")
k.set_contents_from_filename(&lt;path to image file&gt;)

# now run the VM
reservation = ec2conn.run_instances("my_image_name")
instance = reservation.instances[0]

# poll for status
while instance.state != 'running':
    print 'instance is %s' % instance.state
    time.sleep(30)
    instance.update()

# terminate the instance
reservation.instances[0].terminate()

</pre></div>


<a name="ec2-api-tools"> </a>
<h2>Using the EC2 SOAP frontend from the console _NAMELINK(ec2-api-tools)</h2>

<p>
    When using a cloud running the <a href="faq.html#ec2-frontend">EC2
    frontend</a>, you can download this
    <a href="http://ec2-downloads.s3.amazonaws.com/ec2-api-tools-1.3-57419.zip">EC2
    client</a> from Amazon or try a number of different client that are
    <a href="http://www.google.com/search?hl=en&q=ec2%20client">out there</a>.
</p>

<div class="note"><p class="note-title">EC2 Upgrades</p>
<p>
    Amazon EC2 upgrades happen without warning and so there is sometimes
    a sync error between their default tools and the tools needed to work with
    particular Nimbus elastic services.
</p>
</div>
<ul>
	<li>
        Nimbus 2.7+ supports the "2010-08-31" WSDL
        (<a href="http://ec2-downloads.s3.amazonaws.com/ec2-api-tools-1.3-57419.zip">this EC2
        client</a>).
	</li>
    <li>
	    Nimbus 2.3 - 2.6 supports the "2009-08-15" WSDL
        (<a href="http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-42584.zip">this EC2
        client</a>) .
	</li>
    <li>
        Nimbus TP2.2 supports the "2008-05-05" WSDL
        (<a href="http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-24159.zip">this EC2
        client</a>).
    </li>
    <li>
        The older Nimbus TP2.0 supports the "2008-02-01" WSDL
        (<a href="http://s3.amazonaws.com/ec2-downloads/ec2-api-tools-1.3-19403.zip">this EC2
        client</a>).
    </li>
    <li>
        So <a href="http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip">the
        default client</a> may not always be the one to use.  See a specific cloud's
        documentation for the definitive tools URL (for example, the
        <a href="/nimbus_cloud">Nimbus cloud</a>).
        And see
        <a href="http://bugzilla.globus.org/globus/show_bug.cgi?id=6558">enhancement 6558</a>.
    </li>
</ul>

<br />
<hr />
<p>
    With the client from Amazon, you need to adjust the following environment variables
    (using other clients, the configurations can vary).
</p>
<ul>
    <li>
        <i>EC2_HOME</i> - Set the base directory (such that $EC2_HOME/bin is valid)
    </li>
    <li>
        <i>EC2_URL</i> - Set the service endpoint to the Nimbus based service
    </li>
    <li>
        <i>EC2_CERT</i> - Set the public credential to your public .pem
        based cert
    </li>
    <li>
        <i>EC2_PRIVATE_KEY</i> - Set the private credential to an unencrypted
        .pem based key
    </li>
</ul>

<p>
    Example:
</p>
_EXAMPLE_GENERICCMD_BEGIN
export EC2_HOME=`pwd`
_EXAMPLE_CMD_END
_EXAMPLE_GENERICCMD_BEGIN
export EC2_URL=https://HOST:PORT/wsrf/services/ElasticNimbusService
_EXAMPLE_CMD_END
_EXAMPLE_GENERICCMD_BEGIN
export EC2_CERT=/tmp/ec2clientcert.pem
_EXAMPLE_CMD_END
_EXAMPLE_GENERICCMD_BEGIN
export EC2_PRIVATE_KEY=/tmp/ec2clientkey.pem
_EXAMPLE_CMD_END


<p>
    The URL for <i>EC2_URL</i> will be provided to you by an administrator or
    some documentation
    (<i>HOST:PORT</i> in the example
    is a placeholder for an actual hostname and port number)
</p>
<p>
   For example, see the
    <a href="_SCIENCECLOUDS_WEBSITE/clouds/nimbus.html">Nimbus cloud</a>
    docs for running EC2 clients.
</p>
<p>
    If you have a problem, consult the
    <a href="admin/troubleshooting.html">troubleshooting</a> guide.
</p>

<br />
<hr />

<p>
    Before being able to launch an instance, you will need to register a keypair
    like so:
</p>
<ul>
<li>
_EXAMPLE_GENERICCMD_BEGIN
MYPUBKEY=`cat .ssh/id_rsa.pub`
_EXAMPLE_CMD_END
</li>
<li>
<p>
    Check to make sure those contents look OK:
</p>
_EXAMPLE_GENERICCMD_BEGIN
echo $MYPUBKEY
_EXAMPLE_CMD_END
</li>

<li>
<p>
    Send the public key value to the service, labelling it "mykey"
</p>
_EXAMPLE_GENERICCMD_BEGIN
ec2-add-keypair "mykey||$MYPUBKEY"
_EXAMPLE_CMD_END
</li>
</ul>

<hr />
        
<p>
    List the available images in your personal directory:
</p>
_EXAMPLE_GENERICCMD_BEGIN
ec2-describe-images
_EXAMPLE_CMD_END

<p>
    Pick one, for example "hello-cloud".  And with the "-k" argument, use the
    ssh key label you used above to register an ssh public key.
</p>
_EXAMPLE_GENERICCMD_BEGIN
ec2-run-instances -k mykey hello-cloud
_EXAMPLE_CMD_END

<p>
    Check status:
</p>
_EXAMPLE_GENERICCMD_BEGIN
ec2-describe-instances
_EXAMPLE_CMD_END

<p>
    After seeing 'pending' change to 'running', log in at the address printed:
</p>
_EXAMPLE_GENERICCMD_BEGIN
ssh <b>root</b>@abc.def.com
_EXAMPLE_CMD_END

<p>
    Terminate
    using the "instance ID" printed when you first launched (and also available
    from <i>ec2-describe-instances</i>).  It looks like "i-4662834e":
</p>
_EXAMPLE_GENERICCMD_BEGIN
ec2-terminate-instances i-4662834e
_EXAMPLE_CMD_END

<br/><br/>


<a name="spot"> </a>
<h2>Using spot instances _NAMELINK(spot)</h2>

<p>
    Amazon EC2 spot instances (SIs) introduces a new approach to purchase virtual machine instances on Infrastructure as a Service clouds. In this approach, the cloud provider reserves part of the unused capacity for auction, and clients bid on this capacity. Each available VM type has an associated Spot Price, that changes periodically based on supply and demand. Clients with bids that exceed current SP gain access to available VMs for as long as their bid exceeds that price. More information on Amazon EC2 spot instances can be found at <a href="http://aws.amazon.com/ec2/spot-instances/">http://aws.amazon.com/ec2/spot-instances/</a>.
</p>
<p>
    You may also be interested in the 
	<a href="admin/reference.html#backfill-and-spot-instances">spot instances
	introductory material</a> in the Nimbus administrator's guide.
</p>
<p>
    Nimbus introduces an open source implementation of spot instances that will allow science cloud users to save allocation credits in non-critical tasks during low-demand periods. Currently, the spot instances features are available in Nimbus through the following interfaces: Amazon EC2 WSDLs and Amazon EC2 Query API. Moreover, developers may want to link their own applications with Nimbus' spot instances through the RM API.
</p>
<p>
    The Nimbus' spot instances operations are compliant with the WSDL schema version <b>2010-06-15</b> and higher. You can follow the instructions above in order to connect your preferred EC2 client to a Nimbus cloud.
</p>
<p>
    Before submitting a spot instance request you should define a maximum bid for that request. There is no exact formula for defining an ideal bid. It basically depends on your needs and how much are you willing to pay to run your tasks.
</p>
<p>
    The "price" is represented in Nimbus as a discount applied to minutes charged to your account.  Thus, you likely do not want to bid above "1.0" since regular requests will always trump spot instances but your spot instance requests will be more "expensive" than the regular ones (this is akin to bidding above the normal instance prices on EC2).
</p>
<p>
    The spot price history is a valuable information in determining the maximum bid for your requests. A good way to start is to place your bid between the historical maximum and minimum spot prices. If you want your instances to have a good chance of being executed straight away, you should place your bid somewhere near the maximum historical spot price. In case your tasks are not so urgent and can wait a couple of days (or weeks) to get executed on a low cost, you should place your bid somewhere close to the minimum historical spot price.
</p>

<p>
    The Spot Price history is available in the EC2 APIs through the Describe Spot Price History operation. The supported parameters of this operation are:
</p>
<ul>
    <li><i>Start time</i>: Start date and time of the spot instance price history data. (optional). Date string should conform with the ISO 8601 standard.</li>
    <li><i>End time</i>: End date and time of the spot instance price history data. (optional). Date string should conform with the ISO 8601 standard.</li>
    <li><i>Instance type</i>: The instance type of the spot instance price history data (optional). Currently Nimbus only supports one type of spot instance per site. Please contact your administrator to know which spot instance type is supported in your site.</li>
</ul>

<p><b>SOAP Client (Amazon)</b></p>

<p>Operation command and parameters:</p>

<i>ec2-request-spot-instances image_id --price price [--instance-count count] [--type type] [--user-data data] [--key key-pair][--instance-type type]</i>

<p>This example creates a spot instances request for ten m1.small instances.</p>

<div class="panel"><pre>
$ ec2-request-spot-instances --price 0.50 myimage.img --key MyKeypair --instance-type m1.small --instance-count 10 --type one-time
SPOTINSTANCEREQUEST sir-f102a405 0.50 one-time active 2009-12-12T22:58:47+0200 i-3597b470 myimage.img m1.small default
</pre></div>

<p><b>Query Client (Boto)</b></p>

<p>Operation signature and parameters:</p>

<i>boto.ec2.connection.request_spot_instances(price, image_id, count=1, type=None, key_name=None, user_data=None, instance_type='m1.small')</i>

<p>Example:</p>

<div class="panel"><pre>
import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Creates a spot instances request for ten m1.small instances.
request = conn.request_spot_instances(0.50, 'myimage.img', key_name='MyKeypair', instance_type='m1.small')
</pre></div>

<h4>Retrieving spot instance Requests</h4>

<p>Once submitted, you can check the status of your spot instance requests through the Describe spot instance Requests operation. Besides request launch information, this operation returns the request state (open, active, canceled or failed) and the id of the spot instance (in case the request is in the active state). Executing this operation without parameters will return information about all spot instance requests submitted by the user. Another way of using this operation is to request information about a single request or set of requests.</p>

<p>The supported parameter for this operation is:</p>

<ul>
    <li><i>spot instance Request ID</i>: Specifies the ID of the spot instance request to be described. (optional). This parameter can be a collection of spot instance request ids.</li>
</ul>

<p><b>SOAP Client (Amazon)</b></p>

<p>Operation command and parameters:</p>

<i>ec2-describe-spot-instance-requests [request_id [request_id...]]</i>

<p>This example returns information about current spot instance requests.</p>

<div class="panel"><pre>
$ ec2-describe-spot-instance-requests sir-f102a405
SPOTINSTANCEREQUEST sir-f102a405 0.1 one-time active
2009-12-12T22:58:47+0200 i-3597b470 ami-7d3b6a38 m1.small default
</pre></div>


<p><b>Query Client (Boto)</b></p>
<p>Operation signature and parameters:</p>

<i>boto.ec2.connection.get_all_spot_instance_requests(request_ids=None)</i>
<div class="panel"><pre>
import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Create a spot instances request for ten m1.small instances.
conn.request_spot_instances(0.50, 'myimage.img', count=10, key_name='MyKeypair', instance_type='m1.small')

# Describe spot instance requests, and print id and state
si_reqs = conn.get_all_spot_instance_requests()
print 'Nr. of Requests: ' + str(len(si_reqs))
print 'Id: ' + siReqs[0].id + ' State: ' + si_reqs[0].state
</pre></div>

<p>Output:</p>
<div class="panel"><pre>
Nr. of Requests: 1
Id: sir-f102a405 State: active
</pre></div>

<h4>Canceling Spot Instance Requests</h4>

<p>The Cancel spot instance Requests operation cancels spot instance requests in the "active" or "open" state. Canceled requests aren't considered for fulfillment. However, running instances from a canceled request, will remain running until they are explicitly terminated or the spot price rises above the request bid. This operation can cancel a single request, or a set of requests.</p>

<p>The supported parameter for this operation is:</p>
<ul>
    <li><i>spot instance Request ID</i>: Specifies the ID of the spot instance request to be canceled.  This parameter can be a collection of spot instance request ids.</li>
</ul>

<p><b>SOAP Client (Amazon)</b></p>
<p>Operation command and parameters:</p>
<i>ec2-cancel-spot-instance-requests request_id [request_id...]</i>
<p>Example:</p>
<div class="panel"><pre>
$ ec2-cancel-spot-instance-requests sir-ed3a2006
SPOTINSTANCEREQUEST sir-ed3a2006 canceled
</pre></div>

<p><b>Query Client (Boto)</b></p>
<p>Operation signature and parameters:</p>
<i>cancel_spot_instance_requests(request_ids)</i>

<p>Example:</p>

<div class="panel"><pre>
import boto
from boto.ec2.regioninfo import RegionInfo

# first create a region object and connection
region = RegionInfo(name="nimbus", endpoint="service.hostname.com")
conn = boto.connect_ec2("YOUR_ACCESS_ID_HERE", "YOUR_SECRET_KEY_HERE", port=SERVICE_PORT_NUMBER, region=region)

# Creates a spot instances request for ten m1.small instances.
spot_req = conn.request_spot_instances(0.50, 'myimage.img', count=10, type='one-time', key_name='MyKeypair', instance_type='m1.small')

# Cancel the submitted spot request
si_reqs = conn.cancel_spot_instance_requests([spot_req[0].id])
</pre></div>


<h4>Managing Running Spot Instances</h4>

<p>Spot instances perform exactly like ordinary Nimbus instances while running, and like other Nimbus instances, running spot instances can be managed (described or terminated) by the Nimbus Cloud Client and by usual EC2 API management operations, such as:</p>

<ul>
    <li><i>ec2-describe-instances</i> - Report on currently running instances.</li>
    <li><i>ec2-terminate-instances</i> - Destroy currently running instances.</li>
</ul>

<p>In the "Describe Instances" operation, spot instances will differ from ordinary running instances in the following attributes:</p>

<ul>
    <li><i>Instance Life Cycle</i>: 'spot' if the running instance is a spot instance, null or 'normal' otherwise.</li>
    <li><i>spot instance Request ID</i>: the ID of the spot instance request, if the running instance is a spot instance, null otherwise.</li>
</ul>



<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
        
_NIMBUS_CENTER3_COLUMN_END
_NIMBUS_FOOTER1
_NIMBUS_FOOTER2
_NIMBUS_FOOTER3