Skip to content
Permalink
Browse files
This closes #19
  • Loading branch information
ahgittin committed Mar 5, 2016
2 parents a41e0d7 + 2e02de8 commit bf43bd0042bcb0ed6012f5419f8150be22051631
Show file tree
Hide file tree
Showing 4 changed files with 189 additions and 90 deletions.
@@ -1,85 +1,6 @@
---
title: Cloud Setup
title: More Clouds
layout: website-normal
---

To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the "identity" and
"credential" in Brooklyn terminology.

For private clouds (and for some clouds being targeted using a standard API), the "endpoint"
must also be specified, which is the cloud's URL.

The [jclouds guides](https://jclouds.apache.org/guides) includes documentation on configuring
different clouds.


## AWS

### Credentials

AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential
respectively.

These keys are the way for any programmatic mechanism to access the AWS API.

To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws)
and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html).

An example of the expected format is shown below:

brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST
brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm


### Tidying up after jclouds

Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see
https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group
being deleted until there are no VMs using it. However, there is eventual consistency for
recording which VMs still reference those security groups. After deleting the VM, it can sometimes
take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but
does not block for longer.

There is utility written by Cloudsoft for deleting these unused resources:
http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds.


## Google Compute Engine

### Credentials

Google Compute Engine (GCE) uses a service account e-mail address for the identity, and a private key
as the credential.

To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google).

An example of the expected format is shown below (note the credential is one long line,
with `\n` to represent the new line characters):

brooklyn.location.jclouds.google-compute-engine.identity=123456789012@developer.gserviceaccount.com
brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY-----


### Quotas

GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas).

It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en).


### Networks

GCE accounts often have a limit to the number of networks that can be created. One work around
is to manually create a network with the required open ports, and to refer to that named network
in Brooklyn's location configuration.

To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1).

For example, for dev/demo purposes an "everything" network could be created that opens all ports.

| Name | everything |
| Description | opens all tcp ports |
| Source IP Ranges | 0.0.0.0/0 |
| Allowed protocols and ports | tcp:0-65535 and udp:0-65535 |


This page has moved. See [More Clouds](more-clouds.html) instead.
@@ -6,7 +6,7 @@ children:
- { section: Inheritance and Named Locations, title: Named Locations }
- { section: Localhost }
- { section: BYON }
- cloud-credentials.md
- { title: More on Clouds, path: more-clouds.md }
- more-locations.md
- location-customizers.md
- ssh-keys.md
@@ -62,8 +62,12 @@ brooklyn.location.jclouds.aws-ec2.credential=s3cr3tsq1rr3ls3cr3tsq1rr3ls3cr3tsq1
And in this case you can reference the location in YAML with `location: jclouds:aws-ec2`.
The Getting Started [template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties) contains more examples
of configuring cloud endpoints, including information on credential types used in different clouds.
Brooklyn irons out many of the differences between clouds so that blueprints run similarly
in a wide range of locations, including setting up access and configuring images and machine specs.
The configuration options are described in more detail below.
In some cases, cloud providers have special features or unusual requirements.
These are outlined in **[More Details for Specific Clouds](more-clouds.html)**.
#### OS Initial Login and Setup
@@ -91,7 +95,8 @@ Following a successful logon, Brooklyn performs the following steps to configure
1. install the local user's `~/.ssh/id_rsa.pub` as an `authorized_keys` on the new machine,
to make it easy for the operator to `ssh` in
(override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair will be generated;
(override with `privateKeyFile`; or if there is no `id_{r,d}sa{,.pub}` an ad hoc keypair will be generated
for the regular Brooklyn user;
if there is a passphrase on the key, this must be supplied)
1. give `sudo` access to the newly created user (override with `grantUserSudo: false`)
@@ -124,8 +129,10 @@ For more keys and more detail on the keys below, see
- Specific Security Groups can be specified using `securityGroups`, as a list of strings (the existing security group names),
or `inboundPorts` can be set, as a list of numeric ports (selected clouds only)

- A specific existing key pair known at the cloud to use can be specified with `keyPair`
(selected clouds only)
- Where a key pair is registered with a target cloud for logging in to machines,
Brooklyn can be configured to request this when provisioning VMs by setting `keyPair` (selected clouds only).
Note that if this `keyPair` does not correspond your default `~/.ssh/id_rsa`, you must typically
also specify the corresponding `loginUser.privateKeyFile` as a file or URL accessible from Brooklyn.

- A specific VM name (often the hostname) base to be used can be specified by setting `groupId`.
By default, this name is constructed based on the entity which is creating it,
@@ -216,7 +223,7 @@ For more keys and more detail on the keys below, see
This setting prevents scripts executed on the VMs from being deleted on completion.
Note that some scripts run periodically so this can eventually fill a disk; it should only be used for dev/test.

###### Custom template options
###### Custom Template Options

jclouds supports many additional options for configuring how a virtual machine is created and deployed, many of which
are for cloud-specific features and enhancements. Brooklyn supports some of these, but if what you are looking for is
@@ -445,7 +452,7 @@ provisioned. See `FixedListMachineProvisioningLocation.MACHINE_CHOOSER`.

### Other Location Topics

* [Cloud Credentials](cloud-credentials.html)
* [More Details on Specific Clouds](more-clouds.html)
* [More Locations](more-locations.html)
* [Location Customizers](location-customizers.html)
* [SSH Keys](ssh-keys.html)
@@ -1,5 +1,5 @@
---
title: Location customizers
title: Location Customizers
layout: website-normal
---

@@ -0,0 +1,171 @@
---
title: More Details on Specific Clouds
layout: website-normal
---

To connect to a Cloud, Brooklyn requires appropriate credentials. These comprise the `identity` and
`credential` in Brooklyn terminology.

For private clouds (and for some clouds being targeted using a standard API), the `endpoint`
must also be specified, which is the cloud's URL. For public clouds, Brooklyn comes preconfigured
with the endpoints, but many offer different choices of the `region` where you might want to deploy.

Clouds vary in the format of the identity, credential, endpoint, and region.
Some also have their own idiosyncracies. More details for configuring some common clouds
is included below. You may also find these sources helpful:

* The **[template brooklyn.properties]({{ site.path.guide }}/start/brooklyn.properties)** file
in the Getting Started guide
contains numerous examples of configuring specific clouds,
including the format of credentials and options for sometimes-fiddly private clouds.
* The **[jclouds guides](https://jclouds.apache.org/guides)** describes low-level configuration
sometimes required for various clouds.



## Amazon Web Services (AWS)

### Credentials

AWS has an "access key" and a "secret key", which correspond to Brooklyn's identity and credential
respectively.

These keys are the way for any programmatic mechanism to access the AWS API.

To generate an access key and a secret key, see [jclouds instructions](http://jclouds.apache.org/guides/aws)
and [AWS IAM instructions](http://docs.aws.amazon.com/IAM/latest/UserGuide/ManagingCredentials.html).

An example of the expected format is shown below:

brooklyn.location.jclouds.aws-ec2.identity=ABCDEFGHIJKLMNOPQRST
brooklyn.location.jclouds.aws-ec2.credential=abcdefghijklmnopqrstu+vwxyzabcdefghijklm


### Tidying up after jclouds

Security groups are not always deleted by jclouds. This is due to a limitation in AWS (see
https://issues.apache.org/jira/browse/JCLOUDS-207). In brief, AWS prevents the security group
being deleted until there are no VMs using it. However, there is eventual consistency for
recording which VMs still reference those security groups: after deleting the VM, it can sometimes
take several minutes before the security group can be deleted. jclouds retries for 3 seconds, but
does not block for longer.

There is utility written by Cloudsoft for deleting these unused resources:
http://www.cloudsoftcorp.com/blog/2013/03/tidying-up-after-jclouds.


### Using Subnets and Security Groups

Apache Brooklyn can run with AWS VPC and both public and private subnets.
Simply provide the `subnet-a1b2c3d4` as the `networkName` when deploying:

location:
jclouds:aws-ec2:
region: us-west-1
networkName: subnet-a1b2c3d4 # use your subnet ID

Subnets are typically used in conjunction with security groups.
Brooklyn does *not* attempt to open additional ports
when private subnets or security groups are supplied,
so the subnet and ports must be configured appropriately for the blueprints being deployed.
You can configure a default security group with appropriate (or all) ports opened for
access from the appropriate (or all) CIDRs and security groups,
or you can define specific `securityGroups` on the location
or as `provisioning.properties` on the entities.

Make sure that Brooklyn has access to the machines under management.
This includes SSH, which might be done with a public IP created with inbound access
on port 22 permitted for a CIDR range including the IP from which Brooklyn contacts it.
Alternatively you can run Brooklyn on a machine in that same subnet, or
set up a VPN or jumphost which Brooklyn will use.


### EC2 "Classic" Problems with VPC-only Hardware Instance Types

If you have a pre-2014 Amazon account, it is likely configured in some regions to run in "EC2 Classic" mode
by default, instead of the more modern "VPC" default mode. This can cause failures when requesting certain hardware
configurations because many of the more recent hardware "instance types" only run in "VPC" mode.
For instance when requesting an instance with `minRam: 8gb`, Brooklyn may opt for an `m4.large`,
which is a VPC-only instance type. If you are in a region configured to use "EC2 Classic" mode,
you may see a message such as this:

400 VPCResourceNotSpecified: The specified instance type can only be used in a VPC.
A subnet ID or network interface ID is required to carry out the request.

This is a limitation of "legacy" accounts. The easiest fixes are either:

* specify an instance type which is supported in classic, such as `m3.xlarge` (see below)
* move to a different region where VPC is the default
(`eu-central-1` should work as it *only* offers VPC mode,
irrespective of the age of your AWS account)
* get a new AWS account -- "VPC" will be the default mode
(Amazon recommend this and if you want to migrate existing deployments
they provide [detailed instructions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/vpc-migrate.html))

To understand the situation, the following resources may be useful:

* Background on VPC vs Classic: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html)
* Good succinct answers to FAQs: [http://aws.amazon.com/vpc/faqs/#Default_VPCs]()
* Check if a region in your account is "VPC" or "Classic": [http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html#default-vpc-availability]()
* Regarding instance types:
* All instance types: [https://aws.amazon.com/ec2/instance-types]()
* Those which require VPC: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html#vpc-only-instance-types]()

If you want to solve this problem with your existing account,
you can create a VPC and instruct Brooklyn to use it:

1. Use the "Start VPC Wizard" option in [the VPC dashboard](https://console.aws.amazon.com/vpc),
making sure it is for the right region, and selecting a "Single Public Subnet".
(More information is in [these AWS instructions](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html#create-a-vpc).)
2. Once the VPC is created, open the "Subnets" view and modify the "Public subnet"
so that it will "Auto-assign Public IP".
3. Next click on the "Security Groups" and find the `default` security group for that VPC.
Modify its "Inbound Rules" to allow "All traffic" from "Anywhere".
(Or for more secure options, see the instructions in the previous section,
"Using Subnets".)
4. Finally make a note of the subnet ID (e.g. `subnet-a1b2c3d4`) for use in Brooklyn.

You can then deploy blueprints to the subnet, allowing VPC hardware instance types,
by specifying the subnet ID as the `networkName` in your YAML blueprint.
This is covered in the previous section, "Using Subnets".


## Google Compute Engine (GCE)

### Credentials

GCE uses a service account e-mail address for the identity and a private key as the credential.

To obtain these from GCE, see the [jclouds instructions](https://jclouds.apache.org/guides/google).

An example of the expected format is shown below.
Note that when supplying the credential in a properties file, it should be one long line
with `\n` representing the new line characters:

brooklyn.location.jclouds.google-compute-engine.identity=123456789012@developer.gserviceaccount.com
brooklyn.location.jclouds.google-compute-engine.credential=-----BEGIN RSA PRIVATE KEY-----\nabcdefghijklmnopqrstuvwxyznabcdefghijk/lmnopqrstuvwxyzabcdefghij\nabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghij+lm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklm\nnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxy\nzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk\nlmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\nxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghi\njklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstu\nvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefg\nhijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrs\ntuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcde\nfghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvw\n-----END RSA PRIVATE KEY-----


### Quotas

GCE accounts can have low default [quotas](https://cloud.google.com/compute/docs/resource-quotas).

It is easy to requesta quota increase by submitting a [quota increase form](https://support.google.com/cloud/answer/6075746?hl=en).


### Networks

GCE accounts often have a limit to the number of networks that can be created. One work around
is to manually create a network with the required open ports, and to refer to that named network
in Brooklyn's location configuration.

To create a network, see [GCE network instructions](https://cloud.google.com/compute/docs/networking#networks_1).

For example, for dev/demo purposes an "everything" network could be created that opens all ports.

|| Name || everything |
|| Description || opens all tcp ports |
|| Source IP Ranges || 0.0.0.0/0 |
|| Allowed protocols and ports || tcp:0-65535 and udp:0-65535 |


0 comments on commit bf43bd0

Please sign in to comment.