AWS ephemeral storage #858
Comments
|
There may be more nuances in the general case but all of the ephemeral drives for a given instance flavor should already be available as block devices within /dev/. I use the ephemeral drives on instances fired up with terraform currently. Raiding, formatting, and mounting them all occurs in postlaunch scripting. |
|
After some research, I was able to add both EBS and ephemeral storage using the following config (on an m3.xlarge): If I leave out the latter two |
|
I've been thinking about this. I think if you don't provide a block_device configuration that it'll default to providing all of the ephemerals to the instance. That's the case I'm seeing. The case you're seeing is that if you provide a block_device configuration then you need to make sure the ephemerals are attached as well as the ebs drives. I'd say that all the ephemerals available to a given instance type should always be available regardless of whether or not you intend to attach ebs drives on launch. I think the trick is making sure that you don't attach an ebs drive on one of the /dev/{sd,xvd}* devices that the ephemerals default to. That's a bit tricky because they are different devices on different instance types. I generally ping the ec2 metadata service to see what ephemeral drives are available to me and what their device names are before I subsequently auto raid0 them (in most cases). If I get a chance I'll take a peek at the current implementation and see what might need to be done to get all the ephemerals attached by default. IMO, there is no reason to ever omit any storage that is freely available to an instance flavor. It should be there by default if you want to use it. |
|
Thanks for looking into this, @knuckolls . It looks like this is very much related to #859 as well. Upon a |
AWS provides a single `BlockDeviceMapping` to manage three different kinds of block devices: (a) The root volume (b) Ephemeral storage (c) Additional EBS volumes Each of these types has slightly different semantics [1]. (a) The root volume is defined by the AMI; it can only be customized with `volume_size`, `volume_type`, and `delete_on_termination`. (b) Ephemeral storage is made available based on instance type [2]. It's attached automatically if _no_ block device mappings are specified, and must otherwise be defined with block device mapping entries that contain only DeviceName set to a device like "/dev/sdX" and VirtualName set to "ephemeralN". (c) Additional EBS volumes are controlled by mappings that omit `virtual_name` and can specify `volume_size`, `volume_type`, `delete_on_termination`, `snapshot_id`, and `encryption`. After deciding to ignore root block devices to fix #859, we had users with configurations that were attempting to manage the root block device chime in on #913. Terraform does not have the primitives to be able to properly handle a single collection of resources that is partially managed and partially computed, so our strategy here is to break out logical sub-resources for Terraform and hide the BlockDeviceMapping inside the provider implementation. Now (a) is supported by the `root_block_device` sub-resource, and (b) and (c) are still both merged together under `block_device`, though I have yet to see ephemeral block devices working properly. Looking into possibly separating out `ephemeral_block_device` and `ebs_block_device` sub-resources as well, which seem like the logical next step. We'll wait until the next big release for this, though, since it will break backcompat. [1] http://bit.ly/ec2bdmap [2] http://bit.ly/instancestorebytype Fixes #913 Refs #858
Instance block devices are now managed by three distinct sub-resources: * `root_block_device` - introduced previously * `ebs_block_device` - all additional ebs-backed volumes * `ephemeral_block_device` - instance store / ephemeral devices The AWS API support around BlockDeviceMapping is pretty confusing. It's a single collection type that supports these three members each of which has different fields and different behavior. My biggest hiccup came from the fact that Instance Store volumes do not show up in any response BlockDeviceMapping for any EC2 Describe* AWS API calls. They're only available from the instance meta-data service as queried from inside the node. Also note this removes `block_device` altogether for a clean break from old configs. New configs will need to sort their `block_device` declarations into the three new types. Fixes #858
Instance block devices are now managed by three distinct sub-resources: * `root_block_device` - introduced previously * `ebs_block_device` - all additional ebs-backed volumes * `ephemeral_block_device` - instance store / ephemeral devices The AWS API support around BlockDeviceMapping is pretty confusing. It's a single collection type that supports these three members each of which has different fields and different behavior. My biggest hiccup came from the fact that Instance Store volumes do not show up in any response BlockDeviceMapping for any EC2 Describe* AWS API calls. They're only available from the instance meta-data service as queried from inside the node. Also note this removes `block_device` altogether for a clean break from old configs. New configs will need to sort their `block_device` declarations into the three new types. Fixes #858
AWS provides a single `BlockDeviceMapping` to manage three different kinds of block devices: (a) The root volume (b) Ephemeral storage (c) Additional EBS volumes Each of these types has slightly different semantics [1]. (a) The root volume is defined by the AMI; it can only be customized with `volume_size`, `volume_type`, and `delete_on_termination`. (b) Ephemeral storage is made available based on instance type [2]. It's attached automatically if _no_ block device mappings are specified, and must otherwise be defined with block device mapping entries that contain only DeviceName set to a device like "/dev/sdX" and VirtualName set to "ephemeralN". (c) Additional EBS volumes are controlled by mappings that omit `virtual_name` and can specify `volume_size`, `volume_type`, `delete_on_termination`, `snapshot_id`, and `encryption`. After deciding to ignore root block devices to fix hashicorp#859, we had users with configurations that were attempting to manage the root block device chime in on hashicorp#913. Terraform does not have the primitives to be able to properly handle a single collection of resources that is partially managed and partially computed, so our strategy here is to break out logical sub-resources for Terraform and hide the BlockDeviceMapping inside the provider implementation. Now (a) is supported by the `root_block_device` sub-resource, and (b) and (c) are still both merged together under `block_device`, though I have yet to see ephemeral block devices working properly. Looking into possibly separating out `ephemeral_block_device` and `ebs_block_device` sub-resources as well, which seem like the logical next step. We'll wait until the next big release for this, though, since it will break backcompat. [1] http://bit.ly/ec2bdmap [2] http://bit.ly/instancestorebytype Fixes hashicorp#913 Refs hashicorp#858
Instance block devices are now managed by three distinct sub-resources: * `root_block_device` - introduced previously * `ebs_block_device` - all additional ebs-backed volumes * `ephemeral_block_device` - instance store / ephemeral devices The AWS API support around BlockDeviceMapping is pretty confusing. It's a single collection type that supports these three members each of which has different fields and different behavior. My biggest hiccup came from the fact that Instance Store volumes do not show up in any response BlockDeviceMapping for any EC2 Describe* AWS API calls. They're only available from the instance meta-data service as queried from inside the node. Also note this removes `block_device` altogether for a clean break from old configs. New configs will need to sort their `block_device` declarations into the three new types. Fixes #858
Instance block devices are now managed by three distinct sub-resources: * `root_block_device` - introduced previously * `ebs_block_device` - all additional ebs-backed volumes * `ephemeral_block_device` - instance store / ephemeral devices The AWS API support around BlockDeviceMapping is pretty confusing. It's a single collection type that supports these three members each of which has different fields and different behavior. My biggest hiccup came from the fact that Instance Store volumes do not show up in any response BlockDeviceMapping for any EC2 `Describe*` API calls. They're only available from the instance meta-data service as queried from inside the node. This removes `block_device` altogether for a clean break from old configs. New configs will need to sort their `block_device` declarations into the three new types. The field has been marked `Removed` to indicate this to users. With the new block device format being introduced, we need to ensure Terraform is able to properly read statefiles written in the old format. So we use the new `helper/schema` facility of "state migrations" to transform statefiles in the old format to something that the current version of the schema can use. Fixes #858
Instance block devices are now managed by three distinct sub-resources: * `root_block_device` - introduced previously * `ebs_block_device` - all additional ebs-backed volumes * `ephemeral_block_device` - instance store / ephemeral devices The AWS API support around BlockDeviceMapping is pretty confusing. It's a single collection type that supports these three members each of which has different fields and different behavior. My biggest hiccup came from the fact that Instance Store volumes do not show up in any response BlockDeviceMapping for any EC2 `Describe*` API calls. They're only available from the instance meta-data service as queried from inside the node. This removes `block_device` altogether for a clean break from old configs. New configs will need to sort their `block_device` declarations into the three new types. The field has been marked `Removed` to indicate this to users. With the new block device format being introduced, we need to ensure Terraform is able to properly read statefiles written in the old format. So we use the new `helper/schema` facility of "state migrations" to transform statefiles in the old format to something that the current version of the schema can use. Fixes #858
|
Came across this while trying to figure out what I should do in order to have an |
|
Hi @mihasya - if you don't specify any block device mappings in your Launch Configuration, the Block Device Mappings of your AMI will apply. This is more of an AWS behavior question, but I believe that since the available ephemeral devices change per instance type, you'd need to specify a "superset" block device mapping mentioning any possible ephemeral devices in order to ensure that they'd all be available. Probably a good question for your AWS support rep though! |
|
@phinze I'm literally about to type out mappings for ephemeral[0-3] and see if it works on an instance type with only 2 drives :) Would love to not have to do this by hand for every cluster.. |
|
@mihasya definitely keep us posted! If it turns out to be boilerplate applicable anywhere it might be worth wrapping up into a community module or something |
|
@phinze yeah if I end up having to do an instance size-to-volume count mapping, I'll publish it for sure. |
|
@phinze prefilling Inside the module, I was able to do the following: which means that on any instance, all ephemeral devices will be available on |
|
@mihasya, I had to do the same (define possible ephemeral devices) to get them attached, but it still wants to recreate the instance every time (which sounds exactly like root devices in #859, unless this is possibly an IAM permissions problem). Does a plan keep your instance, or want to destroy/add? Here's the line of interest in the output: According to the AWS docs it would appear you can only get the ephemeral device mappings via metadata query, so we may need to ignore them to have plan work as needed (unless I'm missing something). |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
It looks like #440 just supports EBS volumes, and not ephemeral storage. If that's the case, does Terraform support ephemeral storage at all? We treat our instances as being disposable, and make extensive use of ephemeral storage; such support would be great.
Thanks!
-Ameir
The text was updated successfully, but these errors were encountered: