hs1.8xlarge only mounting 4 of the 24 instance stores #202

Closed
jstjohn opened this Issue Jan 18, 2013 · 6 comments

3 participants

@jstjohn

When I launch an hs1.8xlarge instance, it should have 24 2TB drives available in /dev. However there are only 4 that show up. I confirmed that this is the case with AWS support. When you launch the starcluster AMI from the web console though, it comes up with all 24 drives mounted.

Starcluster or boto must not be doing the same kinds of steps to make sure that all drives that need to be mounted are mounted. I am not sure what the workaround is for this. The AWS support people recommended booting up an instance from the web console and taking an image of it from there. Then maybe when it is booted up in the future from the starcluster console, it might work. I have not yet tested this theory.

@egafni

I also ran into this problem, and monkey patched my code to get around this.

The problem is that starcluster.awsutils run_instances is not passing a BlockDeviceMap to boto.ec2.connection.run_instances().

Here's how I modified the awsutils.run_instances method

def run_instances(self, image_id, instance_type='m1.small', min_count=1,
                  max_count=1, key_name=None, security_groups=None,
                  placement=None, user_data=None, placement_group=None):

    #modifications by Erik
    if instance_type == 'hs1.8xlarge':
        from boto.ec2.blockdevicemapping import BlockDeviceType, BlockDeviceMapping
        map = BlockDeviceMapping()
        for i in range(0,24):
            t = BlockDeviceType()
            t.virtual_name = 'ephemeral{0}'.format(i)
            map['/dev/sd{0}1'.format(chr(ord('b')+i))] = t

    return self.conn.run_instances(image_id, instance_type=instance_type,
                                   min_count=min_count,
                                   max_count=max_count,
                                   key_name=key_name,
                                   security_groups=security_groups,
                                   placement=placement,
                                   user_data=user_data,
                                   placement_group=placement_group,block_device_map = map)

Details for the boto side are here: https://groups.google.com/forum/?fromgroups=#!topic/boto-users/7igfHM9CChk

@jstjohn
@egafni

No problem. I also just wrote a starcluster plugin to install gluster and setup all the ephemeral drives of the hs1 as a 48TB shared mount point across the cluster. If you'd like to use it come to the #starcuster channel on irc

@jstjohn
@egafni

Cool, I'm usually idleing in the channel while working (which is most of the time these days). It'd be nice to have someone else playing with gluster setups as there's lots of configurations to try. Yesterday I was using it to create a 2TB share on the hi1 SSD instances but decided to move to the hs1 instance today.

@egafni

Sorry i missed you, i'm erik49

@jtriley jtriley added a commit that closed this issue Jan 25, 2013
@jtriley add all ephemeral drives when launching instances
Generalized create_root_block_device_map to create_block_device_map with
new kwarg root_snapshot_id and num_ephemeral_drives.

The num_ephemeral_drives kwarg is 24 by default which covers all
instance types. Any types that have less than 24 ephemeral drives
available will simply ignore the rest in the block device map.

Updated request_instances() to always add *all* ephemeral drives if
block_device_map is not specified.

closes gh-202
184da7f
@jtriley jtriley closed this in 184da7f Jan 25, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment