Permalink
Find file
Fetching contributors…
Cannot retrieve contributors at this time
136 lines (99 sloc) 5.73 KB

These notes are still disordered. Ping @mrflip if you have questions.

EBS Volumes for a persistent HDFS

  • I HIGHLY recommend making the drives be XFS formatted. Mount options “defaults,nouuid,noatime” give good results.
  • Here’s a sample cluster_ebs_volumes.json (the contents of the cluster_ebs_volumes databag):

  [
  {"id":"zaius",
    "slave":[
      [ { "device": "/dev/sdj1", "volume_id": "vol-7725a61e", "mount_point": "/ebs1", "type":"xfs", "options": "defaults,nouuid,noatime" } ]
    ],
    "master": [
      [ { "device": "/dev/sdj",  "volume_id": "vol-290ed840", "mount_point": "/ebs1", "type":"xfs", "options": "defaults,nouuid,noatime" } ]
    ]
  }
  ]

Volume Creation

First, you need to make one or more EBS volumes for each node in your cluster.

Option #1: Use the Cloudera scripts

If you have the Cloudera hadoop-ec2 scripts, you can use them to create the volumes, attach them, and generate a .json file that you can paste into the EBS mappings databag.

  • Note: you must incorporate the hadoop-ec2 ec2-storage-xxx.json file into the cluster_ebs_volumes.json databag. Additionally, you must add an “id”:“CLUSTER_NAME” field into the hash. See cluster_chef/databags/cluster_ebs_volumes.json for comparison.
  • Make sure the created drives are xfs. If not, use hadoop-ec2 to attach the volumes and then run a script acrosss all the cluster machines to reformat them.

Option 2 pt1: How to create an EBS Volume

  • Create a filesystem:

    $ sudo mkfs.xfs -f /dev/sdh1
    meta-data=/dev/sdh1              isize=256    agcount=4, agsize=26214062 blks
             =                       sectsz=512   attr=2
    data     =                       bsize=4096   blocks=104856247, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0
    log      =internal log           bsize=4096   blocks=51199, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
  • mount it:
    
        $ sudo mkdir /ebs_prep
        $ sudo mount -t xfs -o defaults,nouuid,noatime /dev/sdh1 /ebs_prep
    
  • If you like, poke a file into the drive saying when it was made. sudo bash -c 'cat > /ebs_prep/created-at-`date +%Y%m%d`'
  • detach
  • snapshot with a name like zaius+raw+000+000+/dev/sdh+/ebs_prep+20100620 — note the snapshot id (eg snap-6a1b8202) and wait for it to finish.
  • Create volumes: ec2-create-volume --size 400 --snapshot snap-6a1b8202 --availability-zone us-east-1d

to also partition

But if you’d like to, see http://support.rightscale.com/06-FAQs/FAQ_0012_-How_do_I_partition,_format_and_mount_an_EBSvolume%3F

  • create a new volume
  • attach to instance, for example as /dev/sdh
  • Partion the volume: (This example creates a single primary partition that uses the entire volume.)
    • run sudo fdisk /dev/sdh
    • Enter m for a menu
    • Enter n to add a new partion
    • Enter p for a primary partition
    • Enter 1 for the first partion on the disk
    • Select Enter to accept the default First Cylinder
    • Select Enter to accept the default Last Cylinder
    • Enter w to write the partition table and exit fdisk

Resize on mount


ip-10-218-47-247 ~$ sudo mount -t xfs /dev/sdj /ebs_prep
ip-10-218-47-247 ~$ sudo xfs_growfs /ebs_prep
meta-data=/dev/sdj               isize=256    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 262144 to 41943040

Option 2 pt2: OK, now create {N} EBS Volumes

  • make 1gb xfs drive, attach, partition, format, detach, snapshot
  • use snapshot to make 1 250GB EBS volume
  • use xfsgrow to make it actually occupy the 250 GB
  • once that works, make N more and bring up your cluster.

Using Broham to have instances auto-number themselves.

We need each zaius-slave to know what order it is within the cluster, so that it grabs its own drives. Broham is a workable (but not fully satisfactory) solution.:

  • gem install broham,
  • Log into the aws console and check in as a SimpleDB user. (You have to click through a license agreement, it should approve you within minutes)
  • Follow the setup instructions in the Broham readme file. Specifically, you need to jump into IRC and do

   require 'configliere' ; Settings.read(File.expand_path('~/.hadoop-ec2/poolparty.yaml'))
   require 'broham'
   Broham.establish_connection Settings
   Broham.create_domain

sudo bash -c ‘export HOSTNAME=gibbon.infinitemonkeys.info ; PUBLIC_IP=204.236.225.16 ; echo $HOSTNAME > /etc/hostname ; hostname -F /etc/hostname ; sysctl -w kernel.hostname=$HOSTNAME ; sed -i “s/127.0.0.1 *localhost/127.0.0.1 $HOSTNAME `hostname -s` localhost/” /etc/hosts ; if grep -q $PUBLIC_IP /etc/hosts ; then true ; else echo $PUBLIC_IP $HOSTNAME `hostname -s ` >> /etc/hosts ; fi’

Use chef to attach and mount drives

Get one volume working

Create a databag for the volumes

edit databags/cluster_ebs_volumes.json to suit.

Keep running chef-client until it attaches

chef client should

  • attach as /dev/sdj (etc)
  • mount as /ebs1 (… and so on)
  • be added as a data dir into the /etc/hadoop/conf/site-hdfs.xml
  • create and chmod the appropriate subdirectories

You might need to manually restart the datanode.