Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
5f4cb92
commit 4cb9558
Showing
1 changed file
with
217 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,217 @@ | ||
| --- | ||
| layout: post | ||
| title: DevCloud for CloudStack Development | ||
| excerpt: Xen on VirtualBox | ||
| --- | ||
|
|
||
| [Apache CloudStack](http://incubator.apache.org/cloudstack) development is | ||
| not an easy task, for the simplest of deployments one requires a server where | ||
| the management server, mysql server and NFS server would run, at least | ||
| one host or server which would run a hypervisor (to run virtual machines) or | ||
| would be used for baremetal deployment and some network infrastructure. | ||
|
|
||
| And talk about development, sometimes reproducing a bug can take hours or days | ||
| (been there done that :) and moreover a developer may not have access to such | ||
| an infrastructure all the time. | ||
|
|
||
| ##The Solution | ||
|
|
||
| To solve the problem of infrastructure availability for development and testing, | ||
| earlier this year [Edison](http://www.linkedin.com/pub/disheng-su/5/ab9/90b), | ||
| one of the core committers and PPMC members of Apache CloudStack (incubating), | ||
| created [DevCloud](http://wiki.cloudstack.org/display/COMM/DevCloud). | ||
|
|
||
| `DevCloud` is a virtual appliance shipped as an OVA image which runs on [VirtualBox](http://virtualbox.org) | ||
| (an opensource type-2 or desktop hypervisor) and can be used for CloudStack's | ||
| development and testing. The original DevCloud required 2G of RAM, and ran | ||
| Ubuntu Precise as dom0 over xen.org's Xen server which runs as a VM on VirtualBox. | ||
|
|
||
| A developer would build and deploy CloudStack artifacts (jars, wars) and files | ||
| to DevCloud, deploy database and start the management server inside DevCloud. | ||
| The developer may then use CloudStack running inside DevCloud to add DevCloud as | ||
| a host and whatnot. DevCloud is now used by a lot of people, especially during | ||
| the first release of Apache CloudStack, the 4.0.0-incubating, DevCloud was used | ||
| for the release testing. | ||
|
|
||
| ##My Experiment | ||
|
|
||
| When I tried DevCloud for the first time, I thought it was neat, an awesome all | ||
| in a box solution for offline development. The limitations were; only one host | ||
| could be used that too in basic zone and it would run mgmt server etc. all inside | ||
| DevCloud. I wanted to run mgmt server, MySQL server on my laptop and debug with | ||
| IntelliJ, so I made [my own | ||
| DevCloud](https://cwiki.apache.org/confluence/display/CLOUDSTACK/DIY+DevCloud+Setup) | ||
| setup which would run two XenServers on separate VirtualBox VMs, NFS running on | ||
| a separate VM and all the VMs on a host-only network. | ||
|
|
||
| The `host-only` network in VirtualBox is a special network which is shared by | ||
| all the VMs and the host operating system. My setup allowed me to have two hosts | ||
| so I could do things like VM migration in a cluster etc. But it would crash a lot | ||
| and network won't work. I learnt how bridging in Xen worked and using tcpdump | ||
| found that the packets were dropped but ARP request was allowed, the fix was to | ||
| just enable host-only adapter's promiscuous mode to allow all. I also tried to | ||
| run KVM on VirtualBox, which did not work as KVM does not support PV and requires | ||
| HVM so it cannot run on processors without Intel-VT or Amd-V. None of which is | ||
| emulated by VirtualBox. | ||
|
|
||
| ##Motivation | ||
|
|
||
| CloudStack's build system was changed from Ant to Maven, and this required some | ||
| [changes in DevCloud](https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+devcloud+environment+setup) | ||
| which made it possible to use the original appliance with the new build system. | ||
| The changes were not straight forward so I decided to work on the next iteration | ||
| of DevCloud with the following goals: | ||
|
|
||
| * Two network interfaces, _host-only_ adapter so that the VM is reachable from | ||
| host os and a _NAT_ so VMs can access Internet. | ||
| * Can be used both as an all in one box solution like the original DevCloud but | ||
| the mgmt server and other services can run elsewhere (on host os). | ||
| * Reduce resource requirements, so one could run it in 1G limit. | ||
| * Allow multiple DevCloud VMs hosts. | ||
| * x86 dom0 and xen-i386 so it runs on all host os. | ||
| * Reduce exported appliance (ova) file size. | ||
| * It should be seamless, it should work out of the box. | ||
|
|
||
| ##DevCloud 2.0 | ||
|
|
||
| I started by creating an appliance using Ubuntu 12.04.1 server which failed for me. | ||
| The network interfaces would stop working after reboot and few users reported | ||
| blank screen. I never caught the actual issue, so I tried to create the | ||
| appliance using different distributions including Fedora, Debian and Arch. | ||
| Fedora did not work and stripping down to a bare minimum required a lot of work. | ||
| Arch VM was very small in size but I dropped my idea to work on it as it can be | ||
| unstable, people may not be familiar with pacman and may fail to appreciate the | ||
| simplicity of the distribution. | ||
|
|
||
| Finally, I hit the jackpot with Debian! Debian Wheezy just worked, took me some | ||
| time to create it from scratch (more than ten times) and figure out the correct | ||
| configurations. The new appliance is available for download, [get DevCloud 2.0](http://people.apache.org/~bhaisaab/cloudstack/devcloud/devcloud2.ova) | ||
| (862MB, md5checksum: 94086b3361300c537ab13a60703fc138). | ||
|
|
||
| Install VirtualBox and import the new DevCloud2 appliance and start it. In | ||
| default settings, it is reachable on ip `192.168.56.10` with username `root` and | ||
| password `password`. Next start hacking either inside the DevCloud appliance or | ||
| on your laptop (host os): | ||
|
|
||
| <pre class="prettyprint"> | ||
| # ssh inside DevCloud if building inside it: | ||
| $ ssh -v root@192.168.56.10 | ||
| $ cd to /opt/cloudstack # or any other directory, it does not matter | ||
| # Get the source code: | ||
| $ git pull https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git | ||
| $ cd incubator-cloudstack | ||
| # Build management server: | ||
| $ mvn clean install -P developer | ||
| # Deploy database: | ||
| $ mvn -pl developer,tools/devcloud -Ddeploydb -P developer | ||
| # Run the management server: | ||
| $ mvn -pl client jetty:run | ||
| # In Global Settings check `host` to 192.168.56.1 (or .10 if inside DevCloud) | ||
| # and `system.vm.use.local.storage` to true, restart mgmt server. | ||
| # Set the maximum number of console proxy vms to 0 if you don't need one from | ||
| # CloudStack's global settings, this will save you some RAM. | ||
| # Now add a basic zone with local storage. May be start more DevCloud hosts by | ||
| # importing more appliances and changing default IPs and reboot! | ||
| </pre> | ||
|
|
||
| ##DIY DevCloud | ||
|
|
||
| Install VirtualBox and get the [Debian Wheezy | ||
| 7.0](http://www.debian.org/devel/debian-installer/). I used the netinst i386 | ||
| iso. Create a new VM in VirtualBox with Debian/Linux as the distro, 2G RAM, 20G | ||
| or more disk and two nics: host-only with promiscuous mode "allow-all" and a NAT | ||
| adapter. Next, install a base Debian system with linux-kernel-pae (generic), | ||
| and openssh-server. You may download my [base system from | ||
| here](http://people.apache.org/~bhaisaab/vms/debian-wheezy-basex86.ova). | ||
|
|
||
| Install required tools and Xen-i386: | ||
|
|
||
| <pre class="prettyprint"> | ||
| $ apt-get install git vim tcpdump ebtables --no-install-recommends | ||
| $ apt-get install openjdk-6-jdk genisoimage python-pip mysql-server nfs-kernel-server --no-install-recommends | ||
| $ apt-get install linux-headers-3.2.0-4-686-pae xen-hypervisor-4.1-i386 xcp-xapi xcp-xe xcp-guest-templates xcp-vncterm xen-tools blktap-utils blktap-dkms qemu-keymaps qemu-utils --no-install-recommends | ||
| </pre> | ||
|
|
||
| Remove MySQL password: | ||
| <pre class="prettyprint"> | ||
| $ mysql -u root -p | ||
| > SET PASSWORD FOR root@localhost=PASSWORD(''); | ||
| > exit; | ||
| </pre> | ||
|
|
||
| Setup Xen and XCP/XAPI: | ||
|
|
||
| <pre class="prettyprint"> | ||
| $ echo "bridge" > /etc/xcp/network.conf | ||
| $ update-rc.d xendomains disable | ||
| $ echo TOOLSTACK=xapi > /etc/default/xen | ||
| $ sed -i 's/GRUB_DEFAULT=.\+/GRUB_DEFAULT="Xen 4.1-i386"/' /etc/default/grub | ||
| $ sed -i 's/GRUB_CMDLINE_LINUX=.\+/GRUB_CMDLINE_LINUX="apparmor=0"\nGRUB_CMDLINE_XEN="dom0_mem=400M,max:500M dom0_max_vcpus=1"/' /etc/default/grub | ||
| $ update-grub | ||
| $ sed -i 's/VNCTERM_LISTEN=.\+/VNCTERM_LISTEN="-v 0.0.0.0:1"/' /usr/lib/xcp/lib/vncterm-wrapper | ||
| $ cat > /usr/lib/xcp/plugins/echo << EOF | ||
| #!/usr/bin/env python | ||
|
|
||
| # Simple XenAPI plugin | ||
| import XenAPIPlugin, time | ||
|
|
||
| def main(session, args): | ||
| if args.has_key("sleep"): | ||
| secs = int(args["sleep"]) | ||
| time.sleep(secs) | ||
| return "args were: %s" % (repr(args)) | ||
|
|
||
| if __name__ == "__main__": | ||
| XenAPIPlugin.dispatch({"main": main}) | ||
| EOF | ||
|
|
||
| $ chmod -R 777 /usr/lib/xcp | ||
| $ mkdir -p /root/.ssh | ||
| $ ssh-keygen -A -q | ||
| </pre> | ||
|
|
||
| Network settings, /etc/network/interfaces: | ||
|
|
||
| <pre class="prettyprint"> | ||
| auto lo | ||
| iface lo inet loopback | ||
|
|
||
| auto eth0 | ||
| iface eth0 inet manual | ||
|
|
||
| allow-hotplug eth1 | ||
| iface eth1 inet manual | ||
|
|
||
| auto xenbr0 | ||
| iface xenbr0 inet static | ||
| bridge_ports eth0 | ||
| address 192.168.56.10 | ||
| netmask 255.255.255.0 | ||
| network 192.168.56.0 | ||
| broadcast 192.168.56.255 | ||
| gateway 192.168.56.1 | ||
| dns_nameservers 8.8.8.8 8.8.4.4 | ||
| post-up route del default gw 192.168.56.1; route add default gw 192.168.56.1 metric 100; | ||
|
|
||
| auto xenbr1 | ||
| iface xenbr1 inet dhcp | ||
| bridge_ports eth1 | ||
| dns_nameservers 8.8.8.8 8.8.4.4 | ||
| post-up route add default gw 10.0.3.2 | ||
| </pre> | ||
|
|
||
| Preseed the SystemVM templates in `/opt/storage/secondary`, follow directions | ||
| from | ||
| [here](http://incubator.apache.org/cloudstack/docs/en-US/Apache_CloudStack/4.0.0-incubating/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template). | ||
| Configure NFS server and local storage. | ||
|
|
||
| <pre class="prettyprint"> | ||
| $ mkdir -p /opt/storage/secondary | ||
| $ mkdir -p /opt/storage/primary | ||
| $ hostuuid=`xe host-list |grep uuid|awk '{print $5}'` | ||
| $ xe sr-create host-uuid=$hostuuid name-label=local-storage shared=false type=file device-config:location=/opt/storage/primary | ||
| $ echo "/opt/storage/secondary *(rw,no_subtree_check,no_root_squash,fsid=0)" > /etc/exports | ||
| $ #preseed systemvm template, may be copy files from devcloud's /opt/storage/secondary | ||
| $ /etc/init.d/nfs-kernel-server restart | ||
| </pre> | ||
|
|