Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Initial commit of files

  • Loading branch information...
commit 5f2b95e70abb0d4549a731c9f7e0f6ddbd35cba1 1 parent 6fc15b8
Khalid Ahmed authored
Showing with 7,356 additions and 3 deletions.
  1. +202 −0 LICENSE
  2. +61 −3 README.md
  3. +79 −0 bin/onecmd
  4. +485 −0 bin/oneenv
  5. +400 −0 bin/oneenvd
  6. +617 −0 bin/oneenvd-gs
  7. +554 −0 bin/svccontroller.sh
  8. +29 −0 cgi-bin/updateappstatus.sh
  9. +29 −0 cgi-bin/updatevmstatus.sh
  10. +73 −0 context/balance.sh
  11. +106 −0 context/cassandra.sh
  12. +47 −0 context/cloudfoundry.sh
  13. +130 −0 context/cloudfoundry_scratch.sh
  14. +18 −0 context/cloudfoundrydea.sh
  15. +77 −0 context/dhcpnetwork.sh
  16. +185 −0 context/hbasemaster.sh
  17. +53 −0 context/hbaseslave.sh
  18. +49 −0 context/init.sh
  19. +72 −0 context/jenkins.sh
  20. +28 −0 context/jenkinsslave.sh
  21. +26 −0 context/network.sh
  22. +58 −0 context/redis.sh
  23. +51 −0 context/redisslave.sh
  24. +20 −0 context/tomcat.sh
  25. +480 −0 docs/GETTING_STARTED.txt
  26. +138 −0 docs/INSTALL.txt
  27. +340 −0 etc/config.rb
  28. +38 −0 etc/global.rb
  29. +23 −0 etc/oneenv.conf
  30. +8 −0 etc/system.conf
  31. +115 −0 lib/ds_manager.rb
  32. +719 −0 lib/env_manager.rb
  33. +157 −0 lib/gs_message.rb
  34. +104 −0 lib/gs_proxy.rb
  35. +412 −0 lib/job_manager.rb
  36. +131 −0 lib/plugin_manager.rb
  37. +415 −0 lib/policy_manager.rb
  38. +234 −0 lib/time_window.rb
  39. +52 −0 misc/createschema.sh
  40. +3 −0  misc/start-oneenvd-gs.sh
  41. +3 −0  misc/start-oneenvd.sh
  42. +19 −0 misc/update.sh
  43. +1 −0  packages/authorized_keys
  44. BIN  packages/balance
  45. +18 −0 packages/core-site.xml
  46. +44 −0 packages/hbase-site.xml
  47. +26 −0 packages/hdfs-site.xml
  48. +93 −0 packages/jenkins.config.xml
  49. +144 −0 packages/l7setup.sh
  50. +18 −0 packages/mysite.conf
  51. +106 −0 plugins/Balance.rb
  52. +33 −0 plugins/MemSize.rb
  53. +33 −0 templates/ubuntu1104.vm
View
202 LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
View
64 README.md
@@ -1,4 +1,62 @@
-OpenNebula-Carina
-=================
+Introduction
+------------
+
+The OpenNebula Environment Manager (oneenv) is a component to handle the
+deployment of interconnected multi-VM application services on top of the
+OpenNebula IaaS platform. It supports the automated creation and run-time
+scaling of multi-VM application environments. It leverages the OpenNebula
+contextualization framework to setup clusters of VMs in a master-slave
+configuration or a set of workers with an IP load-balancer in front. Policies
+can be defined to control how VMs are added or removed based on manual, time
+of day, or application load-based triggers.
+
+The system will ensure that an appropriate number of VMs are started even
+in the presence of hypervisor or data center failures to meet application
+service requirements. Environments can be aggregated so that multi-tier
+services consisting of web, app, caching, database clusters can be
+interconnected and deployed as unit. A variety of sample integrations with
+distributed middleware and tools like Tomcat, CloudFoundry, Jenkins, HBase
+are provided.
+
+For service developers, an IaaS platform and layered tools like this can help
+turn their applications from static, manually configured systems that take
+weeks or months to deploy and change into dynamic and adaptive systems.
+Infrastructure becomes just another set of APIs that you use in developing
+your service so that the infrastructure will scale as your service
+workload evolves.
+
+Key Features
+------------
+
+* Automated deployement of clustered services (load-balanced worker pool or master-slave deployment pattern)
+* Automated deployment of multiple inter-dependent clusters
+* Integration with Balance software load-balancer with open hooks to support other load-balancers
+* Auto-recreate VMs that disappear, fail or go into unknown state due to hypervisior or transient network/storage issues
+* Auto-scaling of environments based on time of day or day of week
+* Auto-scaling of environments based on load (cpu by default with support for custom metric plugins)
+* Prioritization of services and resource pools into 'gold','silver','bronze' classes.
+* Preemption and reclaim of resources from lower classes when higher priority classes required them
+* Active-Active and Active-Passive DR support where environments in surviving data centers are scaled up in response to datacenter outages
+* Centralized management and failover of the oneenv controller
+
+Usage
+-----
+Usage: oneenv [options]
+ -c, --create STRING Create environment from configuration
+ -r, --remove STRING Remove environment and delete VMs
+ -u, --scaleup STRING Scale up environment by adding VM
+ -d, --scaledown STRING Scale down environment by removing VM
+ -U, --upload STRING Upload config.rb, vm template or plugin to server
+ -D, --download STRING Download log file from server
+ -J, --jobs List all jobs
+ -v, --vms List all VMs
+ -k, --envvms STRING List all VMs for environment
+ -S, --services List all services managed by Global Scheduler
+ -R, --requests List all requests in Global Scheduler
+ -P, --pools List all pools in Global Scheduler
+ -j, --json Display verbose JSON output
+ -o, --configs List all environment configs
+ -e, --envs List all environments
+
+
-Carina Extensions to OpenNebula
View
79 bin/onecmd
@@ -0,0 +1,79 @@
+#!/usr/bin/env ruby
+#=======================================================================================
+#
+# $File: //depot/main/cloud/iaas/compute/software/opennebula/3.0.0/package/root/src/ozones/Client/bin/onecmd $
+# $Author: andscott $
+# $Change: 5156429 $
+# $DateTime: 2012/04/10 10:35:01 $
+# $Revision: #5 $
+#
+#=======================================================================================
+
+require 'optparse'
+require 'tempfile'
+require ENV["HOME"] + "/config.rb"
+
+endpoint = ""
+all = false
+list = false
+xmlout = false
+OptionParser.new do |opts|
+ opts.on("-e", "--endpoint STRING", String, "ENDPOINT") do |opt|
+ endpoint = opt
+ end
+ opts.on("-a", "--all", "All ENDPOINTs") do
+ all = true
+ end
+ opts.on("-l", "--list", "List configured ENDPOINTs") do
+ list = true
+ end
+ opts.on("-x", "--xml", "XML output") do
+ xmlout = true
+ end
+ begin
+ opts.parse!(ARGV)
+ end
+end
+
+if list
+ printf "ENDPOINTs:"
+ ENDPOINT.each_key do |e|
+ printf " " + e
+ end
+ printf "\n"
+ exit
+end
+
+if all and not endpoint.empty?
+ STDERR.puts "ERROR: -e and -a are mutually exclusive"
+ exit(-1)
+end
+
+if not all and endpoint.empty?
+ STDERR.puts "ERROR: endpoint not specificed"
+ exit(-1)
+end
+
+if not all and not ENDPOINT.has_key?(endpoint)
+ STDERR.puts "ERROR: endpoint does not exist"
+ exit(-1)
+end
+
+def onecmd (e, args)
+ file = Tempfile.new('.one_auth', ENV["HOME"])
+ file.puts(ENDPOINT[e][:oneauth])
+ file.close
+ system("export ONE_AUTH=#{file.path} ; export ONE_XMLRPC=#{ENDPOINT[e][:proxy]} ; #{ARGV.join(" ")}" + args)
+ file.unlink
+end
+
+args = xmlout ? " -x" : ""
+if all
+ ENDPOINT.each_key do |e|
+ puts "|-- start #{e} ---------------------------------------------------------------------|"
+ onecmd(e, args)
+ puts "|-- end #{e} -----------------------------------------------------------------------|"
+ end
+else
+ onecmd(endpoint, args)
+end
View
485 bin/oneenv
@@ -0,0 +1,485 @@
+#!/usr/bin/env ruby
+# -------------------------------------------------------------------------- #
+# Copyright 2011-2012, Research In Motion Limited #
+# #
+# Licensed under the Apache License, Version 2.0 (the "License"); you may #
+# not use this file except in compliance with the License. You may obtain #
+# a copy of the License at #
+# #
+# http://www.apache.org/licenses/LICENSE-2.0 #
+# #
+# Unless required by applicable law or agreed to in writing, software #
+# distributed under the License is distributed on an "AS IS" BASIS, #
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #
+# See the License for the specific language governing permissions and #
+# limitations under the License. #
+#--------------------------------------------------------------------------- #
+
+require 'rubygems'
+require 'rest_client'
+require 'json'
+require 'optparse'
+require ENV["HOME"] + "/config.rb"
+
+# Point this to the per-service oneenvd in a zone
+if ENV["CARINA_IP"] != nil
+ if ENV["CARINA_PORT"] != nil
+ baseurl = 'http://' + ENV["CARINA_IP"] + ':' + ENV["CARINA_PORT"]
+ end
+ if ENV["CARINA_GS_PORT"] != nil
+ baseurlgs = 'http://' + ENV["CARINA_IP"] + ':' + ENV["CARINA_GS_PORT"]
+ end
+else
+ # For testing
+ baseurl = 'http://10.135.41.79:4567'
+ baseurlgs = 'http://10.135.41.79:4321'
+end
+
+
+endpoint = ""
+config = ""
+envid = ""
+listenvs = false
+listjobs = false
+listvms = false
+listconfigs=false
+deploy = false
+scaleup = false
+scaledown = false
+delete = false
+upload=false
+configfile=nil
+download=false
+logfile=nil
+jsonout=false
+
+listsvcs = false
+listpools = false
+listreqs = false
+
+OptionParser.new do |opts|
+ opts.on("-c", "--create STRING", "Create environment from configuration") do |opt|
+ deploy=true
+ config = opt
+ end
+ opts.on("-r", "--remove STRING", "Remove environment and delete VMs") do |opt|
+ delete = true
+ envid = opt
+ end
+ opts.on("-u", "--scaleup STRING", "Scale up environment by adding VM") do |opt|
+ scaleup = true
+ envid = opt
+ end
+ opts.on("-d", "--scaledown STRING", "Scale down environment by removing VM") do |opt|
+ scaledown = true
+ envid = opt
+ end
+ opts.on("-U", "--upload STRING", "Upload config.rb, vm template or plugin to server") do |opt|
+ upload = true
+ configfile = opt
+ end
+ opts.on("-D", "--download STRING", "Download log file from server") do |opt|
+ download = true
+ logfile = opt
+ end
+ opts.on_tail("-o", "--configs", "List all environment configs") do
+ listconfigs = true
+ end
+ opts.on_tail("-e", "--envs", "List all environments") do
+ listenvs = true
+ end
+ opts.on("-J", "--jobs", "List all jobs") do
+ listjobs = true
+ end
+ opts.on("-v", "--vms", "List all VMs") do
+ listvms = true
+ end
+ opts.on("-k", "--envvms STRING", "List all VMs for environment") do |opt|
+ envid = opt
+ listvms = true
+ end
+ opts.on("-S", "--services", "List all services managed by Global Scheduler") do
+ listsvcs = true
+ end
+ opts.on("-R", "--requests", "List all requests in Global Scheduler") do
+ listreqs = true
+ end
+ opts.on("-P", "--pools", "List all pools in Global Scheduler") do
+ listpools = true
+ end
+ opts.on("-j", "--json", "Display verbose JSON output") do
+ jsonout = true
+ end
+ begin
+ opts.parse!(ARGV)
+ end
+end
+
+def getauthinfo
+ if ENDPOINT == nil or ENDPOINT.size == 0
+ puts "No endpoint info in config.rb"
+ exit -1
+ else
+ # Just get one of authinfo from the client in order to match against
+ # the server. This is basic security for the oneenvd server to ensure
+ # only the authorized client oneenv CLI can talk to it
+ ENDPOINT.each_key do |k|
+ auth = ENDPOINT[k][:oneauth]
+ ep = k
+ return ep,auth
+ end
+ end
+end
+
+
+def print_table(table)
+ # Calculate widths
+ widths = []
+ table.each { |line|
+ c = 0
+ line.each { |col|
+ widths[c] = (widths[c] && widths[c] > col.length) ? widths[c] : col.length
+ c += 1
+ }
+ }
+ # Indent the last column left.
+ last = widths.pop()
+ format = widths.collect{|n| "%#{n}s"}.join(" ")
+ format += " %-#{last}s\n"
+ # Print each line.
+ table.each{|line|
+ printf(format, *line)
+ }
+end
+
+def print_jobs_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=5
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="ID"
+ table[1][0]="--"
+ table[0][1]="ENVID"
+ table[1][1]="-----"
+ table[0][2]="CONFIG_NAME"
+ table[1][2]="-----------"
+ table[0][3]="TYPE"
+ table[1][3]="----"
+ table[0][4]="SUBMIT_TIME"
+ table[1][4]="-----------"
+ table[0][5]="STATUS"
+ table[1][5]="------"
+ r=2
+ result.each do |e|
+ table[r][0] = e["jobid"]
+ table[r][1] = e["envid"].to_s
+ table[r][2] = e["name"]
+ table[r][3] = e["type"]
+ table[r][4] = e["submit"]
+ table[r][5] = e["status"]
+ r = r + 1
+ end
+ print_table(table)
+end
+
+def print_env_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=5
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="ID"
+ table[1][0]="--"
+ table[0][1]="NAME"
+ table[1][1]="----"
+ table[0][2]="VMS"
+ table[1][2]="---"
+ table[0][3]="P_STATUS"
+ table[1][3]="--------"
+ table[0][4]="D_STATUS"
+ table[1][4]="--------"
+ table[0][5]="APP_URL"
+ table[1][5]="-------"
+ r=2
+ result.each do |e|
+ table[r][0] = e["envid"]
+ table[r][1] = e["name"]
+ table[r][2] = e["numVMs"].to_s
+ table[r][3] = e["policy_status"].downcase
+ table[r][4] = e["status"].downcase[0..20]
+ table[r][5] = e["url"] == nil ? "-" : e["url"]
+ r = r + 1
+ end
+ print_table(table)
+end
+
+def print_vm_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=5
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="VMID"
+ table[1][0]="----"
+ table[0][1]="ENVID"
+ table[1][1]="----"
+ table[0][2]="HOSTNAME"
+ table[1][2]="--------"
+ table[0][3]="IPADDR"
+ table[1][3]="------"
+ table[0][4]="STATUS"
+ table[1][4]="------"
+ table[0][5]="CPU"
+ table[1][5]="---"
+ r=2
+ result.each do |e|
+ table[r][0] = e["vmid"]
+ table[r][1] = e["envid"]
+ table[r][2] = e["hostname"] == nil ? "-" : e["hostname"]
+ table[r][3] = e["ipaddress"] == nil ? "-" : e["ipaddress"]
+ table[r][4] = e["status"].downcase
+ table[r][5] = e["cpu"].to_s
+ r = r + 1
+ end
+ print_table(table)
+end
+
+def print_request_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=6
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="ID"
+ table[1][0]="--"
+ table[0][1]="ENVID"
+ table[1][1]="----"
+ table[0][2]="SERVICE"
+ table[1][2]="-------"
+ table[0][3]="MEM"
+ table[1][3]="----"
+ table[0][4]="VCPUS"
+ table[1][4]="------"
+ table[0][5]="STATUS"
+ table[1][5]="------"
+ table[0][6]="POOL"
+ table[1][6]="----"
+ r=2
+ result.each do |e|
+ table[r][0] = e["@reqid"]
+ table[r][1] = e["@envid"]
+ table[r][2] = e["@service"]
+ table[r][3] = e["@mem"].to_s
+ table[r][4] = e["@vcpus"].to_s
+ table[r][5] = e["@status"]
+ table[r][6] = e["@pool"]
+ r = r + 1
+ end
+ print_table(table)
+end
+
+
+def print_pool_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=6
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="NAME"
+ table[1][0]="----"
+ table[0][1]="ZONE"
+ table[1][1]="----"
+ table[0][2]="PRIORITY_CLASS"
+ table[1][2]="--------------"
+ table[0][3]="MEM_TOT"
+ table[1][3]="-------"
+ table[0][4]="MEM_AVAIL"
+ table[1][4]="---------"
+ table[0][5]="VCPUS_TOT"
+ table[1][5]="---------"
+ table[0][6]="VCPUS_AVAIL"
+ table[1][6]="-----------"
+ r=2
+ result.each do |e|
+ table[r][0] = e["@name"]
+ table[r][1] = e["@zone"]
+ table[r][2] = e["@priclass"]
+ table[r][3] = e["@mem"].to_s
+ table[r][4] = e["@availmem"].to_s
+ table[r][5] = e["@vcpus"].to_s
+ table[r][6] = e["@availvcpus"].to_s
+ r = r + 1
+ end
+ print_table(table)
+end
+
+
+def print_service_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=4
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="NAME"
+ table[1][0]="----"
+ table[0][1]="PRIORITY_CLASS"
+ table[1][1]="--------------"
+ table[0][2]="MEM(ALLOC/MAX)"
+ table[1][2]="--------------"
+ table[0][3]="VCPUS(ALLOC/MAX)"
+ table[1][3]="---------------"
+ table[0][4]="CLIENT"
+ table[1][4]="------"
+ r=2
+ result.each do |e|
+ table[r][0] = e["@name"]
+ table[r][1] = e["@priclass"]
+ table[r][2] = e["@memallocated"].to_s + "/" + e["@maxmemory"].to_s
+ table[r][3] = e["@vcpusallocated"].to_s + "/" + e["@maxcpus"].to_s
+ c = e["@clientList"]
+ if c != nil && c.size != 0
+ table[r][4] = c[0]["@host"] + ":" + c[0]["@port"]
+ else
+ table[r][4] = "NONE"
+ end
+ r = r + 1
+ end
+ print_table(table)
+end
+
+def print_config_table(data)
+ result = JSON.parse(data)
+ rows=result.length + 2
+ cols=3
+ table = Array.new(rows){Array.new{cols}}
+ table[0][0]="NAME"
+ table[1][0]="----"
+ table[0][1]="TYPE"
+ table[1][1]="----"
+ table[0][2]="ENDPOINT"
+ table[1][2]="-------"
+ table[0][3]="DESCRIPTION"
+ table[1][3]="-----------"
+ r=2
+ result.each do |e|
+ table[r][0] = e["name"]
+ table[r][1] = e["type"]
+ table[r][2] = e["endpoint"] == nil ? "-" : e["endpoint"]
+ table[r][3] = e["desc"][0..40]
+ r = r + 1
+ end
+ print_table(table)
+end
+
+
+if listenvs == true
+ response = RestClient.get baseurl + '/envs'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_env_table(response.body)
+ end
+end
+
+if listjobs == true
+ response = RestClient.get baseurl + '/jobs'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_jobs_table(response.body)
+ end
+end
+
+if listconfigs == true
+ response = RestClient.get baseurl + '/configs'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_config_table(response.body)
+ end
+end
+
+if listvms == true
+ if envid != ""
+ url = baseurl + '/envs/' + envid + '/vms'
+ response = RestClient.get url
+ else
+ response = RestClient.get baseurl + '/vms'
+ end
+ body = response.body
+ if body.include? "ERROR:"
+ puts body
+ else
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_vm_table(response.body)
+ end
+ end
+end
+
+
+if listreqs == true
+ response = RestClient.get baseurlgs + '/requests'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_request_table(response.body)
+ end
+end
+
+if listsvcs == true
+ response = RestClient.get baseurlgs + '/services'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_service_table(response.body)
+ end
+end
+
+if listpools == true
+ response = RestClient.get baseurlgs + '/pools'
+ if jsonout == true
+ puts JSON.pretty_generate(JSON.parse(response.body))
+ else
+ print_pool_table(response.body)
+ end
+end
+
+
+if deploy == true
+ ep,authinfo = getauthinfo()
+ url = baseurl + '/configs/' + config
+ response = RestClient.post url , :content_type => 'text/plain', :endpoint => ep, :authinfo => authinfo
+ puts response.body
+end
+if scaleup == true
+ ep,authinfo = getauthinfo()
+ url = baseurl + '/envs/' + envid + "/scaleup"
+ response = RestClient.put url , :content_type => 'text/plain', :endpoint => ep, :authinfo => authinfo
+ puts response.body
+end
+if scaledown == true
+ url = baseurl+ '/envs/' + envid + "/scaledown"
+ ep,authinfo = getauthinfo()
+ response = RestClient.put url , :content_type => 'text/plain', :endpoint => ep, :authinfo => authinfo
+ puts response.body
+end
+if delete == true
+ ep,authinfo = getauthinfo()
+ url = baseurl + '/envs/' + envid
+ response = RestClient.delete url , :params => {:content_type => 'text/plain', :endpoint => ep, :authinfo => authinfo }
+ puts response.body
+end
+
+if upload == true
+ if configfile == "config.rb"
+ response = RestClient.post baseurl + '/configs' , :type => 'config', :name => 'config.rb', :upload => File.new(ENV["HOME"] + "/" + configfile), :content_type => 'text/plain'
+ elsif configfile.include? ".rb"
+ response = RestClient.post baseurl + '/configs' , :type => 'plugin', :name => configfile , :upload => File.new(ENV["HOME"] + "/plugins/" + configfile), :content_type => 'text/plain'
+ else
+ response = RestClient.post baseurl + '/configs' , :type => 'vm', :name => configfile , :upload => File.new(ENV["HOME"] + "/vm/" + configfile), :content_type => 'text/plain'
+ end
+ puts response.body
+end
+
+if download == true
+ response = RestClient.get baseurl + '/logs/' + logfile
+ puts response.body
+end
+
View
400 bin/oneenvd
@@ -0,0 +1,400 @@
+#!/usr/bin/env ruby
+# -------------------------------------------------------------------------- #
+# Copyright 2011-2012, Research In Motion Limited #
+# #
+# Licensed under the Apache License, Version 2.0 (the "License"); you may #
+# not use this file except in compliance with the License. You may obtain #
+# a copy of the License at #
+# #
+# http://www.apache.org/licenses/LICENSE-2.0 #
+# #
+# Unless required by applicable law or agreed to in writing, software #
+# distributed under the License is distributed on an "AS IS" BASIS, #
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #
+# See the License for the specific language governing permissions and #
+# limitations under the License. #
+#--------------------------------------------------------------------------- #
+
+require 'rubygems'
+require 'sinatra'
+require 'json'
+require 'mysql'
+require 'logger'
+require 'redis'
+require 'socket'
+require 'time'
+require '../lib/time_window.rb'
+require '../lib/policy_manager.rb'
+require '../lib/job_manager.rb'
+require '../lib/gs_message.rb'
+require '../lib/gs_proxy.rb'
+require '../lib/env_manager.rb'
+require '../lib/plugin_manager.rb'
+
+load ENV["HOME"] + "/config.rb"
+
+class Auth
+ def self.checkAuthInfo(params)
+ ep = params[:endpoint]
+ authinfo = params[:authinfo]
+ $Logger.debug "Checking ep=#{ep} authinfo=#{authinfo}"
+ if ENDPOINT[ep] == nil
+ $Logger.debug "Endpoint #{ep} not found"
+ return false
+ end
+ if ENDPOINT[ep][:oneauth] == nil
+ $Logger.debug "Endpoint #{ep} has no oneauth info"
+ return false
+ end
+ if ENDPOINT[ep][:oneauth] != authinfo
+ $Logger.debug "Endpoint not found or no oneauth field for endpoint"
+ return false
+ end
+ return true
+ end
+end
+
+class Params
+ def self.readFile(file)
+ File.readlines(file).each do |line|
+ next if /\S/ !~ line || line.match(/^#/)
+ values = line.split("=")
+ lines2 = values[1].split.join("\n")
+ ENV[values[0]] = lines2
+ end
+ end
+end
+
+chldhandler = trap("CHLD") {
+ #STDERR.puts("Trapping signal CHLD")
+ begin
+ # tmppid, status = Process.wait2(cmdpid)
+ tmppid = Process.wait
+ status = $?
+ cmd_exitstatus = status.exitstatus
+ $jobManager.reportDone(tmppid, cmd_exitstatus)
+ #STDERR.puts("Process::Status => " + status.inspect)
+ #STDERR.puts("exitstatus => " + cmd_exitstatus.inspect)
+ if ! status.exited?
+ #raise "ARGH! The child has not exited yet!"
+ puts "ARGH! The child has not exited yet!"
+ end
+ rescue Errno::ECHILD => e
+ #STDERR.puts("Command failed - probably ok? #{e.inspect}")
+ # bah, ignore it, just means a command failed
+ end
+}
+
+
+if File.exists?("../etc/system.conf")
+ Params.readFile("../etc/system.conf")
+end
+
+if File.exists?("../etc/oneenv.conf")
+ conffile = "../etc/oneenv.conf"
+else
+ conffile = ENV["HOME"] + "/conf/oneenv.conf"
+end
+Params.readFile(conffile)
+
+
+$port = ENV["CARINA_PORT"]
+set :bind, '0.0.0.0'
+set :port, $port
+$useGlobalScheduler = true
+if ENV["USE_GLOBAL_SCHEDULER"] != nil
+ $useGlobalScheduler = ENV["USE_GLOBAL_SCHEDULER"]
+ $useGlobalScheduler == "true" ? $useGlobalScheduler = true: $useGlobalScheduler = false
+end
+
+$service=ENV["SERVICE_NAME"]
+$zone=ENV["ZONE"]
+# Determine if we are going to call ONE to get the load of all the VMs.
+# Only one oneenvd per host should be configured for this
+$vmInfoLoader = false
+if ENV["LOAD_VM_INFO"] != nil
+ $vmInfoLoader = ENV["LOAD_VM_INFO"]
+ $vmInfoLoader == "true" ? $vmInfoLoader = true: $vmInfoLoader = false
+end
+
+$TMPVMLOADFILE = "/tmp/onevmload"
+$LOGDIR = ENV["HOME"] + "/logs/"
+$WORKDIR = ENV["HOME"] + "/work/"
+$PLUGINDIR = ENV["HOME"] + "/plugins/"
+$ENVLOGFILE = $LOGDIR + "env.log"
+$ONEENVDLOGFILE = $LOGDIR + "/oneenvd.log"
+if File.exists?("./svccontroller.sh")
+puts "svccontroller found in cwd"
+ $SVCBIN = Dir.pwd + "/svccontroller.sh"
+puts "SVCBIN=#{$SVCBIN}"
+else
+ $SVCBIN = ENV["ONE_LOCATION"] + "/opennebula-carina/bin/svccontroller.sh"
+end
+
+$VMTABLE="vm_pool_ext"
+$APPENVTABLE="app_environment"
+$JOBSTABLE="jobs"
+$SVCTABLE="service"
+
+
+Thread.abort_on_exception = true
+$Logger = Logger.new($ONEENVDLOGFILE)
+if ENV["LOG_LEVEL"] != nil
+ $Logger.level = Logger::DEBUG
+else
+ $Logger.level = ENV["LOG_LEVEL"]
+end
+con = Mysql.new(ENV["DB_HOST"],'root',ENV["DB_PASS"],'opennebula')
+$jobManager=JobManager.new(con)
+$envManager=EnvManager.new(con)
+$configList=EnvConfigs.new(con)
+if ENV["SERVICE_NAME"] == nil
+ $Logger.fatal "Service name not defined"
+ exit
+end
+
+$SVCID = $envManager.getServiceId(ENV["SERVICE_NAME"])
+
+if $useGlobalScheduler == true
+ $gsProxy = GSchedProxy.new
+end
+$configList.load()
+$jobManager.load()
+lock = Mutex.new
+
+$pluginManager = PluginManager.new
+$pluginManager.load()
+policyManager=PolicyManager.new($envManager, $pluginManager,lock)
+$Logger.debug "Creating thread for PolicyManager"
+Thread.new(){
+ policyManager.run
+}
+
+Thread.new() {
+ $jobManager.run(lock)
+}
+if $useGlobalScheduler == true
+ Thread.new() {
+ $gsProxy.run(lock)
+ }
+ # Have to register after the proxy run() is executed so that we
+ # start subscribing. Otherwise we won't receive the INITIALIZE message
+ # when we REGISTER
+ sleep(2)
+ $gsProxy.register()
+end
+
+get '/configs' do
+ list = $configList.getList()
+ JSON.generate(list)
+end
+
+get '/logs/:logname' do
+ fileout = ""
+ if params[:logname] == "oneenvd"
+ logfile=$ONEENVDLOGFILE
+ else
+ logfile=$LOGDIR + "/#{params[:logname]}"
+ end
+ if File.exists?(logfile)
+ fileout = `tail -n 500 #{logfile}`
+ else
+ fileout = "File not found"
+ end
+ fileout
+end
+
+post '/configs' do
+ type = params[:type]
+ name = params[:name]
+ if type == "config"
+ # Handle config.rb update
+ # Save to received config.rb to temp file first
+ fname = ENV["HOME"] + "/config.rb"
+ File.open(fname + ".tmp.rb", "w") do |f|
+ f.write(params[:upload][:tempfile].read)
+ end
+ if $configList.validate(fname + ".tmp.rb") == false
+ msg = "Invalid config.rb for this service"
+ # Reset back to the old configuration
+ $configList.load()
+ else
+ # Backup the old config.rb
+ if File.exists?(fname) == true
+ File.rename(fname, fname + ".bak")
+ end
+ File.rename(fname + ".tmp.rb", fname)
+ $configList.load()
+ $configList.insertindb()
+ msg = "OK"
+ end
+ elsif type == "plugin"
+ fname = ENV["HOME"] + "/plugins/" + name
+ if File.exists?(fname) == true
+ File.rename(fname, fname + ".bak")
+ end
+ File.open(fname, "w") do |f|
+ f.write(params[:upload][:tempfile].read)
+ end
+ $pluginManager.loadPlugin(fname)
+ else
+ # VM template file
+ fname = ENV["HOME"] + "/vm/" + name
+ if File.exists?(fname) == true
+ File.rename(fname, fname + ".bak")
+ end
+ File.open(fname, "w") do |f|
+ f.write(params[:upload][:tempfile].read)
+ end
+ msg = "OK"
+ end
+ msg
+end
+
+
+post '/configs/:configname' do
+ envid = nil
+ jobid = nil
+ errmsg = nil
+
+ if Auth.checkAuthInfo(params) == false
+ errmsg = "Not authorized to use this service"
+ end
+
+ if errmsg == nil
+ lock.synchronize {
+ config = $configList.getbyname(params[:configname])
+ if config == nil
+ errmsg = "Invalid environment configuration"
+ elsif config.type == "group"
+ # For environment group we launch first environment and
+ # queue up jobs for launching dependent environments
+ members = ENVIRONMENT[config.name][:members]
+ if members == nil || members[0] == nil
+ errmsg = "Group configuration has no members"
+ else
+ firstconfig = $configList.getbyname(members[0])
+ if firstconfig == nil
+ errmsg = "Cannot find configuration for " + members[0]
+ else
+ envid, jobid = firstconfig.launch(nil, $configList.getDBConnection())
+ job = $jobManager.getJobList.select{|j| j.jobid == jobid}
+ job[0].name = firstconfig.name
+ job[0].groupname = config.name
+ job[0].groupid = jobid
+ $jobManager.updateInDb(job[0])
+ membersleft = members[1..-1]
+ $Logger.debug "Queueing remaining jobs after starting first environment of group" + membersleft.to_s
+ $jobManager.queuejobs(config.name, jobid, membersleft)
+ end
+ end
+ else
+ envid, jobid = config.launch(nil, $configList.getDBConnection())
+ end
+ }
+ end
+ if errmsg != nil
+ errmsg
+ else
+ "{\"envid\":\"#{envid}\", \"jobid\":\"#{jobid}\"}"
+ end
+end
+
+get '/envs' do
+ lock.synchronize {
+ $envManager.loadAll()
+ envList = $envManager.getEnvs()
+ envList.to_json
+ }
+end
+
+get '/envs/:envid/vms' do
+ lock.synchronize {
+ env = $envManager.loadEnvVMs(params[:envid])
+ if env != nil
+ vmList = env.getVMList()
+ vmList.to_json
+ else
+ "ERROR: Environment does not exist"
+ end
+ }
+end
+
+put '/envs/:envid/scaleup' do
+ if Auth.checkAuthInfo(params) == false
+ return "Not authorized to use this service"
+ end
+ envid=params[:envid]
+ e = $envManager.getEnv(envid)
+ if e == nil
+ return "Environment not found"
+ end
+ if $jobManager.isJobRunning(envid)
+ return "Operation in progress"
+ end
+ jid = ""
+ lock.synchronize {
+ if $useGlobalScheduler == true
+ jid = e.scaleup(false, 0)
+ else
+ jid = e.scaleup(true, 0)
+ end
+ }
+ "{\"jobid\":\"#{jid}\"}"
+end
+
+
+put '/envs/:envid/scaledown' do
+ if Auth.checkAuthInfo(params) == false
+ return "Not authorized to use this service"
+ end
+ envid=params[:envid]
+ e = $envManager.getEnv(envid)
+ if e == nil
+ return "Environment not found"
+ end
+ if $jobManager.isJobRunning(envid)
+ return "Operation in progress"
+ end
+ jid = ""
+ lock.synchronize {
+ jid = e.scaledown(true,nil)
+ }
+ "{\"jobid\":\"#{jid}\"}"
+end
+
+
+delete '/envs/:envid' do
+ if Auth.checkAuthInfo(params) == false
+ return "Not authorized to use this service"
+ end
+ envid=params[:envid]
+ jid=""
+ lock.synchronize {
+ status,jid = $envManager.removeEnv(envid)
+ }
+ "{\"jobid\":\"#{jid}\"}"
+end
+
+
+get '/jobs' do
+ jobList=$jobManager.getJobList()
+ JSON.generate(jobList)
+end
+
+delete '/jobs/:jobid' do
+ jobid=params[:jobid]
+ lock.synchronize {
+ $jobManager.remove(jobid)
+ }
+end
+
+
+get '/vms' do
+ lock.synchronize {
+ $envManager.loadAll()
+ $envManager.loadVMInfo()
+ vmList = $envManager.getVMs()
+ JSON.generate(vmList)
+ }
+end
View
617 bin/oneenvd-gs
@@ -0,0 +1,617 @@
+#!/usr/bin/env ruby
+# -------------------------------------------------------------------------- #
+# Copyright 2011-2012, Research In Motion Limited #
+# #
+# Licensed under the Apache License, Version 2.0 (the "License"); you may #
+# not use this file except in compliance with the License. You may obtain #
+# a copy of the License at #
+# #
+# http://www.apache.org/licenses/LICENSE-2.0 #
+# #
+# Unless required by applicable law or agreed to in writing, software #
+# distributed under the License is distributed on an "AS IS" BASIS, #
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #
+# See the License for the specific language governing permissions and #
+# limitations under the License. #
+#--------------------------------------------------------------------------- #
+
+require 'rubygems'
+require 'sinatra'
+require 'redis'
+require 'logger'
+require 'json'
+require "../etc/global.rb"
+require "../lib/gs_message.rb"
+
+
+# Represents a client to the global scheduler i.e a oneenvd instance representing
+# the environments for a particular service. Each oneenvd is assigned a different
+# port to run on on the controller node
+class Client < JSONable
+ attr_accessor :name
+ attr_accessor :zone
+ attr_accessor :host
+ attr_accessor :port
+
+
+ def initialize(name, zone, host, port)
+ @name = name
+ @zone = zone
+ @host = host
+ @port = port
+ end
+end
+
+
+# A Pool represnts a VDC in ONE whose capacity is allocted by the global scheduler
+# Pools can be assigned a priority class (gold,silver,bronze) which is used to match
+# requests from services
+class Pool < JSONable
+ attr_accessor :name
+ attr_accessor :zone
+ attr_accessor :vcpus
+ attr_accessor :mem
+ attr_accessor :availvcpus
+ attr_accessor :availmem
+ attr_accessor :priclass
+ attr_accessor :status
+
+ def initialize(name, zone, vcpus, mem, priclass)
+ @name = name
+ @zone = zone
+ @vcpus = vcpus
+ @mem = mem
+ @priclass = priclass
+ @availvcpus = vcpus
+ @availmem = mem
+ end
+end
+
+# This represents a business service that will draw resources from the scheduler
+# Each service is represented by one or more clients (oneenvd's) that will make
+# resource requests for environments that they manage. By associating a prioirty class
+# with the service - and hence the requests - the global scheduler tries to match
+# the request with the right type of resource
+#
+#
+class Service < JSONable
+ attr_accessor :name
+ attr_accessor :id
+ attr_accessor :clientList
+ attr_accessor :zone
+ attr_accessor :vcpusallocated
+ attr_accessor :memallocated
+ attr_accessor :priclass
+ attr_accessor :maxcpus
+ attr_accessor :maxmemory
+
+
+ def initialize (name,authinfo, maxcpus, maxmemory, priclass)
+ @name = name
+ @zone = zone
+ @maxcpus = maxcpus
+ @maxmemory = maxmemory
+ @priclass = priclass
+ @clientList = Array.new
+ @vcpusallocated = 0
+ @memallocated = 0
+ end
+end
+
+# This class represents the global scheduler which periodically wakes up and looks
+# at the list of outstanding requests made by clients and matches them to the available
+# resource pools. It communicates with the client via Redis pub/sub mechanism. Attempts
+# to match requests with available pools can result in reclaim or preempt resources
+# from other clients so that the highest priority services can get access to matching
+# resources.
+class Scheduler
+ def initialize()
+ @poolList = Array.new
+ @svcList = Array.new
+ @reqList = Array.new
+ @preempteeList = Array.new
+ @preemptableList = Array.new
+ @reclaimOutstanding = Hash.new
+ end
+
+ # Load the configuration and resource requests from Redis
+ def loadAll()
+ # Load the configuration from global.rb
+ SHAREDPOOLS.each_key do |k|
+ p = Pool.new(k, SHAREDPOOLS[k][:zone],
+ SHAREDPOOLS[k][:allocateable_cpus],
+ SHAREDPOOLS[k][:allocateable_mem],
+ SHAREDPOOLS[k][:service_class])
+
+ @poolList.push(p)
+ end
+ SERVICES.each_key do |k|
+ s = Service.new(k, SERVICES[k][:authinfo],
+ SERVICES[k][:maxcpus],
+ SERVICES[k][:maxmemory],
+ SERVICES[k][:priority])
+ @svcList.push(s)
+ end
+ # Load the requests from redis
+ @reqList = $eventProcessor.loadRequests()
+ # Compute the free/used stats from state of requests
+ @reqList.each { |req|
+ if req.status == "ALLOCATED"
+ pmatches = @poolList.select {|p| p.name == req.pool}
+ if pmatches == nil || pmatches[0] == nil
+ $Logger.error "Unable to find pool #{req.pool} for ALLOCATED request #{req.reqid}"
+ next
+ end
+ smatches = @svcList.select {|s| s.name == req.service}
+ if smatches == nil || smatches[0] == nil
+ $Logger.error "Unable to find service #{req.service} for ALLOCATED request #{req.reqid}"
+ next
+ end
+ pool = pmatches[0]
+ pool.availvcpus = pool.availvcpus.to_i - req.vcpus.to_i
+ pool.availmem = pool.availmem.to_i - req.mem.to_i
+ service = smatches[0]
+ service.vcpusallocated = service.vcpusallocated.to_i + req.vcpus.to_i
+ service.memallocated = service.memallocated.to_i + req.mem.to_i
+ end
+ }
+ end
+
+
+ def addrequest(request)
+ @reqList.push(request)
+ end
+
+ def getRequests()
+ return @reqList
+ end
+
+ def getServices()
+ return @svcList
+ end
+
+ def getPools()
+ return @poolList
+ end
+
+ def getPriClass(req)
+ match = @svcList.select{|s| s.name == req.service}
+ return match[0].priclass
+ end
+
+ def getReqPoolPriClass(req)
+ match = @poolList.select{|p| p.name == req.pool}
+ return match[0].priclass
+ end
+
+ # Add a client (oneenvd) to a services list. A client can exist for
+ # each zone for a given service
+ def registerClient(client, svcname)
+ match = @svcList.select{|s| s.name == svcname}
+ if match == nil || match[0] == nil
+ $Logger.error "Unable to find service #{svcname}"
+ return
+ end
+ svc = match[0]
+
+ # Only add it if its not the same oneenvd re-registerign
+ match = svc.clientList.select{|c| c.name == client.name && c.zone == client.zone && c.port == client.port && c.host == client.host}
+ if match == nil || match[0] == nil
+ svc.clientList.push(client)
+ end
+ end
+
+ def priclass2Index(priclass)
+ classlist = [ 'bronze', 'silver', 'gold', 'platinum' ]
+ return classlist.index(priclass)
+ end
+
+ # Attempt to find a matching pool for a request and inform the client if found
+ def reserve(req)
+ # Check the services priority
+ match = @svcList.select{|s| s.name == req.service}
+ if match == nil || match[0] == nil
+ $Logger.error "Unable to find service #{req.service}"
+ return
+ end
+ service = match[0]
+ svcpri = service.priclass
+ # CHeck if service limits will be exceeded
+ vcpusallocated = service.vcpusallocated.to_i + req.vcpus.to_i
+ memallocated = service.memallocated.to_i + req.mem.to_i
+ if vcpusallocated > service.maxcpus.to_i || memallocated > service.maxmemory.to_i
+ $Logger.debug "Request #{req.reqid} would exceed limits for #{req.service}"
+ return
+ end
+
+ if SHAREDPOOLS[req.pool] == nil
+ $Logger.error "Pool in request #{req.reqid} is #{req.pool} not known"
+ return
+ end
+ # Find all pools with a equal or better class than the request
+ # We look for pools in the same zone as the pool in which the
+ # environment was originall created in.
+ targetZone = SHAREDPOOLS[req.pool][:zone]
+ match = @poolList.select {|p|
+ priclass2Index(p.priclass) >= priclass2Index(svcpri) &&
+ p.zone == targetZone &&
+ p.availvcpus.to_i >= req.vcpus &&
+ p.availmem.to_i >= req.mem
+ }
+ if match == nil || match[0] == nil
+ $Logger.info "No matching pool found for #{req.reqid}. Adding req to preemptee list"
+ @preempteeList.push(req)
+ return
+ end
+
+ $Logger.info "Number of matching pools found for req #{req.reqid} is #{match.length()}"
+
+
+ match.sort!{ |a,b| a.availvcpus.to_i <=> b.availvcpus.to_i }
+ pool = match[0]
+ if pool == nil
+ $Logger.error "No pools found #{req.service}"
+ return
+ end
+ $Logger.info "Pool info: #{pool.availvcpus} #{pool.availmem}"
+ $Logger.info "Req info: #{req.vcpus} #{req.mem}"
+
+ $Logger.info "Assigning #{req.reqid} to pool #{pool.name}"
+ pool.availvcpus = pool.availvcpus.to_i - req.vcpus.to_i
+ pool.availmem = pool.availmem.to_i - req.mem.to_i
+ service.vcpusallocated = service.vcpusallocated.to_i + req.vcpus.to_i
+ service.memallocated = service.memallocated.to_i + req.mem.to_i
+ req.status = "ALLOCATED"
+ req.pool = pool.name
+ $eventProcessor.updateRequest(req)
+ # Send m GRANT essage to the client
+ g=Grant.new("GRANT",req.zone, req.service, pool.name, req.envid, req.reqid)
+ g.vcpus = req.vcpus
+ g.mem = req.mem
+ $eventProcessor.notifyclient(req.service + "-" + req.zone, g.to_json)
+ return
+ end
+
+ # Process a release request sent from the client in order to free up
+ # the resources
+ def release(rel)
+ # Find the request with this id
+ match = @reqList.select{|r| r.reqid == rel.reqid}
+ if match == nil or match[0] == nil
+ $Logger.error "Internal error. Cannot find req with id #{rel.reqid}"
+ return
+ end
+ req = match[0]
+ freeup(req)
+ end
+
+
+ # Helper function to restore counters when resources are freed. Resources are
+ # freed by either an explicit RELEASE command from the client or a CANCEL when
+ # the entire environment is removed
+ def freeup(req)
+ if req.status != "ALLOCATED"
+ $Logger.error "Internal error. Trying to release unallocated request for #{req.reqid}"
+ return
+ end
+
+ # Find the pool and restore the capacity
+ match = @poolList.select{|p| p.name == req.pool }
+ if match == nil || match[0] == nil
+ $Logger.error "Internal error. No matching pool found for #{req.reqid}"
+ return
+ end
+ pool = match[0]
+ pool.availvcpus = pool.availvcpus.to_i + req.vcpus.to_i
+ pool.availmem = pool.availmem.to_i + req.mem.to_i
+ # Adjust the limit counters for the service
+ match = @svcList.select{|s| s.name == req.service}
+ if match == nil || match[0] == nil
+ $Logger.error " No matching service #{req.service} found for #{req.reqid}"
+ end
+ service = match[0]
+ service.vcpusallocated = service.vcpusallocated.to_i - req.vcpus.to_i
+ service.memallocated = service.memallocated.to_i - req.mem.to_i
+ req.status = "RELEASED"
+ $eventProcessor.updateRequest(req)
+ end
+
+
+ # Handles attempts to release resources for lower priority services which
+ # are using resources from a higher priority pool than they normally should.
+ def reclaim()
+ if @preempteeList.empty?
+ $Logger.info "No one requiring preemption"
+ return
+ end
+ # Look for low priority requests using resources of a higher class
+ @preemptableList.clear
+ @reqList.each do |req|
+ if req.status == "ALLOCATED"
+ match1 = @poolList.select {|p| p.name == req.pool}
+ match2 = @svcList.select {|s| s.name == req.service}
+ poolpri = match1[0].priclass
+ svcpri = match2[0].priclass
+ if priclass2Index(poolpri) > priclass2Index(svcpri)
+ $Logger.info "Adding req #{req.reqid} as preemption candidate"
+ @preemptableList.push(req)
+ end
+ end
+ end
+ $Logger.info "The size of the preemptable list is #{@preemptableList.length}"
+ # Try and match preemptable with preemptee
+ @preempteeList.each do |target|
+ targetPriClass = getPriClass(target)
+ $Logger.info " Trying to find a match for #{target.reqid} with class #{targetPriClass}"
+ match = @preemptableList.select { |r|
+ candPoolPriClass = getReqPoolPriClass(r)
+ targetPriClass == candPoolPriClass}
+ if match == nil or match[0] == nil
+ $Logger.info "No match found for #{target.reqid}"
+ next
+ end
+ # Check if we have already sent a reclaim request for this
+ # environment recently
+ r = match[0]
+ ekey = r.service + "-" + r.zone + "-" + r.envid
+ if @reclaimOutstanding.has_key?(ekey)
+ if Time.now - @reclaimOutstanding.fetch(ekey) < 60
+ $Logger.info "Environment has recent reclaim request #{r.envid}"
+ next
+ end
+ end
+
+ # Send a reclaim message to the target client which we are
+ # preempting. The reclaim will cause a release of the resource
+ # which should be assigned to the target
+
+ $Logger.info " Sending a reclaim for #{r.reqid}"
+ rec = Reclaim.new("RECLAIM", r.reqid, r.envid)
+
+ $eventProcessor.notifyclient(r.service + "-" + r.zone, rec.to_json)
+ @reclaimOutstanding.store(ekey, Time.now)
+ end
+
+ end
+
+ # When an environment is removed by oneenvd, clean up all requests
+ def cancelEnv(cancel)
+ $Logger.debug "Cancelling requests for environment #{cancel.service} #{cancel.zone} #{cancel.envid}"
+ match = @reqList.select{|r| r.zone == cancel.zone && r.service == cancel.service && r.envid == cancel.envid}
+ if match == nil or match[0] == nil
+ # This can happen if oneenvd cancelling environment which didn't
+ # request any resources
+ $Logger.debug "No matching requests for client #{cancel.service} #{cancel.zone} #{cancel.envid}"
+ return
+ end
+ match.each do |r|
+ if r.status == "ALLOCATED"
+ freeup(r)
+ end
+ r.status = "CANCELLED"
+ $eventProcessor.updateRequest(r)
+ $eventProcessor.expireRequest(r)
+ @reqList.delete(r)
+ end
+ end
+
+ def run(lock)
+ loop do
+ $Logger.debug "Scheduler: Examining requests"
+ lock.synchronize {
+ @preempteeList.clear
+ @reqList.each do |req|
+ #puts "Scheduler: Examining request #{req.reqid}"
+ if req.status == "PEND"
+ self.reserve(req)
+ end
+ end
+ reclaim()
+ $Logger.debug "Scheduler: Finished examining requests - going to sleep"
+ }
+ sleep 30
+ end
+ end
+end
+
+
+# Class to handle requests from the clients. Each client has a channel through which
+# which the global scheduler sends messages named <service name>-<zone>.
+# The globl scheduler subscribes to a 'svc' channel that all clients can send messages on.
+# A special 'glob-sched' channel is used to send control/initializaiton messages to
+# any client.
+class EventProcessor
+
+ def initialize()
+ if $redis2.get("nextRequestId") == nil
+ puts "Initilizing nextRequestId"
+ $redis2.set("nextRequestId", "1")
+ end
+ end
+
+ def insertRequest(req)
+ puts "Incrementing request id"
+ $redis2.incr("nextRequestId")
+ rid = $redis2.get("nextRequestId")
+ puts "Request id #{rid}"
+ req.reqid = rid
+ msg = req.to_json
+ $redis2.set("req:#{rid}",msg)
+ puts "Inserted request with id #{rid}"
+ end
+
+ def updateRequest(req)
+ msg = req.to_json
+ rid = req.reqid
+ $redis2.set("req:#{rid}",msg)
+ end
+
+ def expireRequest(req)
+ $redis2.expire("req:#{req.reqid}", 600)
+ end
+
+
+ def loadRequests()
+ keylist=$redis2.keys("req*")
+ reqList = Array.new
+ keylist.each { |k|
+ val = $redis2.get(k)
+ req = Request.new("","","")
+ req.from_json(val)
+ if req.status != "RELEASED" && req.status != "CANCELLED"
+ reqList.push(req)
+ end
+ }
+ return reqList
+ end
+
+
+ def notifyclient(channelname, message)
+ $Logger.info "Sending message to client #{channelname}"
+ $redis2.publish(channelname, message)
+ end
+
+ def processmsg(channel, msg)
+ if msg.include? "REGISTER"
+ reg = Register.new("","","")
+ reg.from_json(msg)
+ $Logger.debug "Processing REGISTER request #{reg.service} #{reg.type} #{reg.zone} "
+ client = Client.new(reg.service, reg.zone, reg.host, reg.port)
+ $scheduler.registerClient( client, reg.service)
+
+ #Send back an INITIALIZE message to the oneenvd telling it
+ #all the outstanding resources it has by giving the list
+ #of reqids
+ reqList = $scheduler.getRequests()
+ if reqList == nil
+ return
+ end
+ match = reqList.select{|r| r.status == "ALLOCATED" && r.zone == reg.zone && r.service == reg.service }
+ if match == nil || match[0] == nil
+ $Logger.info " No outstanding allocatred resources for #{reg.service} in zone #{reg.zone}"
+ end
+
+ initresp = Initialize.new("INITIALIZE")
+ match.each { |r|
+ initresp.reqIds[r.reqid] = r.envid
+ }
+ msg = initresp.to_json
+ self.notifyclient(reg.service+"-"+reg.zone, msg)
+ end
+
+ if msg.include? "ALLOCATE"
+ req = Request.new("","","")
+ req.from_json(msg)
+ $Logger.debug "Processing ALLOCATE request #{req.service} #{req.type} #{req.vcpus} #{req.mem} #{req.pool}"
+ # Persist the request into redis
+ req.submittime = Time.now
+ req.status = "PEND"
+ self.insertRequest(req)
+ # Tell the scheduler about request
+ $scheduler.addrequest(req)
+ # Update the service and VDC counters in redis
+ end
+
+ if msg.include? "RELEASE"
+ release = Release.new("","")
+ release.from_json(msg)
+ $scheduler.release(release)
+ end
+
+ if msg.include? "CANCEL"
+ cancel = Cancel.new("","","","")
+ cancel.from_json(msg)
+ $scheduler.cancelEnv(cancel)
+ end
+
+ end
+
+ def run(lock)
+ # Publish STARTUP message so that clients will know to REGISTER
+ # This will handle the case where global scheduler is restarted after
+ s = Startup.new("STARTUP")
+ notifyclient("glob-sched", s.to_json)
+ $redis.subscribe("svc") do |on|
+ on.message do |channel, msg|
+ $Logger.debug "Received msg on channel #{channel}: #{msg}"
+ lock.synchronize {
+ self.processmsg(channel, msg)
+ }
+ end
+ end
+ end
+end
+
+
+class Params
+ def self.readFile(file)
+ File.readlines(file).each do |line|
+ next if /\S/ !~ line || line.match(/^#/)
+ values = line.split("=")
+ lines2 = values[1].split.join("\n")
+ ENV[values[0]] = lines2
+ end
+ end
+end
+
+if File.exists?("../etc/system.conf")
+ Params.readFile("../etc/system.conf")
+end
+
+if File.exists?("../etc/oneenv.conf")
+ conffile = "../etc/oneenv.conf"
+else
+ conffile = ENV["HOME"] + "/conf/oneenv.conf"
+end
+Params.readFile(conffile)
+
+
+$port = ENV["CARINA_GS_PORT"]
+set :bind, '0.0.0.0'
+set :port, $port
+
+
+Thread.abort_on_exception = true
+$Logger = Logger.new(ENV["HOME"]+'/logs/oneenvd-gs.log')
+$Logger.level = Logger::DEBUG
+$scheduler = Scheduler.new()
+# We create two connections to the Redis server. One for sending requests asynchronously
+# and the other for receiving subscriptions. These are handled by two separate threads
+# hence we need two connections.
+$redis = Redis.new(:host => ENV["GS_REDIS_IP"], :port => ENV["GS_REDIS_PORT"])
+$redis2 = Redis.new(:host => ENV["GS_REDIS_IP"], :port => ENV["GS_REDIS_PORT"])
+
+
+
+lock = Mutex.new
+$eventProcessor = EventProcessor.new()
+Thread.new() {
+ $eventProcessor.run(lock)
+}
+
+$scheduler.loadAll()
+Thread.new() {
+ $scheduler.run(lock)
+}
+
+get '/pools' do
+ lock.synchronize {
+ poolList = $scheduler.getPools()
+ poolList.to_json
+ }
+end
+
+get '/services' do
+ lock.synchronize {
+ svcList = $scheduler.getServices()
+ svcList.to_json
+ }
+end
+
+get '/requests' do
+ lock.synchronize {
+ reqList = $scheduler.getRequests()
+ reqList.to_json()
+ }
+end
+
View
554 bin/svccontroller.sh
@@ -0,0 +1,554 @@
+#!/bin/bash
+# -------------------------------------------------------------------------- #
+# Copyright 2002-2012, Research In Motion Limited #
+# #
+# Licensed under the Apache License, Version 2.0 (the "License"); you may #
+# not use this file except in compliance with the License. You may obtain #
+# a copy of the License at #
+# #
+# http://www.apache.org/licenses/LICENSE-2.0 #
+# #
+# Unless required by applicable law or agreed to in writing, software #
+# distributed under the License is distributed on an "AS IS" BASIS, #
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #
+# See the License for the specific language governing permissions and #
+# limitations under the License. #
+#--------------------------------------------------------------------------- #
+
+getsvcid() {
+ ID=`mysql -u$USER -h$HOST --password=$PASS -e "select svcid from $SVCTABLE where name='$1'" $DB | awk 'NR>1{print $1}'`
+ echo $ID
+}
+
+getvmip() {
+ if [[ $USE_DDNS == "FALSE" ]]; then
+ IP=`onecmd -e $ENDPOINT onevm show $1 | grep IP | tr '=' ' ' | tr ',' ' ' | awk 'NR>1 {print $2}'`
+ echo $IP
+ return
+ fi
+
+ cnt=0
+ while [ $cnt -le 5 ]; do
+ IP=`mysql -u$USER -h$HOST --password=$PASS -e "select * from $VMTABLE where vmid=$1" $DB | grep $1 | awk '{print $4}'`
+ if [[ $IP != "NULL" ]]; then
+ echo $IP
+ return
+ else
+ sleep 30
+ fi
+ cnt=`expr $cnt + 1`
+ done
+ return
+}
+
+waitvmstarted() {
+ while true; do
+ CMDOUT=`onecmd -e $ENDPOINT onevm show $1 | grep STATE | awk '{print $3}'`
+ STATE1=`echo $CMDOUT | awk '{print $1}'`
+ STATE2=`echo $CMDOUT | awk '{print $2}'`
+ if [[ $STATE1 == "FAILED" ]]; then
+ CMDOUT=`onecmd -e $ENDPOINT onevm show $1 | grep MESSAGE | cut -d'=' -f2 | sed "s/\"//g"`
+
+ updateappenvdb $2 $1 "$3_CREATE_FAIL:$CMDOUT"
+ echo "VM $1 is in FAILED state ..aborting"
+ exit -1
+ fi
+ if [[ $STATE2 == "RUNNING" ]]; then
+ echo "VM $1 is RUNNING ..return"
+ return 0
+ fi
+ echo "VM $1 state is $STATE1 $STATE2 . Retrying...."
+ sleep 30
+ done
+}
+
+waitmasterappinit() {
+ while true; do
+ STATUS=`mysql -u$USER -h$HOST --password=$PASS -e "SELECT status from $APPENVTABLE where envid=$1" $DB`
+ STATUS=`echo $STATUS | awk '{print $2}'`
+ if [[ $STATUS == "MASTER_INIT_DONE" ]]; then
+ echo "Master VM for Env $1 is initialized ..return"
+ return 0
+ fi
+ if [[ $STATUS == "MASTER_INIT_FAIL" ]]; then
+ echo "Master VM for Env $1 failed initialization ..Check the contextualization log files on the VM ..aborting"
+ exit -1
+ fi
+ echo "Env $1 not initialized . Retrying...."
+ sleep 30
+ done
+}
+
+insertvmpooldb() {
+ mysql -u$USER -h$HOST --password=$PASS -e "INSERT INTO $VMTABLE (vmid,envid,status, svcid) VALUES('$1', '$2', '$3', '$SVCID') " $DB
+
+}
+
+updateappenvdb() {
+ mysql -u$USER -h$HOST --password=$PASS -e "UPDATE $APPENVTABLE SET mastervmid=$2 , status=\"$3\" WHERE envid=$1" $DB
+
+}
+
+updateappenvpolicystatusdb() {
+ mysql -u$USER -h$HOST --password=$PASS -e "UPDATE $APPENVTABLE SET policy_status=\"$2\" WHERE envid=$1" $DB
+}
+
+updateappurl() {
+ mysql -u$USER -h$HOST --password=$PASS -e "UPDATE $APPENVTABLE SET app_url=\"$2\" WHERE envid=$1" $DB
+}
+
+getnextslaveindex() {
+ COUNT=`mysql -u$USER -h$HOST --password=$PASS -e "select * from $VMTABLE where envid=$1" $DB | wc -l`
+ COUNT=`expr $COUNT - 2`
+ echo $COUNT
+}
+
+getlastslavevm() {
+ LASTVM=`mysql -u$USER -h$HOST --password=$PASS -e "select vmid,ipaddress from $VMTABLE where envid=$1 ORDER BY vmid DESC LIMIT 1" $DB | awk 'NR>1 {print $0}' `
+ echo $LASTVM
+}
+
+getmastervmip() {
+ MASTERID=`mysql -u$USER -h$HOST --password=$PASS -e "SELECT mastervmid from $APPENVTABLE WHERE envid=$1" $DB`
+ MASTERID=`echo $MASTERID | awk '{print $2}'`
+ MASTERIP=$(getvmip $MASTERID)
+ echo $MASTERIP
+}
+
+getmastervm() {
+ MASTERID=`mysql -u$USER -h$HOST --password=$PASS -e "SELECT mastervmid from $APPENVTABLE WHERE envid=$1" $DB`
+ MASTERID=`echo $MASTERID | awk '{print $2}'`
+ MASTERIP=$(getvmip $MASTERID)
+ echo $MASTERID " " $MASTERIP
+}
+
+
+getallenvvmids() {
+ IDLIST=`mysql -u$USER -h$HOST --password=$PASS -e "select vmid from $VMTABLE where envid=$1" $DB | awk 'NR> 1 {print $1}' `
+ echo $IDLIST
+}
+
+getallenvvmips() {
+ IPLIST=`mysql -u$USER -h$HOST --password=$PASS -e "select ipaddress from $VMTABLE where envid=$1" $DB | awk 'NR> 1 {print $1}' `
+
+ echo $IPLIST
+}
+
+getvmipfromdb() {
+ VMIP=`mysql -u$USER -h$HOST --password=$PASS -e "select ipaddress from $VMTABLE where vmid=$1" $DB | awk 'NR> 1 {print $1}' `
+ echo $VMIP
+}
+
+
+deleteenvvm() {
+ mysql -u$USER -h$HOST --password=$PASS -e "DELETE from $VMTABLE where envid=$1" $DB
+
+ mysql -u$USER -h$HOST --password=$PASS -e "DELETE from $APPENVTABLE where envid=$1" $DB
+}
+
+deletevm() {
+ mysql -u$USER -h$HOST --password=$PASS -e "DELETE from $VMTABLE where vmid=$1" $DB
+}
+
+
+checknumeric() {
+ if [[ $1 == ?(+|-)+([0-9]) ]]; then
+ echo 0
+ else
+ echo 1
+ fi
+}
+
+# Adds on sections to Vm template before creating the VM
+insertvmtemplatevars() {
+ if [ -n "${PLACEMENT_POLICY+x}" ]; then
+ if [[ $PLACEMENT_POLICY == "pack" ]]; then
+ echo "RANK=RUNNING_VMS" >> $1
+ fi
+ if [[ $PLACEMENT_POLICY == "stripe" ]]; then
+ echo "RANK=\"-RUNNING_VMS\"" >> $1
+ fi
+ if [[ $PLACEMENT_POLICY == "load" ]]; then
+ echo "RANK=FREECPU" >> $1
+ fi
+ fi
+}
+
+createmaster() {
+ ENVID=$1
+ # Add the environments dependency to MASTER_CONTEXT_VAR
+ if [ -n "${ENVDEP+x}" ]; then
+ MASTER_CONTEXT_VAR=`echo $MASTER_CONTEXT_VAR $ENVDEP`
+ fi
+ MASTER_CONTEXT_VAR=`echo $MASTER_CONTEXT_VAR | sed -e "s/%NUM_SLAVES%/$NUM_SLAVES/g;"`
+
+ # Check if master template is there
+ if [ ! -f $MASTER_TEMPLATE ]; then
+ updateappenvdb $ENVID "-1" "MASTER_CREATE_FAIL: Template $MASTER_TEMPLATE not found"
+ echo "ABORT"
+ return 1
+ fi
+
+ # Create the master
+ sed -e "s/%ENVID%/$ENVID/g; s/%SERVICE_NAME%/$SERVICE_NAME/g; s/%CARINA_IP%/$CARINA_IP/g; s/%NAME%/$MASTER_VM_NAME/g; s/%NETWORK_ID%/$MASTER_NETWORK_ID/g; s/%IMAGE_ID%/$MASTER_IMAGE_ID/g; s/%CPU%/$MASTER_NUM_CPUS/g; s/%MEMORY%/$MASTER_MEMORY/g; s/%MASTER_SERVICE_PORT%/$MASTER_SERVICE_PORT/g; s/%APP_CONTEXT_SCRIPT%/$MASTER_CONTEXT_SCRIPT/g; s/%APP_CONTEXT_VAR%/$MASTER_CONTEXT_VAR/g " $MASTER_TEMPLATE > $MASTER_TEMPLATE.tmp.$$
+
+ insertvmtemplatevars $MASTER_TEMPLATE.tmp.$$
+
+ CMDOUT=`onecmd -e $ENDPOINT onevm create $MASTER_TEMPLATE.tmp.$$`
+ MASTERID=`echo $CMDOUT | awk '{print $2}'`
+ RES=$(checknumeric $MASTERID)
+ if [ $RES -eq 1 ]; then
+ updateappenvdb $ENVID "-1" "MASTER_CREATE_FAIL:$CMDOUT"
+ echo "ABORT"
+ return 1
+ fi
+ echo $MASTERID
+ return 0
+}
+
+
+
+createnv () {
+ # Add it to to the DB and get an the enviroment id
+ ENVID=$1
+ echo "Environment ID: " $ENVID
+
+ # Need some value for context vars so that the substitution in the
+ # template doesn't leave a dangling ',' which onevm create doesn't like
+ if [ -z "$MASTER_CONTEXT_VAR" ]; then
+ MASTER_CONTEXT_VAR="MASTER=true"
+ fi
+ if [ -z "$SLAVE_CONTEXT_VAR" ]; then
+ SLAVE_CONTEXT_VAR="SLAVE=true"
+ fi
+
+ MASTERID=$(createmaster $ENVID)
+ if [[ $MASTERID -eq "ABORT" ]]; then
+ echo "Fatal error when creating master. Check template, account or quotoa"
+ exit -1
+ fi
+
+ echo "Master is " $MASTERID
+
+ insertvmpooldb $MASTERID $ENVID "VM_CREATE"
+
+ updateappenvdb $ENVID $MASTERID "MASTER_CREATE"
+
+ # Wait for master to start
+ waitvmstarted $MASTERID $ENVID "MASTER"
+
+ # Get the IP address of the master
+ MASTERIP=$(getvmip $MASTERID)
+ echo "Master's IP address is " $MASTERIP
+
+ # Delay to allow application to be setup on the master
+ if [ -n "${MASTER_SETUP_TIME+x}" ]; then
+ echo "Waiting for application deployment on master"
+ sleep $MASTER_SETUP_TIME
+ fi
+
+ waitmasterappinit $ENVID
+
+ # Check if slave template is there
+ if [ ! -f $SLAVE_TEMPLATE ]; then
+ updateappenvdb $ENVID "-1" "SLAVE_CREATE_FAIL: Template $SLAVE_TEMPLATE not found"
+ exit -1
+ fi
+
+ # Create the slaves
+ SLAVELIST=""
+ for (( i=0; i < $NUM_SLAVES; i++))
+ do
+ # Add the environments dependency to SLAVE_CONTEXT_VAR
+ if [ -n "${ENVDEP+x}" ]; then
+ SLAVE_CONTEXT_VAR=`echo $SLAVE_CONTEXT_VAR $ENVDEP`
+ fi
+ # Do some variable transform
+ SLAVE_CONTEXT_VAR_T=`echo $SLAVE_CONTEXT_VAR | sed -e "s/%MASTER%/$MASTERIP/g; s/%SLAVE_INDEX%/$i/g; s/%NUM_SLAVES%/$NUM_SLAVES/g;"`
+
+ sed -e "s/%MASTER%/$MASTERIP/g; s/%SERVICE_NAME%/$SERVICE_NAME/g; s/%CARINA_IP%/$CARINA_IP/g; s/%NAME%/$SLAVE_VM_NAME/g; s/%NETWORK_ID%/$SLAVE_NETWORK_ID/g; s/%IMAGE_ID%/$SLAVE_IMAGE_ID/g; s/%CPU%/$SLAVE_NUM_CPUS/g; s/%MEMORY%/$SLAVE_MEMORY/g; s/%ENVID%/$ENVID/g; s/%APP_PACKAGE%/$APP_PACKAGE/g; s/%APP_CONTEXT_SCRIPT%/$SLAVE_CONTEXT_SCRIPT/g; s/%APP_CONTEXT_VAR%/$SLAVE_CONTEXT_VAR_T/g;" $SLAVE_TEMPLATE > $SLAVE_TEMPLATE.tmp.$$
+
+ insertvmtemplatevars $SLAVE_TEMPLATE.tmp.$$
+
+ CMDOUT=`onecmd -e $ENDPOINT onevm create $SLAVE_TEMPLATE.tmp.$$`
+ SLAVEID=`echo $CMDOUT | awk '{print $2}'`
+ RES=$(checknumeric $SLAVEID)
+ if [ $RES -eq 1 ]; then
+ echo "Error encountered creating slave VM. Check template"
+ updateappenvdb $ENVID "-1" "SLAVE_CREATE_FAIL:$CMDOUT"
+ exit -1
+ fi
+
+ insertvmpooldb $SLAVEID $ENVID "VM_CREATE"
+ SLAVELIST=`echo $SLAVELIST $SLAVEID`
+ done
+ echo "SLAVELIST is " $SLAVELIST
+
+
+ for slave in $SLAVELIST
+ do
+ # Wait for slaves to start
+ waitvmstarted $slave $ENVID "SLAVE"
+ # Get IP address of the slave servers
+ SLAVEIP=$(getvmip $slave)
+ # Tell the master IP address of slave
+ yes | ssh -o StrictHostKeyChecking=no $ADMINUSER@$MASTERIP /home/$ADMINUSER/$MASTER_CONTEXT_SCRIPT add $SLAVEIP $SLAVEDATA
+ done
+
+ if [ -n "${APP_URL+x}" ]; then
+ URL=`echo $APP_URL | sed -e "s/%MASTER%/$MASTERIP/g"`
+ updateappurl $ENVID $URL
+ fi
+
+ #Final update indicating we have succesfully completed
+ updateappenvpolicystatusdb $ENVID "READY"
+
+ rm $MASTER_TEMPLATE.tmp.$$
+ if [ -f $SLAVE_TEMPLATE.tmp.$$ ]; then
+ rm $SLAVE_TEMPLATE.tmp.$$
+ fi
+ exit 0
+}
+
+
+# Create 1 slave and add it to master
+flexup () {
+ #Get the master IP for this app env
+ MASTERINFO=$(getmastervm $1)
+ MASTERID=`echo $MASTERINFO | awk '{print $1}'`
+ MASTERIP=`echo $MASTERINFO | awk '{print $2}'`
+ echo "Environment master's IP address is " $MASTERIP
+
+ echo "Flexing up application id $1"
+
+ SLAVE_INDEX=$(getnextslaveindex $1)
+ if [ -z "$SLAVE_CONTEXT_VAR" ]; then
+ SLAVE_CONTEXT_VAR="SLAVE=true"
+ fi
+
+ if [ -n "${ENVDEP+x}" ]; then
+ SLAVE_CONTEXT_VAR=`echo $SLAVE_CONTEXT_VAR $ENVDEP`
+ fi
+
+ SLAVE_CONTEXT_VAR_T=`echo $SLAVE_CONTEXT_VAR | sed -e "s/%MASTER%/$MASTERIP/g; s/%SLAVE_INDEX%/$SLAVE_INDEX/g; s/%NUM_SLAVES%/$SLAVE_INDEX/g;"`
+ sed -e "s/%MASTER%/$MASTERIP/g; s/%SERVICE_NAME%/$SERVICE_NAME/g; s/%CARINA_IP%/$CARINA_IP/g; s/%NAME%/$SLAVE_VM_NAME/g; s/%NETWORK_ID%/$SLAVE_NETWORK_ID/g; s/%IMAGE_ID%/$SLAVE_IMAGE_ID/g; s/%CPU%/$SLAVE_NUM_CPUS/g; s/%MEMORY%/$SLAVE_MEMORY/g; s/%ENVID%/$1/g; s/%APP_PACKAGE%/$APP_PACKAGE/g; s/%APP_CONTEXT_SCRIPT%/$SLAVE_CONTEXT_SCRIPT/g; s/%APP_CONTEXT_VAR%/$SLAVE_CONTEXT_VAR_T/g;" $SLAVE_TEMPLATE > $SLAVE_TEMPLATE.tmp.$$
+
+ insertvmtemplatevars $SLAVE_TEMPLATE.tmp.$$
+
+ CMDOUT=`onecmd -e $ENDPOINT onevm create $SLAVE_TEMPLATE.tmp.$$`
+ SLAVEID=`echo $CMDOUT | awk '{print $2}'`
+ RES=$(checknumeric $SLAVEID)
+ if [ $RES -eq 1 ]; then
+ echo "Error encountered creating slave VM. Check template"
+ updateappenvdb $1 $MASTERID "SLAVE_CREATE_FAIL:$CMDOUT"
+ exit -1
+ fi
+ echo "Created new slave for $1 with VM ID $SLAVEID"
+ # Add entry to DB
+ insertvmpooldb $SLAVEID $1 "VM_CREATE"
+
+ # Wait for slaves to start
+ waitvmstarted $SLAVEID $1 "SLAVE"
+ # Get IP address of the slave servers
+ SLAVEIP=$(getvmip $SLAVEID)
+ # Tell the master IP address of slave
+ yes | ssh -o StrictHostKeyChecking=no $ADMINUSER@$MASTERIP /home/$ADMINUSER/$MASTER_CONTEXT_SCRIPT add $SLAVEIP $SLAVEDATA
+ if [ $? -eq 0 ]; then
+ updateappenvdb $1 $MASTERID "SLAVE_SCALEUP_SUCCESS"
+ else
+ updateappenvdb $1 $MASTERID "SLAVE_SCALEUP_FAIL: ssh to master failed"
+ fi
+
+ rm $SLAVE_TEMPLATE.tmp.$$
+}
+
+# Remove 1 slave and inform master
+flexdown () {
+ echo "Flexing down application id $1"
+ #Get the master IP for this app env
+ MASTERINFO=$(getmastervm $1)
+ MASTERID=`echo $MASTERINFO | awk '{print $1}'`
+ echo "Environment Master's id is " $MASTERID
+
+ LASTSLAVE=$(getlastslavevm $1)
+ LASTSLAVEID=`echo $LASTSLAVE | awk '{print $1}'`
+ if [[ $LASTSLAVEID == $MASTERID ]]; then
+ echo "No slaves for this environment"
+ return
+ fi
+ MASTERIP=`echo $MASTERINFO | awk '{print $2}'`
+ LASTSLAVEIP=`echo $LASTSLAVE | awk '{print $2}'`
+
+ # Tell the environment master to remove the slave
+ echo "Telling master VM to remove slave IP $LASTSLAVEIP"
+ yes | ssh -o StrictHostKeyChecking=no $ADMINUSER@$MASTERIP /home/$ADMINUSER/$MASTER_CONTEXT_SCRIPT delete $LASTSLAVEIP $SLAVEDATA
+
+ # Wait for the slave to drain
+ echo "Waiting for slave to complete existing requests"
+ sleep 30
+
+ # Shutdown the slave VM. Dont delete because we could restart it in a flexup
+ LASTSLAVEID=`echo $LASTSLAVE | awk '{print $1}'`
+ echo "Shu