Permalink
Browse files

improve docs

  • Loading branch information...
1 parent fe705ac commit 4f91ae6e556183b806bda9c9d3092a568ae16ae6 @wujj123456 wujj123456 committed Apr 9, 2012
Showing with 65 additions and 46 deletions.
  1. +19 −5 README
  2. +1 −1 build.xml
  3. +3 −4 machine.cfg
  4. +15 −12 powercaplocal.py
  5. +4 −2 sqs.py
  6. +23 −22 src/experiments/PowerCappingExperiment.java
View
@@ -5,7 +5,7 @@ Release v0.1
1. Introduction
-Bighouse targets specifically to investigate issues of data center design at scale. At its core, Bighouse is a methodology for system characterization and discrete-event simulation to enable quantitative exploration of data center-level challenges, such as performance optimization, power provisioning, power management, distributed data placement, and fault-tolerant design.
+BigHouse targets specifically to investigate issues of data center design at scale. At its core, Bighouse is a methodology for system characterization and discrete-event simulation to enable quantitative exploration of data center-level challenges, such as performance optimization, power provisioning, power management, distributed data placement, and fault-tolerant design.
2. Platforms
@@ -14,7 +14,7 @@ BigHouse is developed under Mac OS and Linux in Java and Python. BigHouse shoul
2.1 Platform requirements for a local experiment
-Only Java installation is required to run a local experiment. We tested the code with Java 1.6.0. Versions too early might not work as the code uses RMI functions, but later versions should be OK. Both OpenJDK and Sun Java works. However, we recommend that you use the default openjdk package in your distro.
+Only Java installation is required to run a local experiment. We tested the code with Java 1.6.0. Versions too early might not work as the code uses RMI functions, but later versions should be OK. Both OpenJDK and Sun Java works. However, we recommend that you use the default OpenJDK package in your distro.
2.2 Platform requirements for a distributed experiment
@@ -52,6 +52,11 @@ After you compile the local experiment, You can run the compiled experiment with
"java -jar powercap.jar ./ csedns 100"
This experiment uses csedns as workload, and simulates a power capping experiment with 100 servers. It will output average response time and its 95 percentile.
+Optionally, you can use Python to run a local experiment. Please refer to 2.2 for the software setup. To run the Python experiment, execute:
+./powercaplocal.py
+
+The Java and Python code perform the same experiment. BigHouse periodically outputs checkpoint information. Results are displayed at the end.
+
If the code runs correctly and you don not plan to run experiments on a cluster, you can directly skip to section 6.
@@ -72,17 +77,17 @@ Follow these instructions for first-time setup:
3) Setup autossh across slaves. The master should be able to silently log on to all slaves via ssh and execute commands. You can run "./sqs.py setup machine.cfg" to setup autossh. You can use "./sqs.py copy machine.cfg" to test autossh funtionality.
Once your servers are configured, you can compile your code (Section 2), and run the experiment with sqs.py.
-1) "./sqs.py run machine.cfg" will start the default experiment defined in powercap.py. Please refer to powercap.py if you want to change experiment parameters (e.g. statistic targets, datacenter configurations, etc).
+1) "./sqs.py run machine.cfg powercap.py" will start the example experiment defined in powercap.py. Please refer to powercap.py if you want to change experiment parameters (e.g. statistic targets, datacenter configurations, etc).
2) You can use "./sqs.py kill machine.cfg" to cleanup if necessary.
-When the simulation finishes, "Final Statistics" sections will contain SOJOURN_TIME and TOTAL_CAPPING results.
+When the simulation finishes, "Final Statistics" section will contain SOJOURN_TIME and TOTAL_CAPPING results.
6. Development and Debugging
The local power capping experiment is located in /src/experiments/PowerCappingExperiment.java. Start by understanding the Java source code. If you don't plan to run experiments on multiple machines, it's not necessary to use the Python interface.
-Once you've done debugging the Java class and decided to run experiments on multiple machines, it should be straight forward to port it to a Python file. Please refer to the comments in /powercap.py for instructions.
+Once you've done debugging the Java class and decided to run experiments on multiple machines, it should be straight forward to port it to a Python file. Please refer to next section.
Python serves two purposes in BigHouse:
1) Distribute and run the experiment on multiple machines (sqs.py)
@@ -91,3 +96,12 @@ Python serves two purposes in BigHouse:
Due to limitation of JPype's exception handling, it won't produce the correct traceback when an exception occurs. In addition, during development, it's not necessary to run the code on multiple machines. Therefore, development is always done in Java.
Sample Java classes are provided in src/experiments folder, and the Java counterpart of powercap.py is DistributedPowerCappingExperiment.java. A local experiment PowerCappingExperiment.java and a Python port powercaplocal.py are also provided for reference.
+
+
+7. Porting a Java experiment to Python
+
+JPype directly invokes Java methods in a Python file. Therefore, porting a Java experiment to Python is quite easy. Follow the following steps:
+1) Import jar files and packages. These formats are specified by JPype, and is required for invoking Java methods.
+2) Port the createExperiment() method to Python. It's mostly copy and paste, as Python code should follow exactly the same procedure to create an experiment. In case some data type conversion is required, please refer to: http://jpype.sourceforge.net/doc/user-guide/userguide.html#conversion
+3) For a local experiment, you also need to manage the result output. (Again, duplicate your Java code). For a distributed experiment, you only need to replace createExperiment() in powercap.py, and the rest of the script takes care of warming up the experiment on the master, distribute it to slaves and collect results.
+
View
@@ -52,7 +52,7 @@
<property name="test-dir" location="${src-dir}/test" />
<property name="test-bin-dir" location="${bin-dir}/test" />
<property name="test-src-dir" location="${src-dir}/test" />
- <property name="junit-jar" location="junit-4.10.jar" />
+ <property name="junit-jar" location="junit.jar" />
<property name="ssj-jar" location="ssj.jar" />
<property name="hamcrest-jar" location="org.hamcrest.core_1.1.0.jar" />
View
@@ -5,8 +5,7 @@
# Hostname can be localhost if you also want to run slave experiments on the master
# Path can be temporary directories
# Each slave will launch NumberOfInstances clients for simulation
-absolut.eecs.umich.edu wujj /tmp 2
-boru.eecs.umich.edu wujj /tmp 4
-#stoli.eecs.umich.edu wujj /tmp 4
-skyy.eecs.umich.edu wujj /tmp 4
+example.eecs.umich.edu test /tmp 2
+192.168.50.10 test /tmp 4
+localhost test /tmp 4
View
@@ -51,23 +51,25 @@ def createExperiment(xValues = []):
scaledQps = (qps/arrivalScale)
# debug output
- print "Cores: %s" % cores
- print "rho: %s" % rho
- print "recalc rho: %s" % (scaledQps/(cores*(1/averageServiceTime)))
- print "arrivalScale: %s" % arrivalScale
- print "Average interarrival time: %s" % averageInterarrival
- print "QPS as is %s" % qps
- print "Scaled QPS: %s" % scaledQps
- print "Service rate as is %s" % serviceRate
- print "Service rate x: %s" % cores + " is: %s" % ((serviceRate)*cores)
- print "\n------------------\n"
+# print "Cores: %s" % cores
+# print "rho: %s" % rho
+# print "recalc rho: %s" % (scaledQps/(cores*(1/averageServiceTime)))
+# print "arrivalScale: %s" % arrivalScale
+# print "Average interarrival time: %s" % averageInterarrival
+# print "QPS as is %s" % qps
+# print "Scaled QPS: %s" % scaledQps
+# print "Service rate as is %s" % serviceRate
+# print "Service rate x: %s" % cores + " is: %s" % ((serviceRate)*cores)
+# print "\n------------------\n"
# setup experiment
experimentInput = core.ExperimentInput()
rand = generator.MTRandom(long(1))
arrivalGenerator = generator.EmpiricalGenerator(rand, arrivalDistribution, "arrival", arrivalScale)
serviceGenerator = generator.EmpiricalGenerator(rand, serviceDistribution, "service", 1.0)
+
+ # add experiment outputs
experimentOutput = core.ExperimentOutput()
experimentOutput.addOutput(StatName.SOJOURN_TIME, meanPrecision, quantileSetting, quantilePrecision, warmupSamples)
experimentOutput.addOutput(StatName.SERVER_LEVEL_CAP, meanPrecision, quantileSetting, quantilePrecision, warmupSamples)
@@ -111,13 +113,14 @@ def createExperiment(xValues = []):
experiment.run()
# experiment finished
+print "====== Results ======"
responseTimeMean = experiment.getStats().getStat(StatName.SOJOURN_TIME).getAverage()
print "SOJOURN_TIME mean : %s" % responseTimeMean
responseTimeQuantile = experiment.getStats().getStat(StatName.SOJOURN_TIME).getQuantile(quantileSetting)
print "%s quantile SOJOURN_TIME : %s" % (quantileSetting, responseTimeQuantile)
cappingMean = experiment.getStats().getStat(StatName.SERVER_LEVEL_CAP).getAverage()
print "Average Server Cap : %s" % cappingMean
-print "========== JPype Output Begin =========="
+print "====== JPype Output Begin (ignore this)======="
jpype.shutdownJVM()
-print "========== JPype Output End ============"
+print "====== JPype Output End (ignore this) ======="
View
@@ -57,8 +57,7 @@ def usage():
- setup: setup auto-ssh connection
- kill: kill all running slave.jar, rmiregistry processes
- copy: push binaries from master to slaves without running the simulation
- - run: run the simulation
- - default experimentconfig is powercap.py''')
+ - run: run the simulation, must specify experimentconfig''')
sys.exit(1)
def dPrint(text):
@@ -279,6 +278,9 @@ def main(argv):
if argv[0] == "run":
if len(argv) == 3:
experimentCfg = argv[2]
+ else:
+ dPrint('''To run an experiment, experimentconfig must be specified''')
+ usage()
dPrint('''Experiment config file: %s''' % experimentCfg)
runSimulation()
elif argv[0] == "setup":
@@ -57,16 +57,12 @@ public PowerCappingExperiment(){
}//End PowerCappingExperiment()
public void run(String workloadDir, String workload, int nServers) {
-
-
- ExperimentInput experimentInput = new ExperimentInput();
-// String arrivalFile = workloadDir+"workloads/www.arrival.cdf";
-// String serviceFile = workloadDir+"workloads/www.service.cdf";
+ // service file
String arrivalFile = workloadDir+"workloads/"+workload+".arrival.cdf";
String serviceFile = workloadDir+"workloads/"+workload+".service.cdf";
- System.out.println("arrival file "+arrivalFile);
- System.out.println("service file "+arrivalFile);
+
+ // specify distribution
int cores = 4;
int sockets = 1;
double targetRho = .5;
@@ -83,32 +79,33 @@ public void run(String workloadDir, String workload, int nServers) {
double serviceRate = 1/averageServiceTime;
double scaledQps =(qps/arrivalScale);
+// System.out.println("Cores " + cores);
+// System.out.println("rho " + rho);
+// System.out.println("recalc rho " + scaledQps/(cores*(1/averageServiceTime)));
+// System.out.println("arrivalScale " + arrivalScale);
+// System.out.println("Average interarrival time " + averageInterarrival);
+// System.out.println("QPS as is " +qps);
+// System.out.println("Scaled QPS " +scaledQps);
+// System.out.println("Service rate as is " + serviceRate);
+// System.out.println("Service rate x" + cores + " is: "+ (serviceRate)*cores);
+// System.out.println("\n------------------\n");
- System.out.println("Cores " + cores);
- System.out.println("rho " + rho);
- System.out.println("recalc rho " + scaledQps/(cores*(1/averageServiceTime)));
- System.out.println("arrivalScale " + arrivalScale);
- System.out.println("Average interarrival time " + averageInterarrival);
- System.out.println("QPS as is " +qps);
- System.out.println("Scaled QPS " +scaledQps);
- System.out.println("Service rate as is " + serviceRate);
- System.out.println("Service rate x" + cores + " is: "+ (serviceRate)*cores);
- System.out.println("\n------------------\n");
+ // setup experiment
+ ExperimentInput experimentInput = new ExperimentInput();
MTRandom rand = new MTRandom(1);
-
EmpiricalGenerator arrivalGenerator = new EmpiricalGenerator(rand, arrivalDistribution, "arrival", arrivalScale);
EmpiricalGenerator serviceGenerator = new EmpiricalGenerator(rand, serviceDistribution, "service", 1.0);
+
+ // add experiment outputs
ExperimentOutput experimentOutput = new ExperimentOutput();
experimentOutput.addOutput(StatName.SOJOURN_TIME, .05, .95, .05, 5000);
experimentOutput.addOutput(StatName.SERVER_LEVEL_CAP, .05, .95, .05, 5000);
-// experimentOutput.addTimeWeightedOutput(TimeWeightedStatName.SERVER_POWER, .01, .5, .01, 50000, .001);
Experiment experiment = new Experiment("Power capping test", rand, experimentInput, experimentOutput);
+ // setup datacenter
DataCenter dataCenter = new DataCenter();
-// public PowerCappingEnforcer(Experiment experiment, double capPeriod, double globalCap, double maxPower, double minPower) {
-// int nServers = 100;
double capPeriod = 1.0;
double globalCap = 65*nServers;
double maxPower = 100*nServers;
@@ -137,9 +134,13 @@ public void run(String workloadDir, String workload, int nServers) {
dataCenter.addServer(server);
}//End for i
-
experimentInput.addDataCenter(dataCenter);
+
+ // run the experiment
experiment.run();
+
+ // display results
+ System.out.println("====== Results ======");
double responseTimeMean = experiment.getStats().getStat(StatName.SOJOURN_TIME).getAverage();
System.out.println("Response Mean: " + responseTimeMean);
double responseTime95th = experiment.getStats().getStat(StatName.SOJOURN_TIME).getQuantile(.95);

0 comments on commit 4f91ae6

Please sign in to comment.