Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse code

Merge branch 'master' of github.com:twitter/iago

Conflicts:
	README.md
  • Loading branch information...
commit 4294510ed110206ec031e5a405c078b9dd0bf780 2 parents d8ea987 + 7a070c1
authored December 27, 2012

Showing 23 changed files with 431 additions and 43 deletions. Show diff stats Hide diff stats

  1. 68  README.md
  2. 28  examples/sbt/.gitignore
  3. 29  examples/sbt/README
  4. 22  examples/sbt/config/echo.scala
  5. BIN  examples/sbt/libs/iago-0.5.1.jar
  6. 8  examples/sbt/project/build.properties
  7. 39  examples/sbt/project/build/Project.scala
  8. 13  examples/sbt/project/plugins/Plugins.scala
  9. 36  examples/sbt/src/main/java/com/twitter/jexample/EchoLoadTest.java
  10. 33  examples/sbt/src/main/scala/com/twitter/example/EchoClient.scala
  11. 26  examples/sbt/src/main/scala/com/twitter/example/EchoLoadTest.scala
  12. 67  examples/sbt/src/main/scala/com/twitter/example/EchoServer.scala
  13. 6  examples/sbt/src/main/thrift/echo/EchoService.thrift
  14. 48  examples/sbt/src/scripts/echo-server.sh
  15. 20  pom.xml
  16. 4  src/main/resources/scripts/parrot-feeder.sh
  17. 4  src/main/resources/scripts/parrot-server.sh
  18. 2  src/main/scala/com/twitter/parrot/config/ParrotServerConfig.scala
  19. 3  src/main/scala/com/twitter/parrot/launcher/ParrotLauncher.scala
  20. 4  src/main/scala/com/twitter/parrot/processor/SimpleRecordProcessor.scala
  21. 7  src/main/scala/com/twitter/parrot/server/FinagleTransport.scala
  22. 4  src/test/scala/com/twitter/parrot/integration/EndToEndSpec.scala
  23. 3  src/test/scala/com/twitter/parrot/server/ParrotUdpTransportSpec.scala
68  README.md
Source Rendered
@@ -25,9 +25,9 @@
25 25
 
26 26
 ## Iago Quick Start
27 27
 
28  
-NOTE: This repo has only recently been made public and our velocity is high at the moment, with significant work being done on documentation in particular. Please join iago-users@googlegroups.com (https://groups.google.com/d/forum/iago-users) for updates and to ask pressing questions.
  28
+NOTE: This repo has only recently been made public and our velocity is high at the moment. Please join [iago-users@googlegroups.com](https://groups.google.com/d/forum/iago-users) for updates and to ask questions.
29 29
 
30  
-If you are already familiar with the Iago Load Generation tool, follow these steps to get started; otherwise, start with the <a href="#Iago Overview">Iago Overview</a>. For questions, please contact <a href="mailto:iago-users@googlegroups.com">iago-users@googlegroups.com</a>.
  30
+If you are already familiar with the Iago Load Generation tool, follow these steps to get started; otherwise, start with the <a href="http://twitter.github.com/iago/">Iago Overview</a> and perhaps <a href="http://twitter.github.com/iago/philosophy.html">Iago Philosophy</a>, also known as "Why Iago?". For questions, please contact [iago-users@googlegroups.com](https://groups.google.com/d/forum/iago-users).
31 31
 
32 32
 <a name="Iago Prerequisites"></a>
33 33
 
@@ -36,7 +36,7 @@ If you are already familiar with the Iago Load Generation tool, follow these ste
36 36
 1. Download and unpack the Iago distribution.
37 37
 We support Scala 2.9 and recommend you clone the latest master: <a href="https://github.com/twitter/iago/zipball/master">master</a>.
38 38
 
39  
-2. Read the documentation. We'll be adding more recommended steps here shortly.
  39
+2. Read the documentation.
40 40
 
41 41
 <a name="Preparing Your Test"></a>
42 42
 
@@ -50,7 +50,18 @@ We support Scala 2.9 and recommend you clone the latest master: <a href="https:/
50 50
 
51 51
 ### Executing Your Test
52 52
 
53  
-Launch Iago from the distribution with `java` `-jar` *iago_jar* `-f` *your_config*. This will create the Iago processes for you and configure it to use your transactions. To kill a running job, add `-k` to your launch parameters: `java` `-jar` *parrot_jar* `-f` *your_config* `-k`.
  53
+Launch Iago from the distribution with `java` `-jar` *iago_jar* `-f` *your_config*. This will create the Iago processes for you and configure it to use your transactions. To kill a running job, add `-k` to your launch parameters: `java` `-jar` *iago_jar* `-f` *your_config* `-k`.
  54
+
  55
+If you launch your Iago job on your local machine and an old Iago job is still running, it probably won't get far: it will attempt to re-use a port and fail. You want to kill the running job, as described above.
  56
+
  57
+<em>If you build via Maven,</em> then you might wonder "How do I launch Iago 'from the distribution'?" The steps are:
  58
+<pre>
  59
+% <kbd>mvn package -DskipTests</kbd>
  60
+% <kbd>mkdir tmp; cd tmp</kbd>
  61
+% <kbd>unzip ../target/iago-<var>version</var>-package-dist.zip</kbd>
  62
+% <kbd>java -jar iago-<var>version</var>.jar -f config/<var>my_config</var>.scala</kbd>
  63
+</pre>
  64
+Don't assume that you can skip the package/unzip steps if you're just changing a config file. You need to re-package and unzip again.
54 65
 
55 66
 If you are using Iago as a library, for example, in the case of testing over the Thrift protocol or building more complex tests with HTTP or Memcached/Kestrel, you should instead add a task to your project's configuration. See <a href="#Configuring Your Test">Configuring Your Test</a> for more information.
56 67
 
@@ -72,12 +83,13 @@ Replaying transactions at a fixed rate enables you to study the behavior of your
72 83
 
73 84
 ### Supported Services
74 85
 
75  
-Iago understands service requests in the following formats:
  86
+Iago can generate service requests that travel the net in different ways and are in different formats. The code that does this is in a Transport, a class that extends <code>ParrotTransport</code>. Iago comes with several Transports already defined. When you configure your test, you will need to set some parameters; to understand which of those parameters are used and how they are used, you probably want to look at the source code for your test's Transport class.
76 87
 
77  
-* HTTP
78  
-* Thrift
79  
-* Memcached / Kestrel
80  
-* UDP
  88
+* HTTP: Use <a href="https://github.com/twitter/iago/blob/master/src/main/scala/com/twitter/parrot/server/FinagleTransport.scala">FinagleTransport</a>
  89
+* Thrift: Use <a href="https://github.com/twitter/iago/blob/master/src/main/scala/com/twitter/parrot/server/ThriftTransport.scala">ThriftTransport</a>
  90
+* Memcached: Use <a href="https://github.com/twitter/iago/blob/master/src/main/scala/com/twitter/parrot/server/MemcacheTransport.scala">MemcacheTransport</a>
  91
+* Kestrel: Use <a href="https://github.com/twitter/iago/blob/master/src/main/scala/com/twitter/parrot/server/KestrelTransport.scala">KestrelTransport</a>
  92
+* UDP: Use <a href="https://github.com/twitter/iago/blob/master/src/main/scala/com/twitter/parrot/server/ParrotUdpTransport.scala">ParrotUdpTransport</a>
81 93
 
82 94
 Your service is typically an HTTP or Thrift service written in either Scala or Java.
83 95
 
@@ -92,6 +104,8 @@ For replay, Iago recommends you scrub your logs to only include requests which m
92 104
 * **Idempotent**, meaning that re-execution of a transaction any number of times yields the same result as the initial execution.
93 105
 * **Commutative**, meaning that transaction order is not important. Although transactions are initiated in replay order, Iago's internal behavior may change the actual execution order to guarantee the transaction rate. Also, transactions that implement `Future` responses are executed asynchronously. You can achieve ordering, if required, by using Iago as a library and initiating new requests in response to previous ones. Examples of this are available.
94 106
 
  107
+Unless you change your configuration's <code>reuseFile</code> parameter, make sure that your sample log has at least 1000 items.
  108
+
95 109
 [Top](#Top)
96 110
 
97 111
 <a name="Sources of Transactions"></a>
@@ -318,7 +332,11 @@ You define your Iago subclass to execute your service and map transactions to re
318 332
 
319 333
 ## Configuring Your Test
320 334
 
321  
-To configure your test, create a `launcher.scala` file that that creates a `ParrotLauncherConfig` instance with the configuration parameters you want to set. The following example shows parameters for testing a Thrift service:
  335
+To configure your test, create a `launcher.scala` file that that creates a `ParrotLauncherConfig` instance with the configuration parameters you want to set.
  336
+
  337
+There are several parameters to set. A good one to <a href="#Supported Services">figure out early is <code>transport</code></a>; that will in turn help you to find out what, e.g., <code>responseType</code> you need.
  338
+
  339
+The following example shows parameters for testing a Thrift service:
322 340
 
323 341
 ```scala
324 342
 import com.twitter.parrot.config.ParrotLauncherConfig
@@ -332,8 +350,7 @@ new ParrotLauncherConfig {
332 350
   requestRate = 1
333 351
   numInstances = 1
334 352
   duration = 5
335  
-  timeUnit = "MINUTES"
336  
-  role = "preflight"
  353
+  timeUnit = "MINUTES" // affects duration; does not affect requestRate
337 354
 
338 355
   imports = "import com.twitter.example.EchoLoadTest"
339 356
   responseType = "Array[Byte]"
@@ -363,7 +380,7 @@ You can specify any of the following parameters:
363 380
 </tr>
364 381
 <tr>
365 382
     <td><code>log</code></td>
366  
-    <td><p>A string value that specifies the complete path to the log you want Iago to replay. The log should be on your local file system.</p>
  383
+    <td><p>A string value that specifies the complete path to the log you want Iago to replay. The log should be on your local file system. The log should have at least 1000 items or you should change the <code>reuseFile</code> parameter.</p>
367 384
     <p><b>Example: </b><code>log = "logs/yesterday.log"</code></p></td>
368 385
     <td><b>Required</b></td>
369 386
 </tr>
@@ -401,7 +418,7 @@ You can specify any of the following parameters:
401 418
     <td><code>header</code></td>
402 419
     <td><p>A string value that specifies the HTTP Host header.</p>
403 420
     <p><b>Example: </b><code>header = "api.yourdomain.com"</code></p></td>
404  
-    <td><code>api.yourdomain.com</code></td>
  421
+    <td><code>""</code></td>
405 422
 </tr>
406 423
 <tr>
407 424
     <td><code>duration</code></td>
@@ -436,23 +453,30 @@ You can specify any of the following parameters:
436 453
     <td><code>requestRate</code></td>
437 454
     <td><p>An integer value that specifies the number of requests per second to submit to your service.</p>
438 455
     <p><b>Example: </b><code>requestRate = 10</code></p>
439  
-    <p>Note: if using multiple server instances, requestRate is per-instance, not aggregate, so effective rate becomes numInstances * requestRate.</p></td>
  456
+    <p>Note: if using multiple server instances, requestRate is per-instance, not aggregate.</p></td>
440 457
     <td><code>1</code></td>
441 458
 </tr>
442 459
 <tr>
443 460
     <td><code>loggers</code></td>
444 461
     <td><p>A List of LoggerFactories; allows you to define the type and level of logging you want</p>
445  
-    <p><b>Example: </b><code>loggers = new LoggerFactory(
446  
-    level = Level.DEBUG
447  
-    handlers = ConsoleHandler
448  
-  ) = "preflight"</code></p></td>
  462
+    <p><b>Example:</b></p>
  463
+<pre>import com.twitter.logging.LoggerFactory
  464
+import com.twitter.logging.config._
  465
+
  466
+new ParrotLauncherConfig {
  467
+  ...
  468
+  loggers = new LoggerFactory(
  469
+    level = Level.DEBUG,
  470
+    handlers = new ConsoleHandlerConfig()
  471
+  )
  472
+} </pre></td>
449 473
     <td><i>Nil</i></td>
450 474
 </tr>
451 475
 <tr>
452 476
     <td><code>numFeederInstances</code></td>
453 477
     <td><p>Will bring up the specified number of feeder instances</p>
454 478
     <p><b>Example: </b><code>numFeederInstances = 2</code></p></td>
455  
-    <td><i>1</i></td>
  479
+    <td>1</td>
456 480
 </tr>
457 481
 <tr>
458 482
     <td><code>numInstances</code></td>
@@ -540,7 +564,9 @@ You can specify any of the following parameters:
540 564
 <tr>
541 565
     <td><code>createDistribution</code></td>
542 566
     <td><p>You can use this field to create your own distribution rate, instead of having a constant flow. You will need to create a subclass of RequestDistribution and import it.</p>
543  
-    <p><b>Example: </b><code>createDistribution = "createDistribution = { rate => new MyDistribution(rate) }"</code></p></td>
  567
+    <p><b>Example: </b><pre>createDistribution = """createDistribution = {
  568
+    rate => new MyDistribution(rate)
  569
+}"""</pre></p></td>
544 570
     <td><i>""</i></td>
545 571
 </tr>
546 572
 </tbody>
28  examples/sbt/.gitignore
... ...
@@ -0,0 +1,28 @@
  1
+target/
  2
+dist/
  3
+project/boot/
  4
+project/plugins/project/
  5
+project/plugins/src_managed/
  6
+**/project/boot/
  7
+**/project/plugins/project/
  8
+**/project/plugins/src_managed/
  9
+*.log
  10
+*.tmproj
  11
+lib_managed/
  12
+.*.sw[a-z]
  13
+*.iml
  14
+*.pyc
  15
+.idea/
  16
+.idea/*
  17
+.DS_Store
  18
+.ensime
  19
+.ivyjars
  20
+.run_classpath
  21
+Gemfile.lock
  22
+/.py/
  23
+#*
  24
+*#
  25
+.#*
  26
+*~
  27
+.*~
  28
+/bin/sbt-launch-0.7.5.jar
29  examples/sbt/README
... ...
@@ -0,0 +1,29 @@
  1
+This is the README for an example usage of Iago. This brings up an echo server on your localhost,
  2
+then creates an Iago job to send traffic to your server.
  3
+
  4
+NOTE: If you are unfamiliar with sbt, check out project/build/Project.scala
  5
+NOTE2: If you are unfamiliar with how to use Iago, check out src & config/echo.scala. Also read the Iago README.
  6
+
  7
+
  8
+SBT COMMANDS
  9
+sbt update:                   will pull in all of your dependencies
  10
+sbt compile*:                  compiles.
  11
+sbt clean-dist:               cleans out any previous distributions
  12
+sbt package-dist*:             packages and creates a distribution at dist/
  13
+
  14
+*prepend NO_VALIDATE=1 and NO_TESTS=1 to stop the tests from running
  15
+(there's a currently known bug in our dependencies that causes the tests to fail)
  16
+
  17
+
  18
+BEFORE YOU DO ANYTHING
  19
+0) copy the iago jar to libs/
  20
+1) run "sbt update"
  21
+
  22
+TO START UP ECHO SERVER
  23
+0) run "NO_TESTS=1 NO_VALIDATE=1 sbt clean-dist package-dist" to create a distribution
  24
+1) "cd dist/sbt-example"
  25
+2) "./scripts/echo-server.sh start-local"
  26
+
  27
+TO RUN IAGO
  28
+0) "cd dist/sbt-example"
  29
+1) "java -jar sbt-example-1.0.jar -f config/echo.scala"
22  examples/sbt/config/echo.scala
... ...
@@ -0,0 +1,22 @@
  1
+import com.twitter.parrot.config.ParrotLauncherConfig
  2
+
  3
+new ParrotLauncherConfig {
  4
+  distDir = "."
  5
+  jobName = "load_echo"
  6
+  port = 50017
  7
+  victims = "localhost"
  8
+  log = "config/sample.log"
  9
+  requestRate = 1
  10
+  numInstances = 1
  11
+  duration = 5
  12
+  timeUnit = "MINUTES"
  13
+  reuseFile = true
  14
+  localMode = true
  15
+
  16
+  imports = "import com.twitter.example.EchoLoadTest"
  17
+  responseType = "Array[Byte]"
  18
+  transport = "ThriftTransport"
  19
+  loadTest = "new EchoLoadTest(service.get)"
  20
+  parser = "thrift"
  21
+}
  22
+
BIN  examples/sbt/libs/iago-0.5.1.jar
Binary file not shown
8  examples/sbt/project/build.properties
... ...
@@ -0,0 +1,8 @@
  1
+#Project properties
  2
+#Tue Aug 16 15:22:32 PDT 2011
  3
+project.organization=com.twitter
  4
+project.name=sbt-example
  5
+sbt.version=0.7.4
  6
+project.version=1.0
  7
+build.scala.versions=2.9.2
  8
+project.initialize=false
39  examples/sbt/project/build/Project.scala
... ...
@@ -0,0 +1,39 @@
  1
+import sbt._
  2
+import com.twitter.sbt._
  3
+
  4
+class Project(info: ProjectInfo) extends StandardServiceProject(info)
  5
+  with PackageDist
  6
+  with CompileThriftFinagle
  7
+  with DefaultRepos {
  8
+
  9
+  //Assumes you are using SBT for iago as well, and that it's in ivy
  10
+  // (ex. ~/.ivy2/local/com/twitter/iago/0.5.1/iago-0.5.1.jar)
  11
+
  12
+  val libThrift = "thrift" % "libthrift" % "0.5.0"
  13
+  val finagleCore = "com.twitter" % "finagle-core" % "5.3.0"
  14
+  val finagleServerSets = "com.twitter" % "finagle-serversets" % "5.3.0"
  15
+  val finagleMemcached = "com.twitter" % "finagle-memcached" % "5.3.0"
  16
+  val finagleStream = "com.twitter" % "finagle-stream" % "5.3.0"
  17
+  val finagleThrift = "com.twitter" % "finagle-thrift" % "5.3.0"
  18
+  val finagleOstrich = "com.twitter" % "finagle-ostrich4" % "5.3.0"
  19
+  val iago = "com.twitter" % "iago" % "0.5.1"
  20
+  val ostrich = "com.twitter" % "ostrich" % "8.0.1"
  21
+  val utilThrift = "com.twitter" % "util-thrift" % "5.3.0"
  22
+  def customRun(mainClass: String, args: String*) = task { _ =>
  23
+    runTask(Some(mainClass), runClasspath, args) dependsOn(compile, copyResources)
  24
+  }
  25
+
  26
+  lazy val server = customRun("com.twitter.example.EchoServer")
  27
+  lazy val client = customRun("com.twitter.example.EchoClient")
  28
+
  29
+  lazy val startParrot = customRun("com.twitter.parrot.launcher.LauncherMain", "-f", "config/echo.scala")
  30
+  lazy val killParrot = customRun("com.twitter.parrot.launcher.LauncherMain", "-f", "config/echo.scala", "-k")
  31
+
  32
+  override def mainClass = Some("com.twitter.parrot.launcher.LauncherMain")
  33
+  override def ivyXML =
  34
+    <dependencies>
  35
+      <exclude org="javax.jms"/>
  36
+      <exclude org="com.sun.jdmk"/>
  37
+      <exclude org="com.sun.jmx"/>
  38
+    </dependencies>
  39
+}
13  examples/sbt/project/plugins/Plugins.scala
... ...
@@ -0,0 +1,13 @@
  1
+import sbt._
  2
+import scala.collection.jcl
  3
+
  4
+class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
  5
+  val environment = jcl.Map(System.getenv())
  6
+
  7
+  override def repositories = super.repositories++ Seq("twitter.com" at "http://maven.twttr.com/")
  8
+  
  9
+  override def ivyRepositories = super.ivyRepositories ++ repositories
  10
+
  11
+    val defaultProject = "com.twitter" % "standard-project" % "0.12.10"
  12
+    val sbtThrift      = "com.twitter" % "sbt-thrift" % "1.4.2"
  13
+}
36  examples/sbt/src/main/java/com/twitter/jexample/EchoLoadTest.java
... ...
@@ -0,0 +1,36 @@
  1
+package com.twitter.jexample;
  2
+
  3
+import com.twitter.example.thrift.EchoService;
  4
+import com.twitter.parrot.processor.ThriftLoadTest;
  5
+import com.twitter.parrot.server.ParrotRequest;
  6
+import com.twitter.parrot.server.ParrotService;
  7
+import com.twitter.parrot.thrift.ParrotJob;
  8
+import com.twitter.util.Future;
  9
+import com.twitter.util.FutureEventListener;
  10
+import org.apache.thrift.protocol.TBinaryProtocol;
  11
+
  12
+import java.util.List;
  13
+
  14
+public class EchoLoadTest extends ThriftLoadTest {
  15
+    EchoService.ServiceToClient client = null;
  16
+
  17
+    public EchoLoadTest(ParrotService<ParrotRequest, byte[]> parrotService) {
  18
+	super(parrotService);
  19
+        client = new EchoService.ServiceToClient(service(), new TBinaryProtocol.Factory());
  20
+    }
  21
+
  22
+    public void processLines(ParrotJob job, List<String> lines) {
  23
+        for(String line: lines) {
  24
+            Future<String> future = client.echo(line);
  25
+            future.addEventListener(new FutureEventListener<String>() {
  26
+                public void onSuccess(String msg) {
  27
+                    System.out.println("response: " + msg);
  28
+                }
  29
+
  30
+                public void onFailure(Throwable cause) {
  31
+                    System.out.println("Error: " + cause);
  32
+                }
  33
+            });
  34
+        }
  35
+    }
  36
+}
33  examples/sbt/src/main/scala/com/twitter/example/EchoClient.scala
... ...
@@ -0,0 +1,33 @@
  1
+package com.twitter.example
  2
+
  3
+import java.net.InetSocketAddress
  4
+import org.apache.thrift.protocol.TBinaryProtocol
  5
+
  6
+import com.twitter.finagle.builder.ClientBuilder
  7
+import com.twitter.finagle.Service
  8
+import com.twitter.finagle.thrift.{ThriftClientFramedCodec, ThriftClientRequest}
  9
+
  10
+import thrift.EchoService
  11
+
  12
+object EchoClient {
  13
+  def main(args: Array[String]) {
  14
+    // Create a raw Thrift client service. This implements the
  15
+    // ThriftClientRequest => Future[Array[Byte]] interface.
  16
+    val service: Service[ThriftClientRequest, Array[Byte]] = ClientBuilder()
  17
+      .hosts(new InetSocketAddress(EchoServer.port))
  18
+      .codec(ThriftClientFramedCodec())
  19
+      .hostConnectionLimit(1)
  20
+      .build()
  21
+
  22
+    // Wrap the raw Thrift service in a Client decorator. The client
  23
+    // provides a convenient procedural interface for accessing the Thrift
  24
+    // server.
  25
+    val client = new EchoService.ServiceToClient(service, new TBinaryProtocol.Factory())
  26
+
  27
+    client.echo("hello") onSuccess { response =>
  28
+      println("Received response: " + response)
  29
+    } ensure {
  30
+      service.release()
  31
+    }
  32
+  }
  33
+}
26  examples/sbt/src/main/scala/com/twitter/example/EchoLoadTest.scala
... ...
@@ -0,0 +1,26 @@
  1
+package com.twitter.example
  2
+
  3
+import org.apache.thrift.protocol.TBinaryProtocol
  4
+
  5
+import com.twitter.parrot.processor.ThriftRecordProcessor
  6
+import com.twitter.parrot.thrift.ParrotJob
  7
+import com.twitter.parrot.server.{ParrotRequest,ParrotService}
  8
+import com.twitter.logging.Logger
  9
+
  10
+import thrift.EchoService
  11
+
  12
+class EchoLoadTest(parrotService: ParrotService[ParrotRequest, Array[Byte]]) extends ThriftRecordProcessor(parrotService) {
  13
+  val client = new EchoService.ServiceToClient(service, new TBinaryProtocol.Factory())
  14
+  val log = Logger.get(getClass)
  15
+
  16
+  def processLines(job: ParrotJob, lines: Seq[String]) {
  17
+    lines map { line =>
  18
+      client.echo(line) respond { rep =>
  19
+        if (rep == "hello") {
  20
+          client.echo("OMIGOD IT'S TALKING TO US") 
  21
+        }
  22
+        log.info("response: " + rep)
  23
+      }
  24
+    }
  25
+  }
  26
+}
67  examples/sbt/src/main/scala/com/twitter/example/EchoServer.scala
... ...
@@ -0,0 +1,67 @@
  1
+package com.twitter.example
  2
+
  3
+import java.net.{InetAddress, InetSocketAddress}
  4
+import java.util.concurrent.atomic.AtomicInteger
  5
+import org.apache.thrift.protocol.TBinaryProtocol
  6
+
  7
+import collection.JavaConversions._
  8
+
  9
+import com.twitter.common.quantity.{Time, Amount}
  10
+import com.twitter.common.zookeeper.{ServerSetImpl, ZooKeeperClient}
  11
+import com.twitter.finagle.builder.{ServerBuilder}
  12
+import com.twitter.finagle.thrift.ThriftServerFramedCodec
  13
+import com.twitter.finagle.zookeeper.ZookeeperServerSetCluster
  14
+import com.twitter.logging.Logger
  15
+import com.twitter.util.Future
  16
+
  17
+import thrift.EchoService
  18
+
  19
+object EchoServer {
  20
+  var port = 8081
  21
+  val log = Logger.get(getClass)
  22
+  val requestCount = new AtomicInteger(0)
  23
+
  24
+  def main(args: Array[String]) {
  25
+    args foreach { arg =>
  26
+      val splits = arg.split("=")
  27
+      if (splits(0) == "thriftPort") {
  28
+        port = splits(1).toInt
  29
+      }
  30
+    }
  31
+    serve(port)
  32
+  }
  33
+
  34
+  def serve(port: Int) {
  35
+    // Implement the Thrift Interface
  36
+    val processor = new EchoService.ServiceIface {
  37
+      def echo(message: String) = {
  38
+        log.info("echoing message: %s", message)
  39
+        requestCount.incrementAndGet
  40
+        Future.value(message)
  41
+      }
  42
+    }
  43
+
  44
+    // Convert the Thrift Processor to a Finagle Service
  45
+    val service = new EchoService.Service(processor, new TBinaryProtocol.Factory())
  46
+
  47
+    val address = new InetSocketAddress(port)
  48
+
  49
+    ServerBuilder()
  50
+      .bindTo(address)
  51
+      .codec(ThriftServerFramedCodec())
  52
+      .name("thriftserver")
  53
+      .build(service)
  54
+
  55
+    // Zookeeper Support
  56
+    val zkHost = "localhost" //put your zookeeper host here
  57
+    val zkPort = 2181
  58
+    val zkCluster: Array[InetSocketAddress] = InetAddress.getAllByName(zkHost).map(new InetSocketAddress(_, zkPort))
  59
+    val zkClient = new ZooKeeperClient(Amount.of(50, Time.MILLISECONDS), asJavaIterable(zkCluster.toIterable))
  60
+    val serverSet = new ServerSetImpl(zkClient, "/twitter/services/parrot-examples")
  61
+    val cluster = new ZookeeperServerSetCluster(serverSet)
  62
+
  63
+    cluster.join(address)
  64
+  }
  65
+
  66
+  def getRequestCount = requestCount.get
  67
+}
6  examples/sbt/src/main/thrift/echo/EchoService.thrift
... ...
@@ -0,0 +1,6 @@
  1
+namespace java com.twitter.example.thrift
  2
+namespace rb ParrotEcho
  3
+
  4
+service EchoService {
  5
+    string echo(1: string message)
  6
+}
48  examples/sbt/src/scripts/echo-server.sh
... ...
@@ -0,0 +1,48 @@
  1
+#!/bin/sh
  2
+#
  3
+# echo mesos startup script.
  4
+
  5
+APP_NAME="echo"
  6
+MAIN_JAR="sbt-example-1.0.jar"
  7
+
  8
+APP_HOME=`pwd`
  9
+PIDFILE=$APP_HOME/$APP_NAME.pid
  10
+LOG_HOME=$APP_HOME
  11
+
  12
+MAIN_CLASS="com.twitter.example.EchoServer"
  13
+HEAP_OPTS="-Xmx128m -Xms128m -XX:NewSize=64m"
  14
+GC_OPTS="-XX:+UseConcMarkSweepGC -verbosegc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseParNewGC -Xloggc:$LOG_HOME/gc.log"
  15
+JAVA_OPTS="-server $GC_OPTS $HEAP_OPTS $PROFILE_OPTS"
  16
+
  17
+# Used to set JAVA_HOME sanely if not already set.
  18
+function find_java() {
  19
+  if [ ! -z $JAVA_HOME ]; then
  20
+    return
  21
+  fi
  22
+  potential=$(ls -r1d /opt/jdk /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home /usr/java/default /usr/java/j* 2>/dev/null)
  23
+  for p in $potential; do
  24
+    if [ -x $p/bin/java ]; then
  25
+      JAVA_HOME=$p
  26
+      break
  27
+    fi
  28
+  done
  29
+}
  30
+
  31
+find_java
  32
+
  33
+case "$1" in
  34
+
  35
+  # start-local is meant for development and runs your server in the foreground.
  36
+  start-local)
  37
+    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp ${APP_HOME}/${MAIN_JAR} ${MAIN_CLASS}
  38
+  ;;
  39
+
  40
+  start)
  41
+    echo "Starting Echo Server v1"
  42
+    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp ${APP_HOME}/${MAIN_JAR} ${MAIN_CLASS} $2 $3 $4 $5 $6 $7 $8 $9 ${10}
  43
+    echo "done."
  44
+  ;;
  45
+
  46
+esac
  47
+
  48
+exit 0
20  pom.xml
@@ -68,47 +68,47 @@
68 68
     <dependency>
69 69
       <groupId>com.twitter</groupId>
70 70
       <artifactId>finagle-http</artifactId>
71  
-      <version>5.3.0</version>
  71
+      <version>5.3.4</version>
72 72
     </dependency>
73 73
     <dependency>
74 74
       <groupId>com.twitter</groupId>
75 75
       <artifactId>finagle-kestrel</artifactId>
76  
-      <version>5.3.0</version>
  76
+      <version>5.3.4</version>
77 77
     </dependency>
78 78
     <dependency>
79 79
       <groupId>com.twitter</groupId>
80 80
       <artifactId>finagle-memcached</artifactId>
81  
-      <version>5.3.0</version>
  81
+      <version>5.3.4</version>
82 82
     </dependency>
83 83
     <dependency>
84 84
       <groupId>com.twitter</groupId>
85 85
       <artifactId>finagle-ostrich4</artifactId>
86  
-      <version>5.3.0</version>
  86
+      <version>5.3.4</version>
87 87
     </dependency>
88 88
     <dependency>
89 89
       <groupId>com.twitter</groupId>
90 90
       <artifactId>finagle-thrift</artifactId>
91  
-      <version>5.3.0</version>
  91
+      <version>5.3.4</version>
92 92
     </dependency>
93 93
     <dependency>
94 94
       <groupId>com.twitter</groupId>
95 95
       <artifactId>ostrich</artifactId>
96  
-      <version>8.2.0</version>
  96
+      <version>8.2.3</version>
97 97
     </dependency>
98 98
     <dependency>
99 99
       <groupId>com.twitter</groupId>
100 100
       <artifactId>util-core</artifactId>
101  
-      <version>5.3.0</version>
  101
+      <version>5.3.6</version>
102 102
     </dependency>
103 103
     <dependency>
104 104
       <groupId>com.twitter</groupId>
105 105
       <artifactId>util-eval</artifactId>
106  
-      <version>5.2.0</version>
  106
+      <version>5.3.6</version>
107 107
     </dependency>
108 108
     <dependency>
109 109
       <groupId>com.twitter</groupId>
110 110
       <artifactId>util-logging</artifactId>
111  
-      <version>5.2.0</version>
  111
+      <version>5.3.6</version>
112 112
     </dependency>
113 113
     <!-- Test -->
114 114
     <dependency>
@@ -230,7 +230,7 @@
230 230
       <plugin>
231 231
         <groupId>com.twitter</groupId>
232 232
         <artifactId>maven-finagle-thrift-plugin</artifactId>
233  
-        <version>0.0.2</version>
  233
+        <version>0.0.9</version> 
234 234
         <executions>
235 235
           <execution>
236 236
             <id>thrift-sources</id>
4  src/main/resources/scripts/parrot-feeder.sh
@@ -21,7 +21,7 @@ DEBUG_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=500
21 21
 JAVA_OPTS="-server $GC_OPTS $HEAP_OPTS $PROFILE_OPTS" #$DEBUG_OPTS"
22 22
 
23 23
 # Used to set JAVA_HOME sanely if not already set.
24  
-function find_java() {
  24
+function find_java {
25 25
   if [ ! -z $JAVA_HOME ]; then
26 26
     return
27 27
   fi
@@ -40,7 +40,7 @@ case "$1" in
40 40
 
41 41
   # start-local is meant for development and runs your server in the foreground.
42 42
   start-local)
43  
-    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp ${APP_HOME}/*jar ${MAIN_CLASS} -f ${APP_HOME}/config/target/local-feeder.scala $2 $3 $4 $5 $6 $7 $8 $9 ${10}
  43
+    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp "${APP_HOME}/*:${APP_HOME}/libs/*" ${MAIN_CLASS} -f ${APP_HOME}/config/target/local-feeder.scala $2 $3 $4 $5 $6 $7 $8 $9 ${10}
44 44
 echo "done."
45 45
   ;;
46 46
 
4  src/main/resources/scripts/parrot-server.sh
@@ -20,7 +20,7 @@ DEBUG_OPTS="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=500
20 20
 JAVA_OPTS="-server $GC_OPTS $HEAP_OPTS $PROFILE_OPTS" # $DEBUG_OPTS"
21 21
 
22 22
 # Used to set JAVA_HOME sanely if not already set.
23  
-function find_java() {
  23
+function find_java {
24 24
   if [ ! -z $JAVA_HOME ]; then
25 25
     return
26 26
   fi
@@ -39,7 +39,7 @@ case "$1" in
39 39
 
40 40
   # start-local is meant for development and runs your server in the foreground.
41 41
   start-local)
42  
-    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp ${APP_HOME}/*jar ${MAIN_CLASS} -f ${APP_HOME}/config/target/local-server.scala
  42
+    ${JAVA_HOME}/bin/java ${JAVA_OPTS} -cp "${APP_HOME}/*:${APP_HOME}/libs/*" ${MAIN_CLASS} -f ${APP_HOME}/config/target/local-server.scala
43 43
 echo "done."
44 44
   ;;
45 45
 
2  src/main/scala/com/twitter/parrot/config/ParrotServerConfig.scala
@@ -57,6 +57,8 @@ trait ParrotServerConfig[Req <: ParrotRequest, Rep] extends Config[RuntimeEnviro
57 57
   var hostConnectionLimit = Integer.MAX_VALUE
58 58
   var hostConnectionMaxIdleTimeInMs = 5000
59 59
   var hostConnectionMaxLifeTimeInMs = Integer.MAX_VALUE
  60
+  var requestTimeoutInMs = Integer.MAX_VALUE
  61
+  var tcpConnectTimeoutInMs = Integer.MAX_VALUE
60 62
   var idleTimeoutInSec = 300
61 63
   var reuseConnections = true
62 64
   var thriftClientId = ""
3  src/main/scala/com/twitter/parrot/launcher/ParrotLauncher.scala
@@ -163,8 +163,7 @@ class ParrotLauncher(config: ParrotLauncherConfig) {
163 163
   private[this] def createRemoteConfigs() {
164 164
     log.debug("Creating configs.")
165 165
 
166  
-    List( ("/templates/template.mesos",         targetDstFolder + "/config.mesos"),
167  
-          ("/templates/template-feeder.scala",  targetDstFolder + "/mesos-feeder.scala"),
  166
+    List( ("/templates/template-feeder.scala",  targetDstFolder + "/mesos-feeder.scala"),
168 167
           ("/templates/template-server.scala",  targetDstFolder + "/mesos-server.scala") ) foreach {
169 168
       case (src, dst) => templatize(src, dst, symbols)
170 169
     }
4  src/main/scala/com/twitter/parrot/processor/SimpleRecordProcessor.scala
@@ -32,13 +32,15 @@ import org.jboss.netty.handler.codec.http.HttpResponse
32 32
 class SimpleRecordProcessor(service: ParrotService[ParrotRequest, HttpResponse],
33 33
                             config: ParrotServerConfig[ParrotRequest, HttpResponse])
34 34
   extends RecordProcessor {
  35
+
35 36
   def processLines(job: ParrotJob, lines: Seq[String]) {
36 37
     lines flatMap { line =>
37 38
       val target = job.victims.get(config.randomizer.nextInt(job.victims.size))
38 39
       UriParser(line) match {
39 40
         case Return(uri) =>
40 41
           if (!uri.path.isEmpty && !line.startsWith("#")) {
41  
-            Some(service(new ParrotRequest(target, None, Nil, uri, line)))
  42
+            val request = new ParrotRequest(target, None, Nil, uri, line)
  43
+            Some(service(request))
42 44
           }
43 45
           else
44 46
             None
7  src/main/scala/com/twitter/parrot/server/FinagleTransport.scala
@@ -25,6 +25,7 @@ import com.twitter.parrot.thrift.TargetHost
25 25
 import com.twitter.util.{Promise, Duration, Future}
26 26
 import java.nio.ByteOrder.BIG_ENDIAN
27 27
 import java.util.concurrent.TimeUnit
  28
+import java.util.logging.{Logger => JLogger}
28 29
 import org.jboss.netty.buffer.ChannelBuffers
29 30
 import org.jboss.netty.handler.codec.http.HttpHeaders.Names.CONTENT_LENGTH
30 31
 import org.jboss.netty.handler.codec.http._
@@ -43,7 +44,10 @@ class FinagleTransport(config: ParrotServerConfig[ParrotRequest, HttpResponse])
43 44
     .hostConnectionLimit(config.hostConnectionLimit)
44 45
     .hostConnectionMaxIdleTime(Duration(config.hostConnectionMaxIdleTimeInMs, TimeUnit.MILLISECONDS))
45 46
     .hostConnectionMaxLifeTime(Duration(config.hostConnectionMaxLifeTimeInMs, TimeUnit.MILLISECONDS))
  47
+    .requestTimeout(Duration(config.requestTimeoutInMs, TimeUnit.MILLISECONDS))
  48
+    .tcpConnectTimeout(Duration(config.tcpConnectTimeoutInMs, TimeUnit.MILLISECONDS))
46 49
     .keepAlive(true)
  50
+//    .logger(JLogger.getLogger("com.twitter.finagle")) // enable for extreme debugging
47 51
     .reportTo(new OstrichStatsReceiver)
48 52
 
49 53
   var allRequests = 0
@@ -78,8 +82,9 @@ class FinagleTransport(config: ParrotServerConfig[ParrotRequest, HttpResponse])
78 82
 """
79 83
 ===================== HttpRequest ======================
80 84
 %s
  85
+%s
81 86
 ========================================================"""
82  
-      .format(httpRequest.toString)
  87
+      .format(request.target, httpRequest.toString)
83 88
     )
84 89
 
85 90
     client flatMap { service: Service[HttpRequest, HttpResponse] =>
4  src/test/scala/com/twitter/parrot/integration/EndToEndSpec.scala
@@ -63,8 +63,8 @@ class EndToEndSpec extends SpecificationWithJUnit {
63 63
         case e: Exception => fail(e.getMessage)
64 64
       }
65 65
       finally {
  66
+        // Feeder will shutdown Server
66 67
         feeder.shutdown()
67  
-        server.shutdown()
68 68
       }
69 69
     }
70 70
 
@@ -102,8 +102,8 @@ class EndToEndSpec extends SpecificationWithJUnit {
102 102
         case e: Exception => fail(e.getMessage)
103 103
       }
104 104
       finally {
  105
+        // Feeder will shutdown Server
105 106
         feeder.shutdown()
106  
-        server.shutdown()
107 107
       }
108 108
     }
109 109
   }
3  src/test/scala/com/twitter/parrot/server/ParrotUdpTransportSpec.scala
@@ -120,8 +120,7 @@ class ParrotUdpTransportSpec extends SpecificationWithJUnit {
120 120
         { transport.asInstanceOf[ParrotUdpTransport[ParrotRequest, String]].allRequests.get } must
121 121
           eventually(be(requestStrings.size))
122 122
       } finally {
123  
-        feeder.shutdown()
124  
-        server.shutdown()
  123
+        feeder.shutdown() // this will implicitly shut down the server as well
125 124
       }
126 125
     }
127 126
   }

0 notes on commit 4294510

Please sign in to comment.
Something went wrong with that request. Please try again.