Skip to content
This repository has been archived by the owner on Apr 5, 2022. It is now read-only.

Commit

Permalink
XD-1407 Add sample README and helper script to test the reactor-tcp s…
Browse files Browse the repository at this point in the history
…ource module.
  • Loading branch information
jbrisbin authored and garyrussell committed Mar 19, 2014
1 parent 85fffc7 commit 171516b
Show file tree
Hide file tree
Showing 3 changed files with 61 additions and 0 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ dump.rdb
tomcat.*
activemq-data
spring-shell.log
.testdata
31 changes: 31 additions & 0 deletions reactor-tcp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Reactor TCP Spring XD sample

This is a sample application demonstrating the use of [Reactor](https://github.com/reactor/reactor) in *Spring XD*.

## Requirements

To run this sample you'll need:

* Spring XD ([Instructions](https://github.com/SpringSource/spring-xd/wiki/Getting-Started))
* Ruby
* `sendfile` ruby Gem (`gem install sendfile`)

## Create a Stream

You'll need to create a `reactor-tcp` source in *Spring XD* using the following example as a template:

stream create --name reactortcp --definition "reactor-tcp --port=3000 | throughput-sampler"

You can change the port to which the source is bound by adjusting the `--port` parameter. The default is 3000 and that's what the Ruby script included with this sample will try to connect to, but you can easily change that, of course.

## Throughput Sampling

The `throughput-sampler` sink is a simple NOOP sink that looks for a "start" message which, by default is the String `START`. It will count the messages it receives after that point. When it receives an "end" message, which defaults to the not-surprising String value `END`, it will calculate the throughput for the messages it has seen to that point and log that value to the XD log. It's a simple way to measure overall throughput at a fairly coarse level.

## Send the Data

We've found in microbenchmarking network servers that it's very difficult to create a simple client that streams test data fast enough to not interfere with the benchmark. Even using a tool like `netcat`, which you would assume would be about the fastest way to send data to a server, is surprisingly inefficient. These inefficiencies add up when sending many thousands of messages.

The fastest way we've found to send a lot of data to a server very quickly and efficiently (in the hopes you can achieve saturation) is to use the *NIX kernel function "sendfile". How this is done varies based on whether you're using Linux, UNIX, FreeBSD, or Mac OS X. But this test uses the Ruby GEM `sendfile` which has a native extension in it to call this kernel-level function and stream data directly from disk to the network stack, bypassing user space entirely. To do this, it first generates a file of test data that starts with a `START`, contains many thousands of `Hello World!` messages, and is truncated by an `END`.

Each time your run the `senddata.rb` script, it will generate a file of test data and stream that to the TCP source you created in XD with the above command. The throughput for that run will be logged to the XD log file if you're using the included `throughput-sampler` sink.
29 changes: 29 additions & 0 deletions reactor-tcp/senddata.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env ruby

require 'rubygems'
require 'socket'
require 'sendfile'

PORT = 3000
MSGCNT = 100000

puts "Create an XD stream using the following command:\n\n\tstream create --name reactor --definition \"reactor-tcp --port=#{PORT} | throughput-sampler\""

# generate test data
File.open('.testdata', 'w') do |f|
f.puts 'START'
(1..MSGCNT).each do |i|
f.puts 'Hello World!'
end
f.puts 'END'
end

puts "\nHit [ENTER] when ready to send the data..."
gets

# use sendfile to efficient send the data
outs = TCPSocket.new 'localhost', PORT
File.open('.testdata', 'r') do |f|
outs.sendfile f
end
outs.close

0 comments on commit 171516b

Please sign in to comment.