Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Pull request Compare This branch is even with cloudera:master.

Fetching latest commit…

Cannot retrieve the latest commit at this time

..
Failed to load latest commit information.
README
count-forever
count-forever-cleanup
count-forever-verify
gen-config.py
rpcscaling
test-simple-be
test-simple-be-batch
test-simple-dfo
test-simple-dfo-batch
test-simple-e2e
test-simple-e2e-batch

README

# Licensed to Cloudera, Inc. under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  Cloudera, Inc. licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Test-endtoend scripts.

These scripts are used for testing various end-to-end properties of Flume.

== count-forever tests

The goal of these scripts is to generate data, shove it through an
agent, a collector and then to confirm that the output delivery and
issues with their delivery. 

count-forever -> agent -> collector -> count-forever-verify

count-forever          Simple script that prints lines with count an tag
count-forever-cleanup  Script that pulls out body from avrojson records

count-forever-verify   Script that pulls out body from avrojson records
		       and the n only reports when there are count
		       discontinuities or duplicates.

== Example:

=== Local testing:

In different terminals:
$ flume master
$ flume node -n agent
$ flume node -n collector

In flume shell 
exec config agent 'exec("test-endtoend/count-forever")' 'agentE2EChain("localhost")'
exec config collector 'collectorSource' 'collectorSink("file:///tmp/collector","data")'

In another terminal, you can run the count-forever-verify script to
see different discontinuities or dupes.

# One shot:
$ cat /tmp/collector/* | count-forever-verify

# repeatedly:
$ watch 'cat /tmp/collector/* | count-forever-verify'

The count-forever-clean script can be used to output intermediate
data.

One can kill the collector and then restart the collector after a
while.  You will likely see a discontinuity, and then when it
recovers, see duplicatate entries.
Something went wrong with that request. Please try again.