Skip to content
This repository has been archived by the owner. It is now read-only.
Permalink
Browse files
CHUKWA-721. Updated Chukwa document to reflect changes in Chukwa 0.6.…
… (Eric Yang)

git-svn-id: https://svn.apache.org/repos/asf/chukwa/trunk@1612600 13f79535-47bb-0310-9956-ffa450edef68
  • Loading branch information
macroadster committed Jul 22, 2014
1 parent e4b13e4 commit 865596312a26726fe2c74573a17bff36bfa20be6
Showing 13 changed files with 176 additions and 340 deletions.
@@ -34,6 +34,8 @@ Release 0.6 - Unreleased

IMPROVEMENTS

CHUKWA-721. Updated Chukwa document to reflect changes in Chukwa 0.6. (Eric Yang)

CHUKWA-718. Updated Chukwa Agent REST API document and generation method. (Eric Yang)

CHUKWA-710. Set TCP socket reuse option for server sockets. (Shreyas Subramanya)
@@ -1067,7 +1067,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.0</version>
<version>3.3</version>
<dependencies>
<dependency><!-- add support for ssh/scp -->
<groupId>org.apache.maven.wagon</groupId>
@@ -1239,7 +1239,7 @@
<artifactId>maven-jxr-plugin</artifactId>
<version>2.3</version>
</plugin>
<plugin>
<!-- <plugin>
<artifactId>maven-pmd-plugin</artifactId>
<version>2.6</version>
<reportSets>
@@ -1262,7 +1262,7 @@
<threshold>Normal</threshold>
<effort>Max</effort>
</configuration>
</plugin>
</plugin>-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-project-info-reports-plugin</artifactId>
@@ -24,7 +24,7 @@ Pre-requisites

Chukwa should work on any POSIX platform, but GNU/Linux is the only production platform that has been tested extensively. Chukwa has also been used successfully on Mac OS X, which several members of the Chukwa team use for development.

The only absolute software requirements are Java 1.6 or better and Hadoop 0.20.205.0+. HICC, the Chukwa visualization interface, requires HBase 0.90.4.
The only absolute software requirements are Java 1.6 or better, ZooKeeper 3.4.x, HBase 0.96.x and Hadoop 1.x.

The Chukwa cluster management scripts rely on ssh; these scripts, however, are not required if you have some alternate mechanism for starting and stopping daemons.

@@ -35,9 +35,7 @@ Installing Chukwa

* A Hadoop and HBase cluster on which Chukwa will process data (referred to as the Chukwa cluster).

* A collector process, that writes collected data to HBase.

* One or more agent processes, that send monitoring data to the collector. The nodes with active agent processes are referred to as the monitored source nodes.
* One or more agent processes, that send monitoring data to HBase. The nodes with active agent processes are referred to as the monitored source nodes.

* Data analytics script, summarize Hadoop Cluster Health.

@@ -53,9 +51,9 @@ First Steps

* Un-tar the release, via tar xzf.

* Make sure a copy of Chukwa is available on each node being monitored, and on each node that will run a collector.
* Make sure a copy of Chukwa is available on each node being monitored.

* We refer to the directory containing Chukwa as CHUKWA_HOME. It may be helpful to set CHUKWA_HOME explicitly in your environment, but Chukwa does not require that you do so.
* We refer to the directory containing Chukwa as CHUKWA_HOME. It may be useful to set CHUKWA_HOME explicitly in your environment for ease of use.

Setting Up Chukwa Cluster

@@ -82,19 +80,9 @@ bin/hbase shell < CHUKWA_HOME/etc/chukwa/hbase.schema

This procedure initializes the default Chukwa HBase schema.

* Configuring And Starting Chukwa Collector

[[1]] Edit CHUKWA_HOME/etc/chukwa/chukwa-env.sh. Make sure that JAVA_HOME, HADOOP_HOME, HADOOP_CONF_DIR, HBASE_HOME, and HBASE_CONF_DIR are set correctly.

[[2]] In CHUKWA_HOME, run:

---
bin/chukwa collector
---

* Configuring And Starting Chukwa Agent

[[1]] Add collector hostnames to CHUKWA_HOME/etc/chukwa/collectors. One host per line.
[[1]] Edit CHUKWA_HOME/etc/chukwa/chukwa-env.sh. Make sure that JAVA_HOME, HADOOP_HOME, HADOOP_CONF_DIR, HBASE_HOME, and HBASE_CONF_DIR are set correctly.

[[2]] In CHUKWA_HOME, run:

@@ -146,4 +134,4 @@ http://<server>:4080/hicc/

[[2]] The default user name and password is "admin" without quotes.

[[3]] Metrics data collected by Chukwa collector will be browsable through Graph Explorer widget.
[[3]] Metrics data collected by Chukwa Agent will be browsable through Graph Explorer widget.
@@ -14,7 +14,7 @@
~~ limitations under the License.
~~

Overview
Agent Configuration Guide

In a normal Chukwa installation, an <Agent> process runs on every
machine being monitored. This process is responsible for all the data collection
@@ -38,27 +38,27 @@ Agent Control
use to inspect and control it. By default, Agents listen for incoming commands
on port 9093. Commands are case-insensitive

*--------------------*--------------------------------------*--------------:
| Command | Purpose | Options |
*--------------------*--------------------------------------*--------------:
| <add> | Start an adaptor. | See below |
*--------------------*--------------------------------------*--------------:
| <close> | Close socket connection to agent. | None |
*--------------------*--------------------------------------*--------------:
| <help> | Display a list of available commands | None |
*--------------------*--------------------------------------*--------------:
| <list> | List currently running adaptors | None |
*--------------------*--------------------------------------*--------------:
| <reloadcollectors> | Re-read list of collectors | None |
*--------------------*--------------------------------------*--------------:
| <stop> | Stop adaptor, abruptly | Adaptor name |
*--------------------*--------------------------------------*--------------:
| <stopall> | Stop all adaptors, abruptly | Adaptor name |
*--------------------*--------------------------------------*--------------:
| <shutdown> | Stop adaptor, gracefully | Adaptor name |
*--------------------*--------------------------------------*--------------:
| <stopagent> | Stop agent process | None |
*--------------------*--------------------------------------*--------------:
*--------------------*-----------------------------------------*--------------:
| Command | Purpose | Options |
*--------------------*-----------------------------------------*--------------:
| <add> | Start an adaptor. | See below |
*--------------------*-----------------------------------------*--------------:
| <close> | Close socket connection to agent. | None |
*--------------------*-----------------------------------------*--------------:
| <help> | Display a list of available commands | None |
*--------------------*-----------------------------------------*--------------:
| <list> | List currently running adaptors | None |
*--------------------*-----------------------------------------*--------------:
| <reloadcollectors> | Re-read list of collectors (deprecated) | None |
*--------------------*-----------------------------------------*--------------:
| <stop> | Stop adaptor, abruptly | Adaptor name |
*--------------------*-----------------------------------------*--------------:
| <stopall> | Stop all adaptors, abruptly | Adaptor name |
*--------------------*-----------------------------------------*--------------:
| <shutdown> | Stop adaptor, gracefully | Adaptor name |
*--------------------*-----------------------------------------*--------------:
| <stopagent> | Stop agent process | None |
*--------------------*-----------------------------------------*--------------:


The add command is by far the most complex; it takes several mandatory and
@@ -87,10 +87,8 @@ Command-line options
Normally, agents are configured via the file <conf/chukwa-agent-conf.xml.>
However, there are a few command-line options that are sometimes useful in
troubleshooting. If you specify "local" as an option, then the agent will print
chunks to standard out, rather than to a collector. If you specify a URI, then
that will be used as collector, overriding the collectors specified in
<conf/collectors>. These options are intended for testing and debugging,
not for production use.
chunks to standard out, rather than to pipeline writers.
This option is intended for testing and debugging, not for production use.

---
bin/chukwa agent local
@@ -14,7 +14,7 @@
~~ limitations under the License.
~~

Chukwa Storage Layout
HDFS Storage Layout

Overview

@@ -41,11 +41,11 @@ Raw Log Collection and Aggregation Workflow

What data is stored where is best described by stepping through the Chukwa workflow.

[[1]] Collectors write chunks to <logs/*.chukwa> files until a 64MB chunk size is reached or a given time interval has passed.
[[1]] Agents write chunks to <logs/*.chukwa> files until a 64MB chunk size is reached or a given time interval has passed.

* <logs/*.chukwa>

[[2]] Collectors close chunks and rename them to <*.done>
[[2]] Agents close chunks and rename them to <*.done>

* from <logs/*.chukwa>

@@ -26,10 +26,9 @@ stages. This will facilitate future innovation without breaking existing code.

Chukwa has five primary components:

* <<Agents>> that run on each machine and emit data.
* <<Adaptors>> that collect data from various data source.

* <<Collectors>> that receive data from the agent and write
it to stable storage.
* <<Agents>> that run on each machine and emit data.

* <<ETL Processes>> for parsing and archiving the data.

@@ -42,7 +41,7 @@ Chukwa has five primary components:
dwell times at each stage. A more detailed figure is available at the end
of this document.

[./images/datapipeline.png] A picture of the chukwa data pipeline
[./images/chukwa_architecture.png] Architecture

Agents and Adaptors

@@ -102,31 +101,9 @@ it's usually save to specify 0 as an ID, but it's sometimes useful to specify
something else. For instance, it lets you do things like only tail the second
half of a file.

Collectors

Rather than have each adaptor write directly to HDFS, data is sent across
the network to a <collector> process, that does the HDFS writes.
Each collector receives data from up to several hundred hosts, and writes all
this data to a single <sink file>, which is a Hadoop sequence file of
serialized Chunks. Periodically, collectors close their sink files, rename
them to mark them available for processing, and resume writing a new file.
Data is sent to collectors over HTTP.

Collectors thus drastically reduce the number of HDFS files generated by Chukwa,
from one per machine or adaptor per unit time, to a handful per cluster.
The decision to put collectors between data sources and the data store has
other benefits. Collectors hide the details of the HDFS file system in use,
such as its Hadoop version, from the adaptors. This is a significant aid to
configuration. It is especially helpful when using Chukwa to monitor a
development cluster running a different version of Hadoop or when using
Chukwa to monitor a non-Hadoop cluster.

For more information on configuring collectors, see the
{{{./collector.html}Collector documentation}}.

ETL Processes

Collectors can write data directly to HBase or sequence files.
Chukwa Agents can write data directly to HBase or sequence files.
This is convenient for rapidly getting data committed to stable storage.

HBase provides index by primary key, and manage data compaction. It is
@@ -146,10 +123,10 @@ precisely how they group the data.)

Demux, in contrast, take Chunks as input and parse them to produce
ChukwaRecords, which are sets of key-value pairs. Demux can run as a
MapReduce job or as part of Chukwa Collector.
MapReduce job or as part of HBaseWriter.

For details on controlling this part of the pipeline, see the
{{{./admin.html}Administration guide}}. For details about the file
{{{./pipeline.html}Pipeline guide}}. For details about the file
formats, and how to use the collected data, see the {{{./programming.html}
Programming guide}}.

@@ -169,6 +146,20 @@ which in turn is populated by collector or data analytic scripts
that runs on the collected data, after Demux. The
{{{./admin.html}Administration guide}} has details on setting up HICC.

And now, the architecture picture of Chukwa:

[./images/chukwa_architecture.png] Architecture
Collectors (Deprecated)

The origin design of collector is to reduce the number of TCP connections
for collecting data from various source, provide high availability and wire
compatibility across versions. Data transfer reliability has been improved
in HDFS client and HBase client. The original problem that Chukwa Collector
tried to solve is no longer high priority in Chukwa data collection framework
because both HDFS and HBase are in better position in solving data transport
and replication problem. Hence, datanode and HBase Region servers are
replacement for Chukwa collectors.

Chukwa has adopted to use HBase to ensure data arrival in milli-seconds and
also make data available to down steam application at the same time. This
will enable monitoring application to have near realtime view as soon as
data are arriving in the system. The file rolling, archiving are replaced
by HBase Region Server minor and major compactions.

@@ -14,7 +14,7 @@
~~ limitations under the License.
~~

Overview
HICC Operation Manual

HICC stands for Hadoop Infrastructure Care Center. It is the central dashboard
for visualize and monitoring of metrics collected by Chukwa.
@@ -87,6 +87,13 @@ make additional choices or provide any necessary information.
* If an option is dimmed, it is not available. For example, you can not edit
name of the dashboard, if you are not the owner of the dashboard.

* Dashboard basics

Each dashboard can have multiple tabs and each tab is divided into row and columns.
You can add more widgets by selecting Options > choose a widget > click Add.
To remove a widgets by clicking on Close button on the top right hand corner of
the widget.

* Tab basics

Tab provides a way to organize related information together. As you create widgets,
@@ -115,13 +122,6 @@ information organized.

[[2]] Enter a new name for the tab and press Return.

* Dashboard basics

Each dashboard can have multiple tabs and each tab is divided into row and columns.
You can add more widgets by selecting Options > choose a widget > click Add.
To remove a widgets by clicking on Close button on the top right hand corner of
the widget.

User accounts

You should set up an account for each person who uses HICC on a regular basis.
@@ -16,31 +16,40 @@
Overview

Log processing was one of the original purposes of MapReduce. Unfortunately,
using Hadoop for MapReduce processing of logs is somewhat troublesome.
Logs are generated incrementally across many machines, but Hadoop MapReduce
using Hadoop MapReduce to monitor Hadoop can be inefficient. Batch
processing nature of Hadoop MapReduce prevents the system to provide real time
status of the cluster.

We started this journey at beginning of 2008, and a lot of Hadoop components
have been built to improve overall reliability of the system and
improve realtimeness of monitoring. We have adopted HBase to facilitate lower
latency of random reads and using in memory updates and write ahead logs to
improve the reliability for root cause analysis.

Logs are generated incrementally across many machines, but Hadoop MapReduce
works best on a small number of large files. Merging the reduced output
of multiple runs may require additional mapreduce jobs. This creates some
overhead for data management on Hadoop.

Chukwa is a Hadoop subproject devoted to bridging that gap between logs
processing and Hadoop ecosystem. Chukwa is a scalable distributed monitoring
and analysis system, particularly logs from Hadoop and other large systems.
and analysis system, particularly logs from Hadoop and other distributed systems.

The Chukwa Documentation provides the information you need to get
started using Chukwa. You should start with the {{{./design.html}
Architecture and Design document}}.
started using Chukwa. {{{./design.html} Architecture and Design document}}
provides high level view of Chukwa design.

If you're trying to set up a Chukwa cluster from scratch, you should
read the {{{./admin.html}Chukwa Administration Guide}} which
shows you how to setup and deploy Chukwa.
If you're trying to set up a Chukwa cluster from scratch,
{{{./user.html} User Guide}} describes the setup and deploy procedure.

If you want to configure the Chukwa agent process, to control what's
collected, you should read the {{{./agent.html}Agent Guide}}. There's
also a {{{./collector.html}Collector Guide}} describing that part of
the pipeline.
collected, you should read the {{{./agent.html} Agent Guide}}. There is
also a {{{./pipeline.html} Pipeline Guide}} describing configuration
parameters for ETL processes for the data pipeline.

And if you want to use collected data, read the
{{{./programming.html}User and Programming Guide}}
And if you want to develop Chukwa to monitor other data source,
{{{./programming.html} Programming Guide}} maybe handy to learn
about Chukwa programming API.

If you have more questions, you can ask on the
{{{mailto:chukwa-user@incubator.apache.org}Chukwa mailing lists}}
{{{mailto:user@chukwa.apache.org}Chukwa mailing lists}}

0 comments on commit 8655963

Please sign in to comment.