Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Fetching contributors…

Cannot retrieve contributors at this time

file 270 lines (203 sloc) 8.862 kb
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270
Flume Developer Notes
=====================
Jonathan Hsieh <jon@cloudera.com>
6/29/10


// This is in asciidoc markup

== Introduction

This is meant to be a a guide for issues that occur when building,
debugging and setting up flume as developer.

== High level directory and file structure.

----
./bin/ flume scripts
./conf/ flume configuration files
./lib/ libraries used by flume
./libbuild/ libraries used by flume for building
./libtest/ libraries used by flume for testing
./src/ahocorasick a library for multple string search
./src/antlr flume config language ANTLR grammer files
./src/gen-java autogenrated java source files (from antlr/thrift)
./src/java flume java source code
./src/javaperf flume performance tests (out of date)
./src/javatest flume unit tests
./src/javatest-torture flume reliability tests (out of date)
./src/thrift flume thrift idl files (for rpc)
./src/webapps flume webapp jsp source code
----

Files created by build:
----
./build
----

== Files in `.gitignore`

The exclusions in .gitignore are either autogenerated or build/eclipse
specific

== eclipse project setup.

Run "ant eclipse", then create a new java project in eclipse with the
current directory as the base project directory.

----
./.eclipse default working directory for eclipse
----

Note: eclipse class files are not used by bin/flume, you must either
a) compile via ant for bin/flume to pick up your modified code, or b)
specify eclipse on the flume classpath, e.g.:

FLUME_CLASSPATH=./.eclipse/classes-main:./.eclipse/classes-test bin/flume

== Building thrift

This will create a repository in ./apache-thrift
----
git clone git://git.apache.org/thrift.git
cd thrift
git fetch
git checkout -b thrift-0.2.0 origin/tags/thrift-0.2.0
bootstrap.sh
configure
make
sudo make install
----

Problem: During bootstrap.sh
----
configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
----
Solution: install libtool
----
sudo apt-get install libtool
----

== Generated source

These files should not be checked in unless their source files are modified.

For ANTLR
----
src/gen-java/com/cloudera/flume/conf/FlumeDeployLexer.java
src/gen-java/com/cloudera/flume/conf/FlumeDeployParser.java
----

== Running test with ant.

----
ant test
----

=== Running only specified test cases from ant

----
ant test -Dtestcase=<TestFile>
----

where <TestFile> is a class name without .java or .class or path.

(How do is specify just a function?)

== origin/master invariants

Always should build.

Ideally tests all pass

== Push invariants

We should tag pushes with JIRA nubmers.

== Flume's web application


The default setup for flume is to run its servlets from precompiled
jsps. The default configuration points jetty (a jsp server) to
information found in the ./webapps directory. We assume that most
developers and users will be in core, so at the git project root dir,
./webapps is a symlink that points to the build/webapps directory.
This is where static files and auto-generated files that are generated when
flume is compiled. Using this symlink makes the flume webapp use
precompiled jsp pages.

One can debug the jsp pages or have them autogenerate at runtime
(useful for development) by changing this symlink to point to
src/webapps. This directory has subapps and jsp source
code. Debugging is easier when it is more dynamic and our servlet
container Jetty can dyamically comile jsp pages to hasten the
debugging process.

Here are some tips for getting the web apps for Flume Master or Flume
Node running from inside eclipse.

* Make sure `tools.jar` is in your java classpath. If JAVA_HOME is
  set to a JDK JVM path (as opposed to a JRE) this jar should be
  included. This jar includes the java compiler which is required to
  enable the compilation of jsp's so they can be served on the fly.
* Ant is used to compile jsps. Make sure some version of
  ant-launcher.jar and ant-1.x.x.jar is in your build path. (if you
  are in eclipse for example). These files live in ./libbuild
* The default is to point to a the web app at a precompiled version of
  of the servlets. There is a hook in flume-site.xml to point the
  jetty at a directory full of jsps. It assumes that hte flume
  directory is the base for relative paths or can use a fully
  qualified path

Environment variables can be set in the +bin/flume-env.sh+ script.
----
# bin/flume-env.sh for Ubuntu installs
export JAVA_HOME=/usr/lib/jvm/java-6-sun
----

Alternately, instead of using symlnks, one can set the following
property in the system's flume-site.conf file, like below.

----
  <property>
    <name>flume.webapps.path</name>
    <value>src/webapps</value>
    <description>This is the path use to the web apps that display
    flume node/master data

      Use src webapps for development.
    </description>
  </property>
----

=== Problems when compiling JSPs

BUILD FAILED
/home/jon/flume/build.xml:471: java.lang.ExceptionInInitializerError
        at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.java:197)
        at org.apache.jasper.JspC.processFile(JspC.java:772)
        at org.apache.jasper.JspC.execute(JspC.java:908)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
...
 
Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@31554233 for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))
        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
        at org.apache.jasper.compiler.Compiler.<clinit>(Compiler.java:55)
        ... 25 more

  
Make sure that log4j-xxx.jar is in your CLASSPATH.

== Avro.

Using a post 1.2.0 version that has reflection that supports
Strings, byte[]'s and extracting fields defined in super classes.

requires (in repo):
----
lib/avro-1.2.0-dev.jar # (trunk hash 8911c848 ; more avro 1.3 than 1.2)
lib/paranamer-1.5.jar # extra reflection stuff
lib/jackson-core-asl-1.1.1.jar # json parser
lib/jackson-mapper-asl-1.1.1.jar # json parser
----

== Developer mode.

This is an option in the bin/flume for using eclipse built class files
instead of ant built class files.

in bash one would set FLUME_DEVMODE to true:

----
$ declare -x FLUME_DEVMODE=true
----

It is assumed that the eclipse build path is build_eclipse/.

== Building Windows packages

Building a full windows package and installer executable requires a
few steps. A cygwin envrionment is currently assumed.

1) Build flume jars ('ant tar').

2) Update the installer script to add versioning information ('ant
winstall-filter'). This generates ./flume.nsi.

3) Run nsis compiler on the generated flume.nsi. We've used v2.46 http://nsis.sourceforge.net/Download.

The current cut does not deal with differences between 32-bit vs
64-bit versions, proper error handling situations, or checks to see if
not run as administrator.

== Building documentation

Documentation for Flume is written in asciidoc. It relys on several
libraries to generate images.

* asciidoc v8.5.2
* graphviz (dot) v2.26.3

Documents can be built by running 'ant docs'

== License

All source files must include the following header:

/**
 * Licensed to Cloudera, Inc. under one
 * or more contributor license agreements. See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. Cloudera, Inc. licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
Something went wrong with that request. Please try again.