Initial commit

From: Henry Robinson <henry@cloudera.com>

git-svn-id: https://svn.apache.org/repos/asf/incubator/flume/trunk@1155581 13f79535-47bb-0310-9956-ffa450edef68

DEVNOTES

@@ -0,0 +1,233 @@
+Flume Developer Notes
+=====================
+Jonathan Hsieh <jon@cloudera.com>
+6/29/10
+
+
+// This is in asciidoc markup
+
+== Introduction 
+
+This is meant to be a a guide for issues that occur when building,
+debugging and setting up flume as developer.
+
+== High level directory and file structure.
+
+----
+./bin/                  flume scripts
+./conf/                 flume configuration files
+./eclipse/              eclipse project environment files
+./lib/                  libraries used by flume
+./libbuild/             libraries used by flume for building
+./libtest/              libraries used by flume for testing
+./src/ahocorasick       a library for multple string search
+./src/antlr             flume config language ANTLR grammer files
+./src/gen-java          autogenrated java source files (from antlr/thrift)
+./src/java              flume java source code
+./src/javaperf          flume performance tests (out of date)
+./src/javatest          flume unit tests
+./src/javatest-torture  flume reliability tests (out of date)
+./src/thrift		flume thrift idl files (for rpc)
+./src/webapps           flume webapp jsp source code
+----
+
+Files created by build:
+----
+./build
+----
+
+Default directory for eclipse build.
+----
+./build_eclipse
+----
+
+== Files in `.gitignore` 
+
+These files are autogenerated or build/eclipse specific
+----
+src/java/com/cloudera/flume/package-info.java
+build/
+build_eclipse/
+*#
+.classpath
+.project
+.settings
+----
+
+== eclipse project setup.
+
+These files are kept out of there normal place because they tend to be
+modified by users and are eclipse specific.  The files can be copied
+into place to get a head start on a project.  The actual dotfiles are
+in the the .gitignore file.
+
+---- 
+cp eclipse/project .project 
+cp eclipse/classpath .classpath 
+----
+
+
+== Building thrift
+
+This will create a repository in ./apache-thrift
+----
+git clone git://git.apache.org/thrift.git
+cd thrift
+git fetch
+git checkout -b thrift-0.2.0 origin/tags/thrift-0.2.0
+bootstrap.sh
+configure
+make
+sudo make install
+----
+
+Problem: During bootstrap.sh
+----
+configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
+      If this token and others are legitimate, please use m4_pattern_allow.
+      See the Autoconf documentation.
+----
+Solution: install libtool
+----
+sudo apt-get install libtool
+----
+
+== Generated source
+
+These files should not be checked in unless their source files are modified.  
+
+For ANTLR
+----
+src/gen-java/com/cloudera/flume/conf/FlumeDeployLexer.java
+src/gen-java/com/cloudera/flume/conf/FlumeDeployParser.java
+----
+
+== Running test with ant.
+
+----
+ant test
+----
+
+=== Running only specified test cases from ant
+
+----
+ant test -Dtestcase=<TestFile>
+----
+
+where <TestFile> is  a class name without .java or .class or path.
+
+(How do is specify just a function?)
+
+== origin/master invariants
+
+Always should build. 
+
+Ideally tests all pass
+
+== push invarients
+
+We should tag pushes with JIRA nubmers.
+
+
+== Debugging JSP web applications in jetty inside eclipse.
+
+The default setup for flume is to run its servlets from precompiled
+jsps.  Debugging is easier when it is more dynamic and our servlet
+container Jetty can dyamically comile jsp pages to hasten the
+debugging process.  This is useful when using running the Flume Master
+or FlumeNode from inside eclipse.
+
+* Make sure `tools.jar` is in your java classpath.  This allows the compiler to be present.
+* Ant is used to compile jsps.  Make sure some version of ant-launcher.jar and ant-1.x.x.jar is in your build path. (if you are in eclipse for example)
+* The default is to point to a the web app at a precompiled version of of the servlets.  There is a hook in flume-site.xml to point the jetty at a directory full of jsps.  It assumes that hte flume directory is the base for relative paths or can use a fully qualified path 
+
+----
+  <property>
+    <name>flume.webapps.path</name>
+    <value>src/webapps</value>
+    <description>This is the path use to the web apps that display
+    flume node/master data
+
+      Use src webapps for development.
+    </description>
+  </property>
+----
+
+
+== Compiling JSPs
+
+BUILD FAILED
+/home/jon/flume/build.xml:471: java.lang.ExceptionInInitializerError
+        at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.java:197)
+        at org.apache.jasper.JspC.processFile(JspC.java:772)
+        at org.apache.jasper.JspC.execute(JspC.java:908)
+        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+...
+
+Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@31554233 for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))
+        at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
+        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
+        at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
+        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
+        at org.apache.jasper.compiler.Compiler.<clinit>(Compiler.java:55)
+        ... 25 more
+
+
+Make sure that log4j-xxx.jar is in your CLASSPATH.
+
+== Avro.  
+
+Using a post 1.2.0 version that has reflection that supports
+Strings, byte[]'s and extracting fields defined in super classes.
+
+requires (in repo):
+----
+lib/avro-1.2.0-dev.jar      # (trunk hash 8911c848 ; more avro 1.3 than 1.2)
+lib/paranamer-1.5.jar       # extra reflection stuff
+lib/jackson-core-asl-1.1.1.jar    # json parser    
+lib/jackson-mapper-asl-1.1.1.jar  # json parser
+----
+
+== Hudson
+
+Hudson should pull the head of the repository and build.  (Chad).
+
+Add 'git clean -xdf' to the build execute shell config panel to force
+hudson to rebuild.  
+
+archive artifacts: build/test/TEST*.txt to get outputs of unit test
+
+== Developer mode.
+
+This is an option in the bin/flume for using eclipse built class files
+instead of ant built class files.  
+
+in bash one would set FLUME_DEVMODE to true:
+
+----
+$ declare -x FLUME_DEVMODE=true
+----
+
+It is assumed that the eclipse build path is build_eclipse/.
+
+== License
+
+----
+Licensed to Cloudera, Inc. under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  Cloudera, Inc. licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+----