Permalink
Browse files

Initial commit

  • Loading branch information...
henryr committed Jun 18, 2010
1 parent d34d182 commit 0efd816e3a82325a7a84bed1f17ad1691fe35202
Showing 545 changed files with 96,527 additions and 0 deletions.
View
233 DEVNOTES
@@ -0,0 +1,233 @@
+Flume Developer Notes
+=====================
+Jonathan Hsieh <jon@cloudera.com>
+6/29/10
+
+
+// This is in asciidoc markup
+
+== Introduction
+
+This is meant to be a a guide for issues that occur when building,
+debugging and setting up flume as developer.
+
+== High level directory and file structure.
+
+----
+./bin/ flume scripts
+./conf/ flume configuration files
+./eclipse/ eclipse project environment files
+./lib/ libraries used by flume
+./libbuild/ libraries used by flume for building
+./libtest/ libraries used by flume for testing
+./src/ahocorasick a library for multple string search
+./src/antlr flume config language ANTLR grammer files
+./src/gen-java autogenrated java source files (from antlr/thrift)
+./src/java flume java source code
+./src/javaperf flume performance tests (out of date)
+./src/javatest flume unit tests
+./src/javatest-torture flume reliability tests (out of date)
+./src/thrift flume thrift idl files (for rpc)
+./src/webapps flume webapp jsp source code
+----
+
+Files created by build:
+----
+./build
+----
+
+Default directory for eclipse build.
+----
+./build_eclipse
+----
+
+== Files in `.gitignore`
+
+These files are autogenerated or build/eclipse specific
+----
+src/java/com/cloudera/flume/package-info.java
+build/
+build_eclipse/
+*#
+.classpath
+.project
+.settings
+----
+
+== eclipse project setup.
+
+These files are kept out of there normal place because they tend to be
+modified by users and are eclipse specific. The files can be copied
+into place to get a head start on a project. The actual dotfiles are
+in the the .gitignore file.
+
+----
+cp eclipse/project .project
+cp eclipse/classpath .classpath
+----
+
+
+== Building thrift
+
+This will create a repository in ./apache-thrift
+----
+git clone git://git.apache.org/thrift.git
+cd thrift
+git fetch
+git checkout -b thrift-0.2.0 origin/tags/thrift-0.2.0
+bootstrap.sh
+configure
+make
+sudo make install
+----
+
+Problem: During bootstrap.sh
+----
+configure.ac:44: error: possibly undefined macro: AC_PROG_LIBTOOL
+ If this token and others are legitimate, please use m4_pattern_allow.
+ See the Autoconf documentation.
+----
+Solution: install libtool
+----
+sudo apt-get install libtool
+----
+
+== Generated source
+
+These files should not be checked in unless their source files are modified.
+
+For ANTLR
+----
+src/gen-java/com/cloudera/flume/conf/FlumeDeployLexer.java
+src/gen-java/com/cloudera/flume/conf/FlumeDeployParser.java
+----
+
+== Running test with ant.
+
+----
+ant test
+----
+
+=== Running only specified test cases from ant
+
+----
+ant test -Dtestcase=<TestFile>
+----
+
+where <TestFile> is a class name without .java or .class or path.
+
+(How do is specify just a function?)
+
+== origin/master invariants
+
+Always should build.
+
+Ideally tests all pass
+
+== push invarients
+
+We should tag pushes with JIRA nubmers.
+
+
+== Debugging JSP web applications in jetty inside eclipse.
+
+The default setup for flume is to run its servlets from precompiled
+jsps. Debugging is easier when it is more dynamic and our servlet
+container Jetty can dyamically comile jsp pages to hasten the
+debugging process. This is useful when using running the Flume Master
+or FlumeNode from inside eclipse.
+
+* Make sure `tools.jar` is in your java classpath. This allows the compiler to be present.
+* Ant is used to compile jsps. Make sure some version of ant-launcher.jar and ant-1.x.x.jar is in your build path. (if you are in eclipse for example)
+* The default is to point to a the web app at a precompiled version of of the servlets. There is a hook in flume-site.xml to point the jetty at a directory full of jsps. It assumes that hte flume directory is the base for relative paths or can use a fully qualified path
+
+----
+ <property>
+ <name>flume.webapps.path</name>
+ <value>src/webapps</value>
+ <description>This is the path use to the web apps that display
+ flume node/master data
+
+ Use src webapps for development.
+ </description>
+ </property>
+----
+
+
+== Compiling JSPs
+
+BUILD FAILED
+/home/jon/flume/build.xml:471: java.lang.ExceptionInInitializerError
+ at org.apache.jasper.JspCompilationContext.createCompiler(JspCompilationContext.java:197)
+ at org.apache.jasper.JspC.processFile(JspC.java:772)
+ at org.apache.jasper.JspC.execute(JspC.java:908)
+ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+...
+
+Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category) (Caused by org.apache.commons.logging.LogConfigurationException: No suitable Log constructor [Ljava.lang.Class;@31554233 for org.apache.commons.logging.impl.Log4JLogger (Caused by java.lang.NoClassDefFoundError: org/apache/log4j/Category))
+ at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:543)
+ at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:235)
+ at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:209)
+ at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:351)
+ at org.apache.jasper.compiler.Compiler.<clinit>(Compiler.java:55)
+ ... 25 more
+
+
+Make sure that log4j-xxx.jar is in your CLASSPATH.
+
+== Avro.
+
+Using a post 1.2.0 version that has reflection that supports
+Strings, byte[]'s and extracting fields defined in super classes.
+
+requires (in repo):
+----
+lib/avro-1.2.0-dev.jar # (trunk hash 8911c848 ; more avro 1.3 than 1.2)
+lib/paranamer-1.5.jar # extra reflection stuff
+lib/jackson-core-asl-1.1.1.jar # json parser
+lib/jackson-mapper-asl-1.1.1.jar # json parser
+----
+
+== Hudson
+
+Hudson should pull the head of the repository and build. (Chad).
+
+Add 'git clean -xdf' to the build execute shell config panel to force
+hudson to rebuild.
+
+archive artifacts: build/test/TEST*.txt to get outputs of unit test
+
+== Developer mode.
+
+This is an option in the bin/flume for using eclipse built class files
+instead of ant built class files.
+
+in bash one would set FLUME_DEVMODE to true:
+
+----
+$ declare -x FLUME_DEVMODE=true
+----
+
+It is assumed that the eclipse build path is build_eclipse/.
+
+== License
+
+----
+Licensed to Cloudera, Inc. under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. Cloudera, Inc. licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+----
Oops, something went wrong.

0 comments on commit 0efd816

Please sign in to comment.