Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 213 lines (149 sloc) 6.172 kb
31ecbbf Initial commit
Andrew Bayer authored
1 Flume Developer Notes
2 =====================
3
4 // This is in asciidoc markup
5
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
6 == Introduction
31ecbbf Initial commit
Andrew Bayer authored
7
8 This is meant to be a a guide for issues that occur when building,
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
9 debugging and setting up Flume as developer.
31ecbbf Initial commit
Andrew Bayer authored
10
11 == High level directory and file structure.
12
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
13 Flume uses the Maven build system and has a Maven project object model
14 (pom) that has many components broken down into Maven modules. Below
15 we describe the contents of different directories.
16
31ecbbf Initial commit
Andrew Bayer authored
17 ----
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
18 ./bin/ Flume startup scripts
19 ./conf/ Flume configuration file samples
e1becc5 FLUME-876. Update README, DEVNOTES and NOTICE
Prasad Suresh Mujumdar authored
20 ./flume-ng-core Flume core module
21 ./flume-ng-dist Flume distribution package module
22 ./flume-ng-channels Flume channels modules (memory, JDBC and File)
23 ./flume-ng-node Flume node module
24 ./flume-ng-sinks Flume sink modules (HDFS)
31ecbbf Initial commit
Andrew Bayer authored
25 ----
26
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
27 The files exclusions in `.gitignore` are either autogenerated by Maven or Eclipse.
28
29 == Building and Testing Flume
30
31
e1becc5 FLUME-876. Update README, DEVNOTES and NOTICE
Prasad Suresh Mujumdar authored
32 === Prerequisites
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
33
2bcc1d4 Mike Percy APACHE-BUILD. Update various files to prepare for 1.2.0 release.
mpercy authored
34 You need to have Apache Maven 3.x installed.
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
35
36 === Using Maven
37
c00c046 FLUME-912: DEVNOTES contains outdated info
Eric Sammer authored
38 We are using Maven v3.x. The Maven build system steps through several phases
39 to create build artefacts. At the highest level, the phases that are relevent
40 to most devs are "compile" -> "test" -> "package" -> "install".
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
41
2c486a3 Mike Percy Add MAVEN_OPTS build workaround to DEVNOTES for 1.2.0
mpercy authored
42 Set MAVEN_OPTS to give the Flume build enough RAM to build.
43
44 export MAVEN_OPTS="-Xmx512M -XX:MaxPermSize=512M"
45
46 Note: If you see a permgen error (below), you need to increase the perm gen size.
47
48 [ERROR] PermGen space -> [Help 1]
49 [ERROR]
50 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
51 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
52 [ERROR]
53 [ERROR] For more information about the errors and possible solutions, please read the following articles:
54 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
55
56 Builds
57 ------
58
c00c046 FLUME-912: DEVNOTES contains outdated info
Eric Sammer authored
59 A development build that runs unit tests and installs to local Maven repo.
60 This builds and tests all plugins.
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
61
31ecbbf Initial commit
Andrew Bayer authored
62 ----
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
63 mvn install
31ecbbf Initial commit
Andrew Bayer authored
64 ----
65
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
66 A development build that skips the execution of unit tests.
31ecbbf Initial commit
Andrew Bayer authored
67
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
68 ----
69 mvn install -DskipTests
70 ----
31ecbbf Initial commit
Andrew Bayer authored
71
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
72 A development build that runs unit tests. (no package generation)
31ecbbf Initial commit
Andrew Bayer authored
73
366bed4 FLUME-47: cleaned up DEVNOTES, specifically around eclipse integration
Andrew Bayer authored
74 ----
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
75 mvn test
31ecbbf Initial commit
Andrew Bayer authored
76 ----
77
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
78 A development build that runs unit tests including only specific tests
79 (where <TestFile> is a regex of a class name without .java or .class
80 or path).
81
82 ----
83 mvn test -Dtest=<ClassRegex>
84 ----
366bed4 FLUME-47: cleaned up DEVNOTES, specifically around eclipse integration
Andrew Bayer authored
85
71f516d FLUME-716: Add ability to exclude or include arbitrary tests to maven bu...
Andrew Bayer authored
86 ==== Including or excluding specific sets of tests.
87
88 We've added hooks to the maven build that will enable you to exclude
89 or include specific tests on a test run. This is useful for excluding
90 flakey tests or making a build that focuses solely upon flakey tests.
91
92 To do this we created two variables:
93
94 # test.include.pattern
95 # test.exclude.pattern
96
97 These variables take regular expression patterns of the files to be
98 included or excluded.
99
100 For the next set of examples, let's say you have flakey test called
101 TestFlaky1 and TestFlaky2.
102
103 You can execute tests that skip TestFlaky1 and TestFlaky2 by using the
104 following command line:
105
106 ----
107 mvn test -Dtest.exclude.pattern=**/TestFlaky*.java
108 ----
109
110 Alternately, you could be more explicit
111
112 ----
113 mvn test -Dtest.exclude.pattern=**/TestFlaky1.java,**/TestFlaky2.java
114 ----
115
116 Conversely, you could execute only the flaky tests by using:
117
118 ----
119 mvn test -Dtest.include.pattern=**/TestFlaky*.java
120 ----
121
122 You can also have a combination of imports and exports. This runs
123 TestFlaky* but skips over TestFlaky2:
124
125 ----
126 mvn test -Dtest.include.pattern=**/TestFlaky*.java -Dtest.exclude.pattern=**/TestFlaky2.java
127 ----
128
129 NOTE: Both test.exclude.pattern and test.include.pattern get
130 overridden if the test parameter is used. Consider:
131
132 ----
133 mvn test -Dtest.exclude.pattern=**/TestFlaky*.java -Dtest=TestFlaky1
134 ---
135
136 In this case, TestFlaky1 will be run despite being in the
137 test.exclude.pattern.
138
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
139 === Running the most recent build
140
141 To run the most recent build of Flume, first build the distribuion
142 packages.
31ecbbf Initial commit
Andrew Bayer authored
143
144 ----
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
145 mvn install -DskipTests
31ecbbf Initial commit
Andrew Bayer authored
146 ----
147
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
148 == Integrated Development Environments for Flume
31ecbbf Initial commit
Andrew Bayer authored
149
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
150 Currently most Flume developers use the Eclipse IDE. We have included
151 some instructions for getting started with Eclipse.
b4f5b6a CDH-1530: Update webapps symlink for git repo
Andrew Bayer authored
152
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
153 === Setting up a Flume Eclipse projects from the Maven POMs.
31ecbbf Initial commit
Andrew Bayer authored
154
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
155 If you use Eclipse we suggest you use the m2eclipse plugin available
156 here to properly create an environment for dev and testing in Eclipse.
31ecbbf Initial commit
Andrew Bayer authored
157
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
158 http://m2eclipse.sonatype.org/
31ecbbf Initial commit
Andrew Bayer authored
159
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
160 After installing it in Eclipse you will want to "Import" the Flume
161 pom.xml project.
31ecbbf Initial commit
Andrew Bayer authored
162
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
163 This can be done by going to the Eclipse applications menu, navigating
164 to File > Import... > Existing Maven Projects. From there, browse to
165 and select the directory that contains the root of the Flume project.
31ecbbf Initial commit
Andrew Bayer authored
166
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
167 == Rules of the Repository
168
169 We have a few basic rules for code in the repository.
97c03ea Bump to v0.9.3
Andrew Bayer authored
170
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
171 The master/trunk pointer:
97c03ea Bump to v0.9.3
Andrew Bayer authored
172
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
173 * MUST always build.
174 * SHOULD always pass all unit tests
31ecbbf Initial commit
Andrew Bayer authored
175
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
176 When commitng code we tag pushes with JIRA numbers, and their short descriptions.
177 Generally these are in the following format:
366bed4 FLUME-47: cleaned up DEVNOTES, specifically around eclipse integration
Andrew Bayer authored
178
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
179 ----
180 FLUME-42: Description from the jira
181 ----
182
183 All source files must include the following header (or a variant
184 depending on comment characters):
185
186 ----
b4f5b6a CDH-1530: Update webapps symlink for git repo
Andrew Bayer authored
187 /**
e1becc5 FLUME-876. Update README, DEVNOTES and NOTICE
Prasad Suresh Mujumdar authored
188 * Licensed to the Apache Software Foundation (ASF) under one
189 * or more contributor license agreements. See the NOTICE file
190 * distributed with this work for additional information
191 * regarding copyright ownership. The ASF licenses this file
192 * to you under the Apache License, Version 2.0 (the
193 * "License"); you may not use this file except in compliance
194 * with the License. You may obtain a copy of the License at
b4f5b6a CDH-1530: Update webapps symlink for git repo
Andrew Bayer authored
195 *
e1becc5 FLUME-876. Update README, DEVNOTES and NOTICE
Prasad Suresh Mujumdar authored
196 * http://www.apache.org/licenses/LICENSE-2.0
b4f5b6a CDH-1530: Update webapps symlink for git repo
Andrew Bayer authored
197 *
e1becc5 FLUME-876. Update README, DEVNOTES and NOTICE
Prasad Suresh Mujumdar authored
198 * Unless required by applicable law or agreed to in writing,
199 * software distributed under the License is distributed on an
200 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
201 * KIND, either express or implied. See the License for the
202 * specific language governing permissions and limitations
203 * under the License.
b4f5b6a CDH-1530: Update webapps symlink for git repo
Andrew Bayer authored
204 */
f9a4d9b FLUME-677: Update build instructions in devnotes
Andrew Bayer authored
205 ----
206
207 No build generated files should be checked in. Here are some examples
208 of generate files that should not be checked:
209
210 * html documentation
211 * avro-generated source
212 * auto-generated versioning annotations
Something went wrong with that request. Please try again.