HadooPHP is a framework for writing Hadoop Streaming jobs in PHP
- Packages jobs as PHAR files for speedy and convenient deployment.
- Also generates shell scripts to invoke a job (requires
$HADOOP_HOMEto be defined).
- Automatic detection of streaming settings (so it knows about the configured key field separators and lengths et cetera).
- Abstracted input parsing and splitting (only supports line-based input at the moment though).
- Supports custom arguments for the Hadoop invocation.
- Something like
KeyValueTextInputFormatdoes not work, partly because passing it as
-inputformatsets the property to
StreamingInputFormat(must investigate why), and
-D mapred.input.format.classseems to have no effect. Unsure how to fix.
- Unit testing capabilities (Mappers and Reducers could be tested locally).
- Support for input formats other than TextInputFormat (only line-based stuff works out of the box right now).
examples sub-directory contains a number of examples.
- A functioning Hadoop installation, with namenode, jobtracker and all other components running.
- If you are developing locally, follow the Hadoop Quick Start Guide to set up pseudo-distributed mode.
- The PHAR PHP extension must be enabled, and
phar.readonlymust be set to
0in php.ini if you want to compile jobs.
Create a folder (this folder name will be the job name later on), with a
Mapper.php containing the mapper class, a
Reducer.php containing a reducer class (if desired), and, if you want, an
ARGUMENTS file with additional arguments.
Note: when you have an ARGUMENTS file, you must include full
-reducer commands, see the examples. Any
-D flags in ARGUMENTS must also precede any other switches such as
-file. Every line in ARGUMENTS must be terminated by a backslash because the raw contents of the file are inserted into the generated shell script.
Assuming your job (and thus folder) name is "TpsReportCount", run:
bin/compile.sh TpsReportCount <BUILDDIR>
Note: the build dir must exist and be writeable.
You can pass path names to include in the package and the default timezone to set for scripts to
compile.sh; invoke it without arguments for usage help. The
--debug switch enables debug mode which means the
HADOOPHP_DEBUG constant will be set to true, you can use this to add counter or message emitters to your code that should only run in such a case.
Assuming your job name is "TpsReportCount", run:
path/to/builddir/TpsReportCount.sh <HDFSINPUTPATH...> <HDFSOUTPUTPATH>
Note: you may pass any number of input path names; the last path name given is the output path where results will be written to.
You may also pass the path to a Hadoop config dir (equivalent to the
--config argument of the
path/to/builddir/TpsReportCount.sh -c path/to/dir/with/remote-cluster-config <HDFSINPUTPATH...> <HDFSOUTPUTPATH>