Large-scale neural network modeling using Giraph
Note: The code has been fully tested on CDH4.4, but it is not working on CDH5 yet.
Software Packages needed:
- Maven 3
- Giraph 1.0.0
- Java 1.6
- Python 2.7 or higher
Generate XML configuration files
A directory called 'xml_input4Hadoop' will be generated under the project home.
Compile & upload to Hadoop cluster
Modify upload.sh first with appropriate user name and Hadoop cluster name.
This command will compile the project and upload necessary files to Hadoop cluster in the directory giraph under $HOME
Neural network generation with MapReduce
ssh into your Hadoop cluster:
chmod +x *.sh
The neural network will be generated in neuron_input on HDFS.
Run Giraph job for modeling
Delete _logs files generated at the previous step, otherwise Giraph will take it as input and an error would occur.
hadoop fs -rm -r neuron_input/_logs
Run Giraph job
- post-analysis/hive/create_table.hql: creating hive table from Giraph's output
- post-analysis/hive/firing_pattern.hql: extracting fired neurons
- post-analysis/hive/firing_rates.hql: extracting firing rates
- post-analysis/plot/graphs.R: plotting firing patterns
- post-analysis/plot/channel_plot.R: plotting firing patterns for multiple channels