This repository has been archived by the owner. It is now read-only.
Data relationships play a vital role in various data analytics, structured machine learning, and data mining applications. These relationships can be expressed as graphs. GraphBuilder is a Java library for preparing graphs from large volumes of unstructured and semi-structured data. Processing large graphs in a distributed computing environment presents multiple system-level challenges, from parallelizing the graph construction algorithms to achieving balanced system utilization. In addition, the resultant graph must be carefully partitioned to ensure that subsequent application processing is properly balanced. GraphBuilder addresses these challenges with a carefully-designed multi-stage pipeline that leverages the Apache Hadoop framework to achieve scalability. How to build the GraphBuilder? ------------------------------ The GraphBuilder is using Maven for building the package, ensure maven is installed on the system. To build package type: $> mvn package How to use the GraphBuilder? ---------------------------- The GraphBuilder library can be used in multiple ways. Library groups multiple MapReduce jobs in four major groups: a) GraphCreation b) ComputeNetworkInfo c) GraphNormalization d) GraphPartitioning. These group can be chained to form graph construction pipeline depending on an application requirements (e.g. Please refer construction pipeline for link and doc-word graph in demoapps dir). Library can be used as tool to construct link and doc-word graph for Wikipedia using following command: * Link graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.linkgraph.LinkGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> * Doc-word graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> <optional stop word filter list> Sample application code ----------------------- Early stage in graph building pipeline is data dependent, hence we abstracted out the data dependent code from the library and provided interface to plug-in data-specific code. Sample code for those interface can be found in demoapps directory. It contains code for generating link-graphs and doc-word graph out of Wikipedia XML dump. Interfaces are documented, please refer a Javadoc in the doc directory. Contact ------- More information visit www.01.org/graphbuilder