This repository contains code for Graph Cities. Graph Cities are 3D visual representations of partitions of a graph edge set into maximal connected subgraphs each of which is called a fixed point of degree peeling. Each such connected subgraph is visually represented as a Building. A poly-log bucketization of the size distribution of the subgraphs represented by the buildings generates a 2D position for each bucket. The Delaunay triangulation of the bucket building locations determines the street network. We illustrate Graph Cities for the Friendster social network(1.8 billion edges), a co-occurrence keywords network derived from the Internet Movie Database (115 million edges), and a patent citation network(16.5 million edges). Up to 2 billion edges, all the elements of their corresponding Graph Cities are rendered in few minutes (excluding I/O time). The actual Graph Cities computation takes about 2 hours on a 64 GB RAM commodity computer. Our ultimate goal is to provide tools to build humanly interpretable descriptions of any graph, without being constrained by the graph size. It consists of four sub-modules: wave-decomposition, Graph_City_Web, fpViewer and graph-strata.
Web interactions are fully tested using Chrome. There are some issues with Safari. Therefore, we recommand for the current version to only use Chrome as a browser. A video illustrating our current interface can be accessed at here.
The main 2022 publication is:
@article{Abello2022GigaGC,
title={Giga Graph Cities: Their Buckets, Buildings, Waves, and Fragments},
author={James Abello and H. Zhang and Daniel Nakhimovich and Chengguizi Han and Mridul Aanjaneya},
journal={IEEE Computer Graphics and Applications},
year={2022},
volume={42},
pages={53-64}
}This is an NSF-funded project (NSF IIS-1563816 and IIS-1563971), led by Prof. James Abello. It is partially funded by Nokia Bell Labs, Siemens Research, and mgvis.com. It constitutes a major part of the incoming Ph.D. thesis of Haoyang Zhang.
- Overall Project Design and Coordination: James Abello
- Algorithms: James Abello, Haoyang Zhang, and Daniel Nakhimovich
- Overall Integration and Platform Deployment: Haoyang Zhang
- Graph Cities Rendering: Chengguizi Han, Daniel Nakhimovich, and Mridul Aanjaneya
- Parallel computation for Meta-DAG: Ishaan Ivaturi
- Force Directed Layout Applications:
- fpViewer: James Abello and Haoyang Zhang
- Graph Strata base code: James Abello and Qi Dong
- Interaction Tools and Documentation: Haoyang Zhang, James Abello, and Shaad Quazi
- Data Sets Exploration: Fatima AlSaadeh, Tim Tangerlini, and Peter Broadwell
- Mini-Cloud Deployment: Haoyang Zhang, and Charles Hedrick
| Name | Source | V | E | cc | FP |
|---|---|---|---|---|---|
| com-friendster | SNAP | 65608366 | 1806067135 | 1 | 72 |
| ogbn-papers100M | OGB | 111059956 | 1614062356 | 151 | 159 |
| movies-tag-cooccurence | IMDb | 218052 | 115050370 | 38 | 78 |
| name | Source | V | E | cc | FP |
|---|---|---|---|---|---|
| ogbn-products | OGB | 2400608 | 61859012 | 4237 | 102 |
| ogbn-proteins | OGB | 132534 | 39561252 | 1 | 157 |
| ogbn-mag | OGB | 1134649 | 21090258 | 1 | 64 |
| cit-Patents | SNAP | 3774768 | 16518947 | 3627 | 41 |
| name | Source | V | E | cc | FP |
|---|---|---|---|---|---|
| Pandora papers | ICIJ Offshore Leaks Database | 1968951 | 2832596 | 31347 | 31 |
| ogbn-arxiv | OGB | 169343 | 1157799 | 1 | 22 |
| name | V | E | cc | FP |
|---|---|---|---|---|
| Danish Fabula | 19738 | 28292 | 414 | 6 |
| Game of Thrones | 796 | 2823 | 1 | 9 |
| Starwars | 111 | 444 | 1 | 7 |
A collection of "interesting" patterns can be accessed at Graph City Patten Gallery.
If you use our code, please consider citing our paper.
@article{Abello2022GigaGC,
title={Giga Graph Cities: Their Buckets, Buildings, Waves, and Fragments},
author={James Abello and H. Zhang and Daniel Nakhimovich and Chengguizi Han and Mridul Aanjaneya},
journal={IEEE Computer Graphics and Applications},
year={2022},
volume={42},
pages={53-64}
}
@inproceedings{Abello2021GraphCT,
title={Graph Cities: Their Buildings, Waves, and Fragments},
author={James Abello and Daniel Nakhimovich and Chengguizi Han and Mridul Aanjaneya},
booktitle={EDBT/ICDT Workshops},
year={2021}
}
@article{Abello2020GraphW,
title={Graph Waves},
author={James Abello and Daniel Nakhimovich},
journal={Big Data Res.},
year={2020},
volume={29},
pages={100327}
}
@article{Abello2013FixedPO,
title={Fixed points of graph peeling},
author={James Abello and François Queyroi},
journal={2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013)},
year={2013},
pages={256-263}
}Detailed subfolder structures and comments refer to corresponding sections.
./
├── wave-decomposition/ #backend computation algorithms and decomposition results
│ ├── scripts/
│ ├── src/
│ └── ${DATASET}/
├── Graph_City_Web/ #main frontend visualization: Graph City web application
│ ├── data/
│ ├── data_dags/
│ ├── data_maps/
│ ├── label-history/
│ ├── lib/
│ ├── models/
│ ├── node_modules/
│ ├── patterns/
│ ├── python/
│ ├── scripts/
│ ├── textures/
│ ├── three.js/
│ ├── fpViewer/ #soft link to ../fpViewer/
│ ├── wave-decomposition/ #soft link to ../wave-decomposition/
│ └── ${DATASET}/ #soft link to ../wave-decomposition/${DATASET}/
├── fpViewer/ #complementary frontend visualization: fpViewer
│ ├── lib/
│ ├── localLib/
│ ├── style/
│ └── wave-decomposition/ #soft link to ../wave-decomposition/
└── graph-strata/ #complementary frontend visualization: Graph Strata
├── bin/
├── data2/
├── doc/
├── lib/
├── public/
├── src/
└── temp2/You may clone this repo just by typing the following line in your console:
git clone https://github.com/endlesstory0428/Graph-Cities.gitThis will create a Graph City work space folder Graph-Cities. We refer to this folder as "Work Space" ./.
Before you start creating your own Graph City, you first need to compile "Graph Cities Infrastructure".
In this version, we provide compiled binary files that one can directly use them after making them executable by the following lines.
cd wave-decomposition
chmod +x buffkcore; chmod +x cc-layers-mat; chmod +x DAGMetaNode_touch; chmod +x edgeCutCompress_touch; chmod +x entropy; chmod +x ewave_next; chmod +x fpmetagraph; chmod +x gridmap; chmod +x lccBuck; chmod +x preproc; chmod +x topSrc_touch; chmod +x waveCC_touch; chmod +x waveFragLevel_touch; chmod +x waveFragLevel_touch_bucket; chmod +x wavelayercc_direct; chmod +x wavemaps; chmod +x wavemapsWaveByWave; chmod +x filterBirdHorse; chmod +x filterGorilla;
chmod +x scripts/freqUsed/getBuildingList; chmod +x scripts/freqUsed/mergeCCLayers; chmod +x scripts/freqUsed/getCityInfo; chmod +x scripts/freqUsed/dagBat_cpp; chmod +x scripts/freqUsed/dagBat_post_cpp; chmod +x scripts/freqUsed/numfixedpoints; chmod +x scripts/freqUsed/convert_ve; chmod +x scripts/freqUsed/getBuildingBucketFromMap_int; chmod +x scripts/freqUsed/wavemapsNames; chmod +x scripts/freqUsed/convert; chmod +x scripts/freqUsed/getMap_noWave; chmod +x scripts/freqUsed/getMapDag2-2; chmod +x scripts/freqUsed/fpmetagraphnormalize_int;
chmod +x scripts/test/bucket2strata; chmod +x scripts/test/addDagInfo; chmod +x scripts/test/bucket2city; chmod +x scripts/test/addBuckESize; chmod +x scripts/test/checkLargeNode; chmod +x scripts/test/bucket2sample; chmod +x scripts/test/lcc-json2csv; Under your Work Space './', type the following commands in your console to compile .cpp files.
cd wave-decomposition
makeUnder your Work Space './', type the following commands in your console to create soft links among the four sub-modules
ln -s $(pwd)/wave-decomposition $(pwd)/Graph_City_Web/
ln -s $(pwd)/wave-decomposition $(pwd)/fpViewer/
ln -s $(pwd)/fpViewer $(pwd)/Graph_City_Web/
ln -s $(pwd)/graph-strata $(pwd)/wave-decomposition/Under your Work Space ./, type the following commands in your console to configure your server address and ports.
python3 config.py -a ${server_address} -c ${graph_city_port} -s ${graph_strata_port}Under your Work Space ./, type the following commands in your console to set up the "Graph Strata" server.
cd ./graph-strata/
mkdir data
npm ci
chmod +x run.sh
./run.shUnder your Work Space ./, type the following commands in your console to set up the "Graph Cities" web application server.
cd ./Graph_City_Web/
ln -s $(pwd) $(pwd)/public
mkdir data_dags
npm ci
export NODE_OPTIONS="--max-old-space-size=65536"
node app_addon.jsInput your graph into the folder ./wave-decomposition/${DATASET}/. The Graph Cities Infrastructure accepts two files as inputs:
- Graph Topology: An ASCII edge list file
${DATASET}.txtwhere each line is a pair of integers separated by a tab, - Vertex Labels: (Optional) a header-free CSV (following
RFC 4180standard) vertex label file${DATASET}_label.csvwith two columns:vertex_index,vertex label.
Your directory file structure should be as follows:
./
├── wave-decomposition/
│ ├── scripts/
│ ├── src/
│ └── ${DATASET}/ #your input graph
│ ├── ${DATASET}.txt #pairs of integers separated by a tab
│ └── ${DATASET}_label.csv #header-free CSV
├── Graph_City_Web/
├── fpViewer/
└── graph-strata/Under your Work Space ./, type the following commands in your console to process your input graph.
cd wave-decomposition/
make GRAPH=${DATASET} prepare
cd ../Graph_City_Web/
make GRAPH=${DATASET} retriveA detailed explanation of the commands above can be found in the wave-decomposition section and Graph_City_Web section.
Use your web browser to access the following webpage containing your Graph City ready for exploration.
http://${server_address}:${graph_city_port}/?city=${DATASET}wave-decomposition/ #backend computation algorithms and decomposition results
├── scripts/ #python source code
├── src/ #c++ source code
├── bindump.sh
├── Makefile
└── ${DATASET}/ #your own graph
├── ${DATASET}.txt #input edge list file
├── ${DATASET}_label.csv #input vertex label file
├── cityMesh/ #folder for Graph City main view geometry files
├── dag/ #folder for Graph City Meta-DAG files
├── ${DATASET}_layers/ #folder for iterative edge fixed point decomposition results
├── ${DATASET}_waves/ #folder for wave-fragment decomposition results
├── building2bucket-${DATASET}.json #mapping from spiral layout bucket to its representative building
├── $cloneCnt.csv #mapping from vertex to its iterative edge fixed point decomposition vertex clone count
├── graph-*-*.json #spiral layout bucket information
├── ${DATASET}-bucket2peels.json #mapping from spiral layout bucket to its corresponding fixed point values
├── ${DATASET}-fpmeta.csv #iterative edge fixed point decomposition vertex intersection graph topology
├── ${DATASET}-fpmeta.ids #iterative edge fixed point decomposition vertex intersection graph vertex labels
├── ${DATASET}-info.json #spiral layout information
├── ${DATASET}-layers-dists.json #iterative edge fixed point decomposition distribution
├── ${DATASET}-lccBuck.l-lcc-b.csv #mapping from connected fixed point to its bucket size indicator
├── ${DATASET}-lccWaves.vBuck.b.p.mm.json #grid map information
├── ${DATASET}-summary.json #summary information of decomposition
├── ${DATASET}.cc #mapping from vertex to its connected component
├── ${DATASET}.deg #mapping from vertex to its degree
├── ${DATASET}_entropy.json #average vertex diversity of each iterative edge fixed point
├── lcc-duplicates.json #information of connected fixed point with the same fixed point value and number of vertices and edges
├── localDeg.csv #mapping from vertex to its local degree on each iterative edge fixed point
└── metagraph_normalized.txt #iterative edge fixed point decomposition vertex intersection graph summarized by spiral layout bucketsFor an input graph ${DATASET} residing in ./wave-decomposition/${DATASET}, a simple one-shot command to process it is the following:
make GRAPH=${DATASET} prepareThis make program automatically invokes the following commands:
- The following make program preprocesses the input graph as a bidirectional binary edge list, and outputs the vertex degrees
${DATASET}.degand connected components${DATASET}.cc.
make GRAPH=${GRAPH} union- The following make program applies the "iterative edge fixed point decomposition" to the preprocessed graph, and outputs a folder containing the "edge fixed point decomposition" results
${DATASET}_layers/
make GRAPH=${GRAPH} decomp- The following make program applies the "wave-fragment vertex decomposition" to fixed points, and outputs a folder containing results
${DATASET}_waves/
make GRAPH=${GRAPH} dwave-all_nextThe "wave-fragment vertex decomposition" of a single fixed point can be invoked as follows:
make GRAPH=${GRAPH} LAYER=$(FIXEDPOINT) dwave_next- The following make program computes connected fixed points, and obtains a mapping from each connected wave to its corresponding unique connected fixed point. The updated outputs are stored under
${DATASET}_layers/and${DATASET}_waves/.
make GRAPH=${GRAPH} cc-layers-uf
make GRAPH=${GRAPH} wave-layer-cc_direct- The following make programs and python scripts collect connected fixed point information and generate a mapping from connected fixed point ids to log bucket ids. The output consists of three collections of files:
graph-*-*.json: connected fixed point information in each bucket.lcc-duplicates.json: a mapping from a representative connected fixed point to a list of connected fixed points with the same number of vertices, edges, and the fixed point value as the representative one.${DATASET}-info.json: log bucket information.
make GRAPH=${GRAPH} fp-info_int
$(PYTHON) scripts/test/lcc-json2csv.py ${GRAPH}
make GRAPH=${GRAPH} bucket_int- The following make program collects wave-fragment decomposition information as building information under the
${DATASET}_waves/folder.
make GRAPH=${GRAPH} wavemap_cpp- The following make program collects iterative edge fixed point vertex local degree
localDeg.csv, computes vertex diversity, and generates summary sculpture information as${DATASET}-layers-dists.json, and${DATASET}_entropy.json.
make GRAPH=${GRAPH} sculpture_int_cpp- The following make program computes "iterative edge fixed point vertex intersection graph" and generates "road network" information as
${DATASET}-fpmeta.csv,${DATASET}-fpmeta.ids, andmetagraph_normalized.txt.
make GRAPH=${GRAPH} intersection_int_cpp- The following make program collects connected fixed point information, and generates grid map information as
building2bucket-${DATASET}.json,${DATASET}-bucket2peels.json,${DATASET}-lccBuck.l-lcc-b.csv, and${DATASET}-lccWaves.vBuck.b.p.mm.json.
make GRAPH=${GRAPH} gridmap_cpp- The following make program generates the Graph City main view geometry files in the folder
cityMesh/. This geometry is derived from the building information contained in the folder${GRAPH}_waves/
make GRAPH=${GRAPH} geom- The following make program generates summary information in the file
${DATASET}-summary.json, and distribution plots${DATASET}_*.png.
$(PYTHON) scripts/freqUsed/getCityInfo.py ${GRAPH}
$(PYTHON) scripts/test/getBestPlot.py ${GRAPH}
$(PYTHON) scripts/test/getLccDist.py ${GRAPH}- The following make programs generate internal building structures represented as Meta-DAGs. Outputs are contained under the folder
dag/.
make GRAPH=${GRAPH} frag-dag-touch-all
make GRAPH=${GRAPH} top-src-span-touch-all
make GRAPH=${GRAPH} wcc-compress-touch-all
make GRAPH=${GRAPH} wave-frag-compress-touch-all
make GRAPH=${GRAPH} edge-cut-compress-touch-all
make GRAPH=${GRAPH} frag-buck-compress-touch-all
make GRAPH=${GRAPH} dag-size- The following make program splits wave-fragment decomposition results according to logarithmic buckets under folder
${DATASET}_waves/
make GRAPH=${GRAPH} lccBuck;- The following python scripts modify buildings' information according to their interior Meta-DAG structure. The result is stored under the folder
cityMesh/
$(PYTHON) scripts/test/addDagInfo.py ${GRAPH}
$(PYTHON) scripts/test/checkLargeNode.py ${GRAPH}
$(PYTHON) scripts/test/addBuckESize.py ${GRAPH}- The following make program filters wave-fragment decomposition results according to Meta-DAG structure under folder
${DATASET}_waves/
make GRAPH=${GRAPH} filterLarge-all;
make GRAPH=${GRAPH} filterSmall-all;graph_city_web/ #main frontend visualization: Graph City web application
├── data/ #main view information and summaries
│ ├── ${DATASET}/ #soft link to ../../wave-decomposition/${DATASET}/cityMesh
│ ├── ${DATASET}-summary.json #copyed from #soft link to ../../wave-decomposition/${DATASET}/${DATASET}-summary.json
│ └── ${DATASET}_*.png #moved from #soft link to ../../wave-decomposition/${DATASET}/${DATASET}_*.png
├── data_dags/ #building interior Meta-DAG inforamtion
│ └── ${DATASET}/ #soft link to ../../wave-decomposition/${DATASET}/dag
├── data_maps/ #grid map information files
├── label-history/ #history of label request responses
├── lib/ #local library modules
├── models/ #3D objects for visualization
├── node_modules/ #node.js modules
├── patterns/ #Graph City pattern gallery
├── python/ #street network python scripts and results
│ └── ${DATASET}/ #street network files
├── scripts/ #js scripts and summary sculpture informations
├── textures/ #2D textures for visualization
├── three.js/ #THREE.js module
├── fpViewer/ #soft link to ../fpViewer/
├── wave-decomposition/ #soft link to ../wave-decomposition/
├── ${DATASET}/ #soft link to ../wave-decomposition/$DATASET/
├── index.html.template #template of Graph City Web Application HTML page
├── Makefile
└── package.json #node.js package dependenciesnode.js: v16.15.0
npm: 8.5.5
The other dependencies are listed in package.json.
- The following make program creates soft links from the source input dataset
wave-decompostion/${DATASET}/to the Graph City main frontend applicationgraph_city_web/. The result of this is the creation of a new city.
make GRAPH=${DATASET} retrive- The following make program creates soft links from the source input dataset
wave-decompostion/${DATASET}/to the Graph City main frontend applicationgraph_city_web/. The result of this is the creation of a minicity room.
make GRAPH=${DATASET} retrive-mall- The following make program creates soft links from the source input dataset
wave-decompostion/${DATASET}/to the Graph City main frontend applicationgraph_city_web/. The result of this is the creation of a vicinity park.
make GRAPH=${DATASET} retrive-vicinityfpViewer/ #complementary frontend visualization: fpViewer
├── lib/ #online modules local version
├── localLib/ #local modules
├── style/ #css
├── wave-decomposition/ #soft link to ../wave-decomposition/
├── Makefile
├── ${SUBGRAPH}_waves/ #soft link to ../wave-decomposition/${SUBGRAPH}/${SUBGRAPH}_waves/
├── ${SUBGRAPH}-idx2LayerBucket.i-l-b.csv #iterative edge fixed point decomposition vertex intersection graph labels
├── ${SUBGRAPH}-layerBucketEdge.s-t-w.csv #iterative edge fixed point decomposition vertex intersection graph summarized by grid map
├── ${SUBGRAPH}-layerBucketEdge.s-t-w.span.csv #spanning iterative edge fixed point decomposition vertex intersection graph summarized by grid map
├── ${SUBGRAPH}-lccBuck.l-lcc-b-v-e.csv #grid map information
├── ${SUBGRAPH}.cc-layers #iterative edge fixed point decomposition vertex intersection graph
└── ${SUBGRAPH}_names.csv #vertex labelsThe number of waves per connected fixed point should be no larger than 50, the number of fragments per wave should be no larger than 60, and the number of edges of each fragment should be no larger than 8192. This in effect allows the fpViewer to process fixed points with at most 50 * 60 * 8192 = 24,576,000 edges provided that a suitable parameterized hierarchy tree for the fixed point is pre-computed. In this release, the provided fpViewer is able to handle detailed visualizations of non-regular general graphs with at most 131,072 edges provided all of its fragments have no more than 8192 edges. The purpose of fpViewer is to locally explore medium-sized graphs that consist of fixed points with about 3,000 "small" fragments. This is feasible by using the intersection graph of the iterative edge fixed point vertex sets.
A full documentation for Graph Strata is availiable here.
There are some issues with Safari. Therefore, we recommand for the current version to only use Chrome as a browser.
The number of edges of the input graph should be no larger than 16,384. The purpose of Graph Strata is to locally explore small-sized graphs with variaty algorithms including iterative fixed point decomposition and wave-fragment vertices decomposition in a layered view.
