Constructing call-based dependency networks of crates.io as conceptually described in
TL;DR: What does RustPräzi do?
With RustPräzi, we go from coarse-grained package-based dependency networks (such as what GitHub uses for their vulnerable package detection) to more fine-grained call-based dependency networks. These allow us to track, for example, whether a vulnerable function of a library is actually being used and whether a security warning really needs to be raised. This is much more precise than package-based dependency networks. In fact, RustPräzi makes such analyses a lot more precise (upto 3x).
RustPräzi opens the door to many new or more precise analyses:
- Fine-grained security vulnerability propagation checking
- Precise license compliance checking
- Change impact and deprecation analysis ("Which clients break if I as a library maintainer remove this deprecated method?")
- Health analyses of an entire ecosystem ("What are the most central functions?", "Where should we focus our testing efforts?", ...)
- ... and more!
- The Rust toolchain with
rustup(download at the offical website)
- Python 2.7 or 3.7
- GNU Parallel
- A pre-built binary of LLVM 4.0 (download at official website). In the
config.ini(root of the repository), specify the path to the uncompressed LLVM binary.
- Recommended OS: Ubuntu 16.04.3 LTS
⚠️Building crates can be dangerous as for some crates, this includes running the tests. Hence, it is advised to do it in a sandboxed environment. 💻We recommend running it on a very powerful system. Compiling 80k+ crates is no easy feat.
1. Create a
conf.ini file at the root of the project with the following content
encoding=utf-8 [llvm] # specify the path to the untared LLVM binary folder. path=/path_where/clang+llvm-4.0.0-[your_platform] [compiler] stable=1.23.0 nightly=1.24.0 [storage] # all data will be stored in this folder path=/where/you/want/to/store/prazi/data
Since the bitcode generation changed in newer versions of Rust, we advise to stick to the compiler versions specified above.
2. Constructing call graphs of crates
- Compile the tool
cargo build --bin prazi --release
- Download crates, the downloader will fetch the latest index data, build a list of releases and then download/untar them
- Rewriting manifests, the manifest rewriter will fix invalid
Cargo.tomlfiles (e.g., specifying a non-existent local dependency) by emulating a dry-run of
- Building crates, it will first attempt to build all downloaded crates using a stable version of the compiler (as specified in
conf.ini). To use a nightly version for failing builds, prepend the flag
- Building LLVM call graphs
2. Construct RustPräzi
rustfiltfor demangling of Rust symbols
cargo install rustfilt
- Run graph generator script
./create_prazi_graph.sh 2> err.log 1> out.log
Two graphs are generated:
../cdn/graphs/callgraph.ufi.merged.graph-- the call-based dependency network (CDN)
../cdn/graphs/crate.dependency.callgraph.graph-- the packaged-based dependency network derived from the CDN
3. Graph analysis with RustPräzi
Loading Präzi with NetworkX
import networkx as nx import re regex = r"^(.*?) \[label:" def load_prazi(file): PRAZI = nx.DiGraph() with open(file) as f: #callgraph.ufi.merged.graph for line in f: if "->" not in line: g = re.match(regex, line) if g: PRAZI.add_node(g.group(1).strip('"')) else: print "error, could not extract node: %s" % line else: g = re.match('\W*"(.*)" -> "(.*)";', line) if g: PRAZI.add_edge(g.group(1), g.group(2)) else: print "error, could not extract edge: %s" % line return PRAZI def load_prazi_dep(file): PRAZI_DEP = nx.DiGraph() with open(file) as f: #crate.dependency.callgraph.graph for line in f: if "io :: crates :: " in line: if "->" not in line: PRAZI_DEP.add_node(line[:-2]) else: g = re.match('\W*"(.*)" -> "(.*)";', line) if g and ("io :: crates" in g.group(1) and "io :: crates" in g.group(2)): PRAZI_DEP.add_edge(g.group(1), g.group(2)) else: print "skip edge: %s" % line else: continue return PRAZI_DEP
This project is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in RustPräzi by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.