Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
metadata add metadata Aug 24, 2019
plots update plots Aug 24, 2019
results update results Aug 24, 2019
script add script Aug 24, 2019
.gitignore initial commit Aug 6, 2019
LICENSE Create LICENSE Jan 16, 2020
README.md Update README. Jan 16, 2020

README.md

Vulnerability Analysis of Smart Contracts using SmartBugs

This repository contains the RAW results of the vulnerability analysis of 9 tools on 47,587 smart contracts. We used two datasets of vulnerabilities: 1) 69 annotated vulnerable contracts 2) 47,518 contracts taken from the Ethereum network.

The raw results of the analysis on the first dataset are stored in results/<name_tool>/curated/<contract_name>. The raw results of the analysis on the second dataset are stored in results/<name_tool>/icse20/<contract_address>.

These results are presented and discussed in the following ICSE 2020 paper:

Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz.
Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts.
In ICSE 2020.

Structure of the repository

├─ metadata
│  ├─ balances.json
│  ├─ duplicates.json
│  ├─ eth_price.json
│  ├─ nb_lines.csv
│  ├─ results_curated.json
│  ├─ results_wild.json
│  ├─ unique_contracts.csv
│  ├─ vulnerabilities.json
│  └─ vulnerabilities_mapping.csv
├─ plots
│  └─ <plot_name>.png
├─ results
│  └─ <tool_name>
│     └─ <dataset_name>
│        └─ <contract_address>
│           ├─ <result.log>  # stdout of the analysis
│           └─ <result.json> # parsable output analysis
├─ script
│  ├─ combine_appraoches.py
│  ├─ generate_plot.py
│  ├─ generate_results_curated.py
│  └─ generate_results_wild.py

SB Curated Results

Execution Time Stats

# Tool Avg. Execution Time Total Execution Time
1 Honeybadger 0:00:46 0:53:11
2 Maian 0:02:57 3:23:50
3 Manticore 0:08:11 5:03:04
4 Mythril 0:01:13 1:23:42
5 Osiris 0:00:44 0:50:03
6 Oyente 0:00:36 0:41:29
7 Securify 0:01:00 1:09:08
8 Slither 0:00:03 0:03:35
9 Smartcheck 0:00:06 0:06:34

Total: 13:34:37

Accuracy

Category Honeybadger Maian Manticore Mythril Osiris Oyente Securify Slither Smartcheck Total
Access Control 0/19 0% 0/19 0% 4/19 21% 4/19 21% 0/19 0% 0/19 0% 0/19 0% 4/19 21% 2/19 11% 5/19 26%
Arithmetic 0/22 0% 0/22 0% 4/22 18% 15/22 68% 11/22 50% 12/22 55% 0/22 0% 0/22 0% 1/22 5% 19/22 86%
Denial Service 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/7 0% 0/ 7 0%
Front Running 0/7 0% 0/7 0% 0/7 0% 2/7 29% 0/7 0% 0/7 0% 2/7 29% 0/7 0% 0/7 0% 2/ 7 29%
Reentrancy 0/8 0% 0/8 0% 2/8 25% 5/8 62% 5/8 62% 5/8 62% 5/8 62% 7/8 88% 5/8 62% 7/ 8 88%
Time Manipulation 0/5 0% 0/5 0% 1/5 20% 0/5 0% 0/5 0% 0/5 0% 0/5 0% 2/5 40% 1/5 20% 3/ 5 60%
Unchecked Low Calls 0/12 0% 0/12 0% 2/12 17% 5/12 42% 0/12 0% 0/12 0% 3/12 25% 4/12 33% 4/12 33% 9/12 75%
Other 2/3 67% 0/3 0% 0/3 0% 0/3 0% 0/3 0% 0/3 0% 0/3 0% 3/3 100% 0/3 0% 3/ 3 100%
Total 2/115 2% 0/115 0% 13/115 11% 31/115 27% 16/115 14% 17/115 15% 10/115 9% 20/115 17% 13/115 11% 48/115 42%

Nb Detected Vulnerabilities

Category Honeybadger Maian Manticore Mythril Osiris Oyente Securify Slither Smartcheck Total
Access Control 0 10 28 24 0 0 6 20 3 91
Arithmetic 0 0 11 92 62 69 0 0 23 257
Denial Service 0 0 0 0 27 11 0 2 19 59
Front Running 0 0 0 21 0 0 55 0 0 76
Reentrancy 0 0 4 16 5 5 32 15 7 84
Time Manipulation 0 0 4 0 4 5 0 5 2 20
Unchecked Low Calls 0 0 4 30 0 0 21 13 14 82
Other 5 2 25 32 0 0 0 28 8 100
Total 5 12 76 215 98 90 114 83 76 769

Combine tools

Honeybadger Maian Manticore Mythril Osiris Oyente Securify Slither Smartcheck
Honeybadger 2/115 2% 15/115 13% 33/115 29% 18/115 16% 19/115 17% 12/115 10% 20/115 17% 15/115 13%
Maian 13/115 11% 31/115 27% 16/115 14% 17/115 15% 10/115 9% 20/115 17% 13/115 11%
Manticore 32/115 28% 26/115 23% 26/115 23% 19/115 17% 27/115 23% 20/115 17%
Mythril 33/115 29% 33/115 29% 31/115 27% 42/115 37% 33/115 29%
Osiris 22/115 19% 21/115 18% 31/115 27% 23/115 20%
Oyente 22/115 19% 32/115 28% 25/115 22%
Securify 25/115 22% 16/115 14%
Slither 25/115 22%
Smartcheck

SB Wild

Execution Time Stat

# Tool Avg. Execution Time Total Execution Time
1 Honeybadger 0:01:38 23 days, 13:40:00
2 Maian 0:05:16 49 days, 10:06:15
3 Manticore 0:24:28 184 days, 1:59:02
4 Mythril 0:01:24 46 days, 7:46:55
5 Osiris 0:00:34 18 days, 10:19:01
6 Oyente 0:00:30 16 days, 4:50:11
7 Securify 0:06:37 217 days, 22:46:26
8 Slither 0:00:05 2 days, 15:09:36
9 Smartcheck 0:00:10 5 days, 12:33:14

Total: 564 days, 3:10:39

Nb Detected Vulnerabilities

Category Honeybadger Maian Manticore Mythril Osiris Oyente Securify Slither Smartcheck Total
Access Control 0 0.00% 44 0.09% 47 0.10% 1076 2.27% 0 0.00% 2 0.00% 614 1.29% 2356 4.97% 384 0.81% 3801 8.01%
Arithmetic 1 0.00% 0 0.00% 102 0.21% 18515 39.02% 13922 29.34% 34306 72.30% 0 0.00% 0 0.00% 7430 15.66% 37597 79.23%
Denial Service 0 0.00% 0 0.00% 0 0.00% 0 0.00% 485 1.02% 880 1.85% 0 0.00% 2555 5.38% 11621 24.49% 12419 26.17%
Front Running 0 0.00% 0 0.00% 0 0.00% 2015 4.25% 0 0.00% 0 0.00% 7217 15.21% 0 0.00% 0 0.00% 8161 17.20%
Reentrancy 19 0.04% 0 0.00% 2 0.00% 8454 17.82% 496 1.05% 308 0.65% 2033 4.28% 8764 18.47% 847 1.78% 14747 31.08%
Time Manipulation 0 0.00% 0 0.00% 90 0.19% 0 0.00% 1470 3.10% 1452 3.06% 0 0.00% 1988 4.19% 68 0.14% 4069 8.58%
Unchecked Low Calls 0 0.00% 0 0.00% 4 0.01% 443 0.93% 0 0.00% 0 0.00% 592 1.25% 12199 25.71% 2867 6.04% 14656 30.89%
Other 26 0.05% 135 0.28% 1032 2.17% 11126 23.45% 0 0.00% 0 0.00% 561 1.18% 9133 19.25% 14113 29.74% 28355 59.76%
Total 46 0.10% 179 0.38% 1203 2.54% 22994 48.46% 14665 30.91% 34764 73.26% 8781 18.51% 22269 46.93% 24906 52.49% 44589 93.97%
You can’t perform that action at this time.