This experiment tutorial help you to finish the evaluation described by this paper.
It's similar with the evaluation section Chucky: Exposing Missing Checks in Source Code for Vulnerability Discovery, but the ROC curves are generated by the middle result(The rank lists of similar functions).
To do the experiment, you should do the following steps:
- Generate the code database.
- Modify the code.
- Run the automatic script.
The database can be generated by joern(2.0-3.0) according to the method Fabian described in Chucky paper. That is, patch the vulnerability as the original version, then remove one check in one function from the original versions in a round robin fashion to generate such many code versions and then use joern to generate the code graph database for each vulnerable version. The version and the respective vulnerability number are listed below.
Project | Vulnerability | Declaration Type | Symbol | TYPE | #With Check | #Symbol Users | #F | LOC |
---|---|---|---|---|---|---|---|---|
firefox-4.0(/js) | CVE-2010-3183 | uintN | argc | parameter | 10 | 557 | 5649 | 372450 |
linux-2.6.34.13(/fs) | CVE-2010-2071 | struct dentry* | dentry | parameter | 8 | 1104 | 19178 | 955943 |
libpng-1.2.44 | CVE-2011-2692 | png_uint_32 | length | parameter | 19 | 29 | 473 | 40255 |
libtiff-3.9.4 | CVE-2010-2067 | TIFFDirEntry* | dir | parameter | 9 | 75 | 609 | 332762 |
pidgin-2.7.3(/libpurple) | CVE-2010-3711 | purple_base64_decode | callee | 18 | 30 | 7390 | 332762 |
Remove the # symbol at the head of the two lines in the
try
block of function analyze():#for n in nearestNeighbors: # print str(n)+"\t"+n.location()
Comment out all the following code in
try
block(that means we just print the neighborhood selection result).Define the environment variable
$NEO4J_HOME
to point it to your neo4j program directory.Change the variable
cfgfile
in the script fileneighbor
to the absolute location of the configuration fileneo4j-server.properties
.change the variable
line
in neighbor to the line of variableorg.neo4j.server.database.location
in the configuration fileconf/neo4j-server.properties
of your Neo4j database.line=11
- Change the value of the
dbpath
to the location of all of your database.Note that the directory must be organized as$dbpath/$projname/$funcname/.joernIndex
. The projenames and funcnames must be equal to the names listed in the script fileneighbor
.
$ cd chucky-ng/chucky $ neighbor $ python ROC.py
The shell script neighbor
dump the result of KNN algorithm to the current file directory,
then the ROC.py
read the directory and generate the points in the directory named ROC
.
- The directory
neighbors
output by script neighbor will hold the hierarchy$neighbors/$projname/$function_name
, for example,neighbors/libpng/png_handle_cHRM
. - The final ROC points will be generated in file
ROC/$projname-neighbors_ROC
, for example,ROC/libpng-neighbors_ROC
).
At last, you can import these files of ROC point lists into drawing program to plot the diagram.
Here is the detail information about the 64 function for evaluation.
Firefox-4.0
Order | Function | Location |
---|---|---|
1 | array_concat | js/src/jsarray.cpp |
2 | array_extra | js/src/jsarray.cpp |
3 | array_indexOfHelper | js/src/jsarray.cpp |
4 | array_slice | js/src/jsarray.cpp |
5 | array_splice | js/src/jsarray.cpp |
6 | array_unshift | js/src/jsarray.cpp |
7 | js::array_sort | js/src/jsarray.cpp |
8 | LookupGetterOrSetter | js/src/xpconnect/src/xpcquickstubs.cpp |
9 | DefineGetterOrSetter | js/src/xpconnect/src/xpcquickstubs.cpp |
10 | PropertyOpForwarder | js/src/xpconnect/src/xpcquickstubs.cpp |
linux-2.6.34.13
Order | Function | Location |
---|---|---|
1 | btrfs_xattr_acl_set | fs/btrfs/acl.c |
2 | jffs2_acl_setxattr | fs/jffs2/acl.c |
3 | ext2_xattr_set_acl | fs/ext2/acl.c |
4 | ext3_xattr_set_acl | fs/ext3/acl.c |
5 | ext4_xattr_set_acl | fs/ext4/acl.c |
6 | ocfs2_xattr_acl_set | fs/ocfs2/acl.c |
7 | generic_acl_set | fs/generic_acl.c |
8 | posix_acl_set | fs/reiserfs/xattr_acl.c |
libpng-1.2.44
Order | Function | Location |
---|---|---|
1 | png_handle_Bkgd | pngrutil.c |
2 | png_handle_cHRM | pngrutil.c |
3 | png_handle_gAMA | pngrutil.c |
4 | png_handle_iCCP | pngrutil.c |
5 | png_handle_IEND | pngrutil.c |
6 | png_handle_IHDR | pngrutil.c |
7 | png_handle_iTXt | pngrutil.c |
8 | png_handle_oFFs | pngrutil.c |
9 | png_handle_pHYs | pngrutil.c |
10 | png_handle_PLTE | pngrutil.c |
11 | png_handle_sBIT | pngrutil.c |
12 | png_handle_sCAL | pngrutil.c |
13 | png_handle_sPLT | pngrutil.c |
14 | png_handle_sRGB | pngrutil.c |
15 | png_handle_tEXt | pngrutil.c |
16 | png_handle_tIME | pngrutil.c |
17 | png_handle_tRNS | pngrutil.c |
18 | png_handle_unknown | pngrutil.c |
19 | png_handle_zTXt | pngrutil.c |
tiff-3.9.4
Order | Function | Location |
---|---|---|
1 | TIFFFetchByteArray | libtiff/tif_dirread.c |
2 | TIFFFetchLongArray | libtiff/tif_dirread.c |
3 | TIFFFetchPerSampleAnys | libtiff/tif_dirread.c |
4 | TIFFFetchPerSampleLongs | libtiff/tif_dirread.c |
5 | TIFFFetchPerSampleShorts | libtiff/tif_dirread.c |
6 | TIFFFetchShortArray | libtiff/tif_dirread.c |
7 | TIFFFetchShortPair | libtiff/tif_dirread.c |
8 | TIFFFetchString | libtiff/tif_dirread.c |
9 | TIFFFetchSubjectDistance | libtiff/tif_dirread.c |
Pidgin-2.7.3
Order | Function | Location |
---|---|---|
1 | digest_md5_handle_chanllenge | lipurple/protocols/jabber/auth_digest_md5.c |
2 | do_buddy_avatar_update_data | lipurple/protocols/jabber/useravatar.c |
3 | got_sessionreq | lipurple/protocols/msn/slp.c |
4 | jabber_data_create_from_xml | lipurple/protocols/jabber/data.c |
5 | jabber_ibb_parse | lipurple/protocols/jabber/ibb.c |
6 | jabber_scram_feed_parser | lipurple/protocols/jabber/auth_scram.c |
7 | jabber_vcard_parse | lipurple/protocols/jabber/buddy.c |
8 | jabber_vcard_parse_avatar | lipurple/protocols/jabber/presence.c |
9 | jabber_vacard_save_mine | lipurple/protocols/jabber/buddy.c |
10 | msim_msg_get_binary_from_element | lipurple/protocols/myspace/message.c |
11 | msn_oim_report_to_user | lipurple/protocols/msn/oim.c |
12 | msn_switchboard_shoe_ink | lipurple/protocols/msn/switchboard.c |
13 | purple_mime_decode_field | lipurple/util.c |
14 | purple_ntlm_parse_type2 | lipurple/ntlm.c |
15 | scram_handle_challenge | lipurple/protocols/jabber/auth_scram.c |
16 | scram_handle_success | lipurple/protocols/jabber/auth_scram.c |
17 | yahoo_process_p2p | lipurple/protocols/yahoo/libymsg.c |
18 | yahoo_process_status | lipurple/protocols/yahoo/libymsg.c |