Skip to content
Results from Cyber Test Lab
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
centos/7
fedora
rhel/7
.gitignore
LICENSE
README.md
afCc.png
centos7_scores.json
comparison_fedora27_rhel7.png
fedora27_by_package.png
fedora27_scores.json
newline.py
rhel7_by_package.png
rhel7_scores.json

README.md

ctl-results

Results of Cyber Test Lab analysis

Caveat hackor

We've now posted a complete analysis run of the Fedora 27 packages. The CTL code is beta.

We have the two following convictions about our results:

  1. They are wrong
  2. They will get better

This will be an iterative process. We're still immature, but this project is being actively improved.

Data points collected

The Cyber Test Lab collects the following data for each ELF binary:

  • ASLR enabled or disabled (position-independant code / executable)
  • Read-only relocations (RELRO) enabled or disabled
  • Stack protected or not
  • Presence or absence of source fortified functions
  • Immediate binding or lazy resolution
  • Cyclomatic complexity
  • Cycle cost

Fedora 27 CTL scoring

Here's a graph of the scored Fedora 27 packages, and the raw data that was generated by cyber-test-lab.

scores by package

We're still dialing in our scoring technique. Expect this to change a lot in the coming days.

Fedora 27 cyclomatic complexity analysis

Here's a deeper sample of some of the data collected by this project.

afcc

This graph shows the cyclomatic complexity values for all Fedora 27 binaries (for which the CTL code cooperated, we have a bug there). It was generated with this gist. The histogram represents a probability density function indiciating the likelihood that a complexity value will fall in that range. We see that most Fedora 27 binaries have a cyclomatic complexity of less than 100, with a mean of about 35. Some binaries are much more complex, the highest in this dataset being 780. Note that we currently time out r2 after 10 minutes of trying to analyze the binary (aa command). As a result, there are many cyclomatic complexity values higher than 780 in the Fedora 27 repositories. We're tracking a bug for that too.

So what does this mean? We won't know for sure until we graduate from this static analysis to dynamic analysis in the form of fuzzing. (For an introduction to fuzzing, the Sully manual is excellent.) The theory is that more complex binaries will be more likely to crash while being fuzzed. Crashes can then be analyzed to find new 0-day vulnerabilities. So a smaller cyclomatic complexity value is better, and hopefully more "secure."

Check back for udates and more analysis.

Sample results data

When the tool works right, here's a sample result, taken from coreutils.

{
    "results": {
    "_comment": "truncated",
        "usr/bin/ls": {
            "find-libc-functions": [
                "GI___obstack_vprintf",
                "GI___vasprintf",
                "GI___vdprintf",
                "GI___vfprintf",
                "GI___vfwprintf",
                "GI___vsnprintf",
                "GI___vsprintf",
                "GI___vswprintf",
                "GI___vsyslog",
                "__longjmp",
                "_fprintf",
                "_printf",
                "_snprintf",
                "_sprintf",
                "_vfprintf",
                "_vprintf",
                "_vsnprintf",
                "_vsprintf",
                "asprintf",
                "confstr",
                "dprintf",
                "explicit_bzero",
                "fdelt",
                "fgets",
                "fgets_unlocked",
                "fgetws",
                "fgetws_unlocked",
                "fprintf",
                "fread",
                "fread_unlocked",
                "fwprintf",
                "getcwd",
                "getdomainname",
                "getgroups",
                "gethostname",
                "getlogin_r",
                "gets",
                "getwd",
                "longjmp",
                "mbsnrtowcs",
                "mbsrtowcs",
                "mbstowcs",
                "memcpy",
                "memmove",
                "mempcpy",
                "memset",
                "obstack_printf",
                "obstack_vprintf",
                "poll",
                "ppoll",
                "pread64",
                "pread",
                "printf",
                "ptsname_r",
                "read",
                "readlink",
                "readlinkat",
                "realpath",
                "recv",
                "recvfrom",
                "snprintf",
                "sprintf",
                "stpcpy",
                "stpncpy",
                "strcat",
                "strcpy",
                "strncat",
                "strncpy",
                "swprintf",
                "syslog",
                "ttyname_r",
                "vasprintf",
                "vdprintf",
                "vfprintf",
                "vfwprintf",
                "vprintf",
                "vsnprintf",
                "vsprintf",
                "vswprintf",
                "vsyslog",
                "vwprintf",
                "wcpcpy",
                "wcpncpy",
                "wcrtomb",
                "wcscat",
                "wcscpy",
                "wcsncat",
                "wcsncpy",
                "wcsnrtombs",
                "wcsrtombs",
                "wcstombs",
                "wctomb",
                "wmemcpy",
                "wmemmove",
                "wmempcpy",
                "wmemset",
                "wprintf"
            ],
            "filename": "usr/bin/ls",
            "complexity": {
                "r2aa": {
                    "afCc": -1,
                    "_comment": "this appears to be a bug, tracked at https://github.com/redteam-project/cyber-test-lab/issues/12",
                    "afC": 69
                }
            },
            "hardening-check": {
                " Read-only relocations": "yes",
                " Position Independent Executable": "yes",
                " Stack protected": "yes",
                " Fortify Source functions": "yes (some protected functions found)",
                " Immediate binding": "yes"
            },
            "report-functions": [
                "__xstat",
                "__uflow",
                "fdopen",
                "__fprintf_chk",
                "gettimeofday",
                "getgrgid",
                "calloc",
                "fscanf",
                "__freading",
                "strtoul",
                "snprintf",
                "timegm",
                "exit",
                "sigaction",
                "__fxstat",
                "strcpy",
                "iswcntrl",
                "__cxa_atexit",
                "fwrite",
                "cap_get_file",
                "strcoll",
                "__ctype_get_mb_cur_max",
                "getpwuid",
                "fputs_unlocked",
                "wcwidth",
                "ioctl",
                "_ITM_registerTMCloneTable",
                "__libc_start_main",
                "__overflow",
                "free",
                "mbsinit",
                "cap_to_text",
                "stpncpy",
                "strcmp",
                "__assert_fail",
                "fclose",
                "sigprocmask",
                "setenv",
                "fwrite_unlocked",
                "close",
                "strncmp",
                "localeconv",
                "opendir",
                "open",
                "fileno",
                "__cxa_finalize",
                "getxattr",
                "wcstombs",
                "isatty",
                "iswprint",
                "getenv",
                "strchr",
                "__ctype_toupper_loc",
                "strrchr",
                "memset",
                "memcpy",
                "closedir",
                "__snprintf_chk",
                "getfilecon",
                "malloc",
                "freecon",
                "sigismember",
                "__stack_chk_fail",
                "__errno_location",
                "__gmon_start__",
                "readlink",
                "tcgetpgrp",
                "_ITM_deregisterTMCloneTable",
                "nl_langinfo",
                "fflush",
                "cap_free",
                "ungetc",
                "gmtime_r",
                "lseek",
                "memcmp",
                "setlocale",
                "mbrtowc",
                "fnmatch",
                "__sprintf_chk",
                "textdomain",
                "wcswidth",
                "_setjmp",
                "strftime",
                "strlen",
                "__ctype_b_loc",
                "bindtextdomain",
                "raise",
                "__lxstat",
                "__strtoul_internal",
                "sigaddset",
                "_exit",
                "clock_gettime",
                "__fpending",
                "lgetfilecon",
                "tzset",
                "__printf_chk",
                "dirfd",
                "localtime_r",
                "getgrnam",
                "getpwnam",
                "realloc",
                "abort",
                "unsetenv",
                "memmove",
                "mbstowcs",
                "dcgettext",
                "signal",
                "__ctype_tolower_loc",
                "fgetfilecon",
                "fseeko",
                "sigemptyset",
                "readdir",
                "mempcpy",
                "error",
                "__memcpy_chk"
            ],
            "rpm": "coreutils-8.27-16.fc27.x86_64.rpm"
        },
        "usr/bin/pwd": {
            "find-libc-functions": [
                "GI___obstack_vprintf",
                "GI___vasprintf",
                "GI___vdprintf",
                "GI___vfprintf",
                "GI___vfwprintf",
                "GI___vsnprintf",
                "GI___vsprintf",
                "GI___vswprintf",
                "GI___vsyslog",
                "__longjmp",
                "_fprintf",
                "_printf",
                "_snprintf",
                "_sprintf",
                "_vfprintf",
                "_vprintf",
                "_vsnprintf",
                "_vsprintf",
                "asprintf",
                "confstr",
                "dprintf",
                "explicit_bzero",
                "fdelt",
                "fgets",
                "fgets_unlocked",
                "fgetws",
                "fgetws_unlocked",
                "fprintf",
                "fread",
                "fread_unlocked",
                "fwprintf",
                "getcwd",
                "getdomainname",
                "getgroups",
                "gethostname",
                "getlogin_r",
                "gets",
                "getwd",
                "longjmp",
                "mbsnrtowcs",
                "mbsrtowcs",
                "mbstowcs",
                "memcpy",
                "memmove",
                "mempcpy",
                "memset",
                "obstack_printf",
                "obstack_vprintf",
                "poll",
                "ppoll",
                "pread64",
                "pread",
                "printf",
                "ptsname_r",
                "read",
                "readlink",
                "readlinkat",
                "realpath",
                "recv",
                "recvfrom",
                "snprintf",
                "sprintf",
                "stpcpy",
                "stpncpy",
                "strcat",
                "strcpy",
                "strncat",
                "strncpy",
                "swprintf",
                "syslog",
                "ttyname_r",
                "vasprintf",
                "vdprintf",
                "vfprintf",
                "vfwprintf",
                "vprintf",
                "vsnprintf",
                "vsprintf",
                "vswprintf",
                "vsyslog",
                "vwprintf",
                "wcpcpy",
                "wcpncpy",
                "wcrtomb",
                "wcscat",
                "wcscpy",
                "wcsncat",
                "wcsncpy",
                "wcsnrtombs",
                "wcsrtombs",
                "wcstombs",
                "wctomb",
                "wmemcpy",
                "wmemmove",
                "wmempcpy",
                "wmemset",
                "wprintf"
            ],
            "filename": "usr/bin/pwd",
            "complexity": {
                "r2aa": {
                    "afCc": 9,
                    "afC": 198
                }
            },
            "hardening-check": {
                " Read-only relocations": "yes",
                " Position Independent Executable": "yes",
                " Stack protected": "yes",
                " Fortify Source functions": "yes (some protected functions found)",
                " Immediate binding": "yes"
            },
            "report-functions": [
                "getenv",
                "textdomain",
                "lseek",
                "__printf_chk",
                "strlen",
                "calloc",
                "__ctype_b_loc",
                "strncmp",
                "closedir",
                "mbsinit",
                "error",
                "__fxstat",
                "bindtextdomain",
                "open",
                "__xstat",
                "fileno",
                "free",
                "_exit",
                "fputs_unlocked",
                "__errno_location",
                "strcmp",
                "readdir",
                "iswprint",
                "setlocale",
                "close",
                "_ITM_registerTMCloneTable",
                "memcmp",
                "__uflow",
                "malloc",
                "strrchr",
                "mbrtowc",
                "strstr",
                "fscanf",
                "__fpending",
                "nl_langinfo",
                "memcpy",
                "fclose",
                "__fprintf_chk",
                "__libc_start_main",
                "realloc",
                "abort",
                "ungetc",
                "getopt_long",
                "__gmon_start__",
                "exit",
                "_ITM_deregisterTMCloneTable",
                "__cxa_finalize",
                "__ctype_get_mb_cur_max",
                "puts",
                "chdir",
                "__freading",
                "dirfd",
                "fwrite",
                "getcwd",
                "fchdir",
                "memset",
                "__lxstat",
                "fdopen",
                "fseeko",
                "fflush",
                "dcgettext",
                "__cxa_atexit"
            ],
            "rpm": "coreutils-8.27-16.fc27.x86_64.rpm"
        },
    "_comment": "truncated",
    "metadata": {
        "spec_data": {
            "Group": " System Environment/Base",
            "Name": " coreutils",
            "License": " GPLv3+",
            "URL": " https://www.gnu.org/software/coreutils/",
            "Relocations": " (not relocatable)",
            "Install Date": " (not installed)",
            "Build Host": " buildvm-05.phx2.fedoraproject.org",
            "Description": "\nThese are the GNU core utilities.  This package is the combination of\nthe old GNU fileutils, sh-utils, and textutils packages.\n",
            "Build Date": " Tue 22 Aug 2017 09:21:46 AM EDT",
            "Source RPM": " coreutils-8.27-16.fc27.src.rpm",
            "Version": " 8.27",
            "Architecture": " x86_64",
            "Signature": " RSA/SHA256, Tue 22 Aug 2017 09:36:25 AM EDT, Key ID f55e7430f5282ee4",
            "Release": " 16.fc27",
            "Vendor": " Fedora Project",
            "Packager": " Fedora Project",
            "Summary": " A set of basic GNU tools commonly used in shell scripts",
            "Size": " 5773848"
        }
    }
}
You can’t perform that action at this time.