remove string.c from the package.

[Niraj-Kamdar]

Removed string.c from the package and also removed usage of it from setup script and string module.

[codecov-io]

Codecov Report

Merging #655 into master will increase coverage by 0.14%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #655      +/-   ##
==========================================
+ Coverage   78.50%   78.64%   +0.14%     
==========================================
  Files          63       63              
  Lines        2549     2538      -11     
  Branches      498      500       +2     
==========================================
- Hits         2001     1996       -5     
+ Misses        406      402       -4     
+ Partials      142      140       -2     

Flag	Coverage Δ
#longtests	78.64% <100.00%> (+0.14%)	⬆️

Impacted Files	Coverage Δ
cve_bin_tool/strings.py	100.00% <100.00%> (+20.00%)	⬆️
test/test_strings.py	88.00% <100.00%> (+1.88%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d4fdc16...36fac32. Read the comment docs.

[Niraj-Kamdar]

Surprisingly, new string parser is as fast as string command during first run and in subsequent run new python parser is faster than strings command.

test_binutils
strings command: 0.0014863650000052075
string.parse: 0.0014164240000127393
#########

test_curl
strings command: 0.0018305610000197703
string.parse: 0.0010337909999407202
#########

test_kerberos
strings command: 0.0015554930000689637
string.parse: 0.0011313310000105048
#########

Here is the script I used for benchmarking

    def _parse_test(self, filename):
        """Helper function to parse a binary file and check whether
        the given string is in the parsed result"""
        self.strings.filename = os.path.join(BINARIES_PATH, filename)
        with tempfile.TemporaryFile() as f:
            t1 = perf_counter()
            subprocess.call(["strings", self.strings.filename], stdout=f)
            binutils_strings = f.readlines()
            t2 = perf_counter()
            print(f"strings command: {t2 - t1}")
            t1 = perf_counter()
            ours = self.strings.parse().split("\n")
            t2 = perf_counter()
            print(f"string.parse: {t2 - t1}")
            for theirs in binutils_strings:
                self.assertIn(theirs.decode("utf-8"), ours)

Should we remove subprocess?

[pdxjohnny]

dist/cve_bin_tool-1.0-py3.7.egg should be removed from this.

Looking good!

[Niraj-Kamdar]

subprocess is slow because of the overhead of system call. current python solution is very fast due to various improvement like set to lookup printable character, use of join and list instead of string concatenation.

[pdxjohnny]

dist/cve_bin_tool-1.0-py3.7.egg is still here. Also let's make sure we have performance numbers for the final implementation before we hit merge on this.

[Niraj-Kamdar]

python implementation is taking half the time of subprocess string

subprocess string: 0.004477346000385296
python string: 0.002734930999849894

subprocess string: 0.005265122999844607
python string: 0.002693021000141016

[pdxjohnny]

Okay looks good. It is faster than pstring but I'm not sure if it's really faster than strings. When benchmarking it's always good to run something for multiple iterations, rather than just one pass.

For example here are the numbers I saw with 10,000 runs.

subprocess string: 18.777919821441174
pstring string: 62.875923331826925
python string: 22.99539073370397

Let's back out the benchmarking code and then we'll merge this. Thanks!

[pdxjohnny]

Thanks! Nice work!