New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use C99, incorporate jody_hash to replace MD5, general cleanups #10

Closed
wants to merge 13 commits into
base: master
from

Conversation

Projects
None yet
1 participant
@jbruchon

jbruchon commented Dec 29, 2014

No description provided.

jbruchon added some commits Dec 29, 2014

Use static/inline where appropriate, switch to C99
fdupes contains many extremely small functions that are called in
few places and can benefit from explicit inlining. Functions
should also be declared "static" where appropriate to allow for
better compiler optimization. Added "static" and "inline" where
appropriate.

Removed "long long" references, replacing with proper C99 types.
Changed Makefile to use C99, higher optimization, and pedantic
compiler warnings. Added headers needed for C99 types.
Update jody_hash license to allow use under the fdupes license terms
Signed-off-by: Jody Bruchon <jody@jodybruchon.com>
Fully incorporate Jody Bruchon's block hash to replace MD5
MD5 is excellent at making unique hashes, but it is also fairly
computationally expensive. In fdupes MD5 hashes are passed around
as text strings which is also inefficient. I have replaced the MD5
functionality with my own block hash algorithm I wrote for another
project that deals in large amounts of disk data and allowed for
switching back to MD5 with a simple Makefile edit.

Performance testing was performed with Valgrind's cachegrind tool
on a linux-3.16.2 source directory with compiled objects inside.
I diff'd the cg_annotate output for MD5 and jody_hash runs and
moved the output around to show the differences between comparable
parts of the MD5 and jody_hash versions. The final output is at
the end of this message.

On my Phenom II X6 machine, taking a "best of three" benchmark
with the time command shows a small improvement over MD5. If tests
were performed on a less powerful (i.e. Celeron, Atom, AMD E1)
machine, this could represent a significant improvement.

The downside is that the chance for slightly more false positives
may cause unnecessary comparisons of very large files to happen.
If further testing shows a performance impact for large files, I
will look for further optimizations.

Signed-off-by: Jody Bruchon <jody@jodybruchon.com>

 --------------------------------------------------------------------------------
            Ir    I1mr  ILmr          Dr      D1mr    DLmr          Dw      D1mw    DLmw
 --------------------------------------------------------------------------------
-2,664,677,525 653,435 1,482 388,569,879 3,944,381 392,629 253,421,670 1,055,624 235,518  PROGRAM TOTALS
+  716,015,138 202,789 1,397 132,964,117 3,172,679 362,666  73,791,299   578,587 222,806  PROGRAM TOTALS

 --------------------------------------------------------------------------------
-           Ir    I1mr ILmr          Dr      D1mr    DLmr         Dw    D1mw    DLmw  file:function
 --------------------------------------------------------------------------------
-1,663,169,202      37   37 150,975,828   218,053     128 58,442,256     151       0  /home/user/fdupes/md5/md5.c:md5_process
-   24,542,627       8    8   1,306,072       315       1  3,666,644  11,055       1  /home/user/fdupes/md5/md5.c:md5_append
-   15,011,398       3    3   1,338,596         0       0  1,338,596     172       0  /home/user/fdupes/md5/md5.c:md5_finish
+  269,985,544       4    4  19,274,079    91,117      71    468,315       0       0  /home/user/fdupes/jody_hash.c:jody_block_hash
-  302,946,571   1,539   74  86,561,561   369,712   1,132 65,623,703 158,161      90  ???:vfprintf
+   61,846,611   2,098   67  16,766,472   111,102   1,120 14,226,945  99,531      90  ???:vfprintf
-  224,234,839 405,128  427  50,903,971   457,883  48,427 54,705,840 515,314 208,888  ???:???
+  131,477,454 113,207  414  33,967,160   364,717  32,819 24,632,495 369,868 196,933  ???:???
-  101,397,914   1,006    6  25,050,486       381       1 20,718,928 123,926       4  ???:_IO_default_xsputn
+   38,236,234       4    4   7,828,678         1       1  6,655,204     235       4  ???:_IO_default_xsputn
-   61,718,088      13   13  23,115,468 2,237,516  38,026 10,889,175  17,747     189  /home/user/fdupes/fdupes.c:checkmatch
+   58,781,569      12   12  22,057,944 2,038,231  32,210 10,324,989  13,331     184  /home/user/fdupes/fdupes.c:checkmatch
-   40,961,036       5    5   4,136,075       976       6          0       0       0  ???:strchrnul
+   10,364,556      40    6   1,076,427     2,315       5          0       0       0  ???:strchrnul
-   34,390,520       1    1   3,439,052         0       0          0       0       0  ???:__xstat
+   32,913,920       1    1   3,291,392         0       0          0       0       0  ???:__xstat
-   29,066,656       3    3   5,354,384         0       0  7,649,120  44,909       1  ???:vsprintf
-   13,003,504  40,985    3     764,912     3,333       0  6,884,208       0       0  ???:sprintf
+    2,160,309   1,299    3     127,077         0       0  1,143,693     600       1  ???:fprintf
-   17,047,840       2    2   4,514,817         0       0  4,465,514   6,447       0  ???:_IO_setb
+    2,514,818       2    2     690,341         0       0    641,032   2,204       7  ???:_IO_setb
-   13,864,030  95,977    4   1,577,631         0       0          0       0       0  ???:rawmemchr
-   13,594,768       0    0   3,021,392    60,828  32,816  3,021,392  46,290       1  /usr/include/sys/stat.h:checkmatch
+   12,930,346       2    2   2,873,744    60,446  31,583  2,873,744  20,469       1  /usr/include/sys/stat.h:checkmatch
-   12,121,092      10    7   1,099,561    12,073       0  2,064,157  18,082       1  /home/user/fdupes/fdupes.c:getcrcsignatureuntil
+    2,815,639       4    4     334,649     4,605       4    542,789   1,372       0  /home/user/fdupes/fdupes.c:getcrcsignatureuntil
-   12,053,821       5    5   3,527,934    13,114       4    881,984   1,287       0  ???:malloc
+   12,053,944       5    5   3,527,970    30,504       4    881,993   2,319       9  ???:malloc
-    7,336,581      10   10   1,729,568     3,841       2    909,779       0       0  ???:_IO_file_xsputn
+    7,336,581      10   10   1,729,568       855       2    909,779       0       0  ???:_IO_file_xsputn
-    6,573,597      24   15     931,396    10,816   4,084          0       0       0  ???:strlen
+    4,948,159      12   12     692,361    11,312   4,084          0       0       0  ???:strlen
-    5,876,267      11   11     932,700    65,538       8  1,323,051   7,847   7,431  /home/user/fdupes/fdupes.c:grokdir
+    5,876,267      10   10     932,700     5,266       8  1,323,051   7,902   7,431  /home/user/fdupes/fdupes.c:grokdir
-    4,936,284       8    8   1,874,262     3,738       0    788,848       1       0  ???:_IO_un_link
+    4,936,908       8    8   1,874,493       242       0    788,944       0       0  ???:_IO_un_link
-    4,921,659  19,190    3   1,101,303    85,814  61,835          0       0       0  ???:free
+    4,921,704  19,193    3   1,101,312    72,187  57,067          0       0       0  ???:free
-    4,614,084       3    3   1,025,352         0       0    640,845  22,333       3  ???:_IO_file_write
+    4,614,084       3    3   1,025,352         0       0    640,845   4,215       3  ???:_IO_file_write
-    4,585,179       9    9   1,626,999    10,691       0    887,454       1       0  ???:_IO_link_in
+    4,585,737       9    9   1,627,197     7,546       0    887,562       0       0  ???:_IO_link_in
-    3,648,422       7    7   1,331,181    17,704       0    493,030   3,337       0  ???:fclose
+    3,648,866       7    7   1,331,343     8,581       0    493,090       5       0  ???:fclose
-    3,451,210      72    9     542,333       141       0    394,424      78       0  ???:_IO_file_fopen
+    3,451,630     107   12     542,399       668       0    394,472      19       0  ???:_IO_file_fopen
-    3,438,632     786   26     930,502   261,826 172,079    521,146     343     105  /home/user/fdupes/fdupes.c:main
+    3,438,698      30   27     930,523   214,762 169,914    521,167     337     100  /home/user/fdupes/fdupes.c:main
-    3,023,165       4    4   1,258,625     4,604       6    478,378     199       0  ???:readdir
+    3,023,165       4    4   1,258,625    33,816       6    478,378     182       0  ???:readdir
-    3,015,102       5    5   1,057,711       471       0    389,683       0       0  ???:fread
+    3,095,346       5    5   1,085,945     1,118       0    400,085       0       0  ???:fread
-    2,839,384     761   11     901,854         0       0    387,968      39       0  ???:_IO_file_overflow
+    2,839,384  36,025   13     901,854         0       0    387,968      39       0  ???:_IO_file_overflow

@jbruchon jbruchon changed the title Use static/inline where appropriate, switch to C99 Use C99, incorporate jody_hash to replace MD5, general cleanups Jan 3, 2015

@jbruchon jbruchon closed this Jan 3, 2015

@jbruchon jbruchon reopened this Jan 3, 2015

jbruchon added some commits Jan 4, 2015

Add progress delay for ~2x speedup when progress indicators are on
Progress indicators were updating for every single operation that
was performed. This can be extremely slow because each update calls
fprintf() to update the display. It is even slower via non-local
terminals like SSH.

I have added a delay counter to space out progress updates in this
patch. The result is that performance is almost equal to the
performance obtained by running with progress indicators disabled.
Typical speedup seems to be approximately two times faster.

@jbruchon jbruchon closed this Jan 19, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment