Browse files

Change name to 'jdupes' and remove all 'fdupes' naming

Due to possible confusion between this program and 'fdupes' by
Adrian Lopez (from which this was originally derived), the program
is being renamed to 'jdupes' to remove any ambiguity permanently.

Due to the name change, versioning is being reset to v1.0
  • Loading branch information...
jbruchon committed Dec 24, 2015
1 parent 6c1d365 commit 67c69e25ccb9d8df0d27fb93c9bff79c0c48c6ff
Showing with 318 additions and 311 deletions.
  1. +2 −218 CHANGES
  2. +6 −17 INSTALL
  3. +10 −10 Makefile
  4. +222 −0 OLD_CHANGES
  5. +25 −18 README
  6. +6 −6 TODO
  7. +0 −26
  8. +28 −0
  9. +11 −8 fdupes-jody.1 → jdupes.1
  10. +6 −6 fdupes-jody.c → jdupes.c
  11. +2 −2 version.h
@@ -1,219 +1,3 @@
The following list, organized by fdupes version, documents changes
to fdupes. Every item on the list includes, inside square brackets,
a list of indentifiers referring to the people who contributed
that particular item. When more than one person is listed the person
who contributed the patch or idea appears first, followed by
those who've otherwise worked on that item. For a list of
contributors names and identifiers please see the CONTRIBUTORS file.

Changes from 2.1 to 2.2 [JLB]

- Changed fdupes-jody executable names and document texts to use the
full 'fdupes-jody' name instead of 'fdupes'. Moved copyrights and
contact information to reflect 'fdupes-jody' code responsibility.
This is primarily intended to keep fdupes-jody distinctly separate
from the fdupes by Adrian Lopez upon which it is based, and to make
certain that the correct person gets harassed if it breaks ;-)
- Added '-B/--dedupe' feature (not compiled in by default) which
sends file match lists directly to the kernel btrfs driver to do
block-level data de-duplication. Patch submitted by Sebastian
Schmidt <>. Thanks!
- Remove and replace some string function calls. Performance increase
shows on benchmarks but not significant in most cases.

Changes from 2.0.2 to 2.1 [JLB]

- Minor performance improvements to hashing and memory allocation code
- Added an experimental tree rebalancing function. It is compiled out
by default because benchmarks indicated either no improvement or a
slight slowdown compared to an unbalanced tree. To compile it in,
- Increased size of string_malloc pages from 64K to 256K since testing
shows a minor performance improvement with large file sets
- Made variable scope and type changes for a tiny performance boost

Changes from 2.0.1 to 2.0.2 [JLB]

- Removed redundant getfilestats() calls for a tiny speed boost
- Added a -D/--debug switch to show a set of statistic counters for
various parts of the fdupes algorithms. Can be used to determine what
fdupes is doing "under the hood" and give insight into why performance
may be slower or behave strangely on your data set. To enable it, use
DEBUG=1 with your make command, i.e. 'make DEBUG=1'
- Performance note: the fdupes algorithm uses a tree data structure and
becomes progressively slower as more files are processed and the tree
depth grows larger. As of this version, a rewrite of the core
algorithm is in progress which will remove this tree structure and
significantly improve performance on most large file sets.

Changes from 2.0 to 2.0.1 [JLB]

- Hard links were treated as identical in match checking but not in
match confirmation. This version fixes the problem, increasing speed
with file sets involving lots of hard links.
- A few minor efficiency improvements were performed

Changes from 1.51-jody5 to 2.0 [JLB]

- Bumped major version to 2.0 due to the number of behavioral changes
and improvements to the code, plus it looks less messy than the
hyphenated versioning
- Increased "chunk size" for better performance and to solve the disk
thrashing problem when checking two large files for a match
- When using the -H option, hard links now automatically match each
other without performing any file reads
- Changed primary memory allocator to string_alloc by Jody Bruchon
to improve performance over generic malloc/calloc
- Progress indicator now lists the number of duplicate pairs found in
addition to the usual file progress count and completion percentage
- Progress is updated more rapidly when full file comparisons happen
so users are less likely to think fdupes is "frozen"
- Floating point code was made optional and is removed by default
- A comparison script was added to check built fdupes behavior against
whatever fdupes is currently installed on the system
- Added "undocumented" -Q / --quick option which is not fully safe but can
be used to drastically reduce run time for large data sets if some risk
of data loss is acceptable to the user
- Added -O/--paramorder option to sort files by the order their parent
directory set was specified on the command line first. This makes it
possible to choose what directories' files get preserved over others
while using -d and -N together
- The file list loading progress indicator was revamped. Rather than a
simple "spinning pipe" indicator, it now shows the number of files and
directories scanned as well as which command line specified set the
scanning is currently happening in
- fdupes was enhanced to support more than 2 million files total by
changing from 'int' internal sizes to the maximum supported by the
platform being compiled for
- Hard link code was modified to be much safer; now any file is only
permanently deleted after a hard link succeeds
- Hard links on Windows (on supporting filesystems) are now supported
- Hashing code was optimized for a benchmarked 8.4% improvement in file
processing overhead (and much more under typical real-world conditions)
- Hard linking checks for more error conditions and output is much
clearer about what action was taken on any given file

Changes from 1.51-jody4-jkl1 to 1.51-jody5 [JLB]

- Less malloc()s so less memory usage and a slight speedup
- Change --order=name to an intelligent numerically correct sort
- Fixed bug where progress text was missing until first update
- Performance boost for small files (4KB or less) by not running
redundant hashes and comparisons
- Test files added for numerically correct sort ordering

Changes from 1.51-jody4 to 1.51-jody4-jkl1 [JKL]

- added `--xsize=SIZE' option: exclude files of size < SIZE
- updated Makefile: `PREFIX = /usr/local'
- updated README: Usage to reflect curent parameters

Changes from 1.51-jody2 to 1.51-jody4 [JLB]

- Add support for hard linking duplicates with -L switch
- Updated jody_hash algorithm with much lower collision rate
- Remove freopen() call that posed a portability problem
- Improved progress indicator behavior
- Many minor bug fixes

Changes from 1.51 to 1.51-jody2 [JLB]

- Switched to C99
- Replaced MD5 with Jody Bruchon's hash function
- Added a delay to progress indications for better performance
- Removed lots of unused code
- Ported fdupes to Microsoft Windows (with MinGW)

Changes from 1.50 to 1.51

- Added support for 64-bit file offsets on 32-bit systems.
- Using tty for interactive input instead of regular stdin. This is to
allow feeding filenames via stdin in future versions of fdupes without
breaking interactive deletion feature.
- Fixed some typos in --help.
- Turned C++ style comments into C style comments.

Changes from 1.40 to 1.50-PR2

- Fixed memory leak. [JB]
- Added "--summarize" option. [AL]
- Added "--recurse:" selective recursion option. [AL]
- Added "--noprompt" option for totally automated deletion of
duplicate files.
- Now sorts duplicates (old to new) for consistent order when
listing or deleteing duplicate files.
- Now tests for early matching of files, which should help speed up
the matching process when large files are involved.
- Added warning whenever a file cannot be deleted. [CHL, AL]
- Fixed bug where some files would not be closed after failure. [AL]
- Fixed bug where confirmmatch() function wouldn't always deal
properly with zero-length files. [AL]
- Fixed bug where progress indicator would not be cleared
when no files were found. [AL]
- Removed experimental red-black tree code (it was slower on
my system than the default code). [AL]
- Modified md5/md5.c to avoid compiler warning. [CHL]
- Changes to fdupes.c for compilation under platforms where
getopt_long is unavailable. [LR, AL]
- Changes to help text for clarity. [AL]
- Various changes and improvements to Makefile. [PB, AL]

Changes from 1.31 to 1.40

- Added option to omit the first file in each group
of matches. [LM, AL]
- Added escaping of filenames containing spaces when
sameline option is specified. [AL]
- Changed version indicator format from "fdupes version X.Y"
to the simpler "fdupes X.Y". [AL]
- Changed ordering of options appearing in the help
text (--help), manpage, and README file. [AL]

Changes from 1.30 to 1.31

- Added interactive option to preserve all files during
delete procedure (something similar was already in
place, but now it's official). [AL]
- Updated delete procedure prompt format. [AL]
- Cosmetic code changes. [AL]

Changes from 1.20 to 1.30

- Added size option to display size of duplicates. [LB, AL]
- Added missing typecast for proper compilation under g++. [LB]
- Better handling of errors occurring during retrieval
of a file's signature. [KK, AL]
- No longer displays an error message when specified
directories contain no files. [AL]
- Added red-black tree structure (experimental compile-time
option, disabled by default). [AL]

Changes from 1.12 to 1.20

- Fixed bug where program would crash when files being
scanned were named pipes or sockets. [FD]
- Fix against security risk resulting from the use of a
temporary file to store md5sum output. [FD, AL]
- Using an external md5sum program is now optional. Started
using L. Peter Deutsh's MD5 library instead. [FD, AL]
- Added hardlinks option to distinguish between hard links
and actual duplicate files. [FD, AL]
- Added noempty option to exclude zero-length files
from consideration [AL]

Changes from 1.11 to 1.12

- Improved handling of extremely long input on preserve
prompt (delete option). [SSD, AL]

Changes from 1.1 to 1.11

- Started checking file sizes before signatures
for better performance. [AB, AL]
- Added fdupes manpage. [AB, AL]

Changes from 1.0 to 1.1

- Added delete option for semi-automatic deletion
of duplicate files. [AL]
jdupes 1.0

First release. For changes before the 'jdupes' name change, see OLD_CHANGES
@@ -1,4 +1,4 @@
Installing fdupes
Installing jdupes
To install the program, issue the following commands:

@@ -31,29 +31,18 @@ CONSIDER_IMBALANCE * Change tree rebalance to analyze weights first
* These options may slow down the program somewhat and are off by
default. Do not enable them unless you are experimenting.

OLD UPGRADE NOTE: When upgrading from a version prior to 1.2, it should
be noted that the default installation directory for the fdupes man
page has changed from "/usr/man" to "/usr/local/man". If installing
to the default location you should delete the old man page before
proceeding. This file would be named "/usr/man/man1/fdupes.1".

This program was originally called "fdupes" but has been renamed to
"fdupes-jody" to keep it distinctly separate from Adrian Lopez's program.
Be aware of the name change and any problems it may cause on your system
before using fdupes-jody.

A test directory is included so that you may familiarise yourself
with the way fdupes operates. You may test the program before
installing it by issuing a command such as "./fdupes testdir"
or "./fdupes -r testdir", just to name a couple of examples. Refer
with the way jdupes operates. You may test the program before
installing it by issuing a command such as "./jdupes testdir"
or "./jdupes -r testdir", just to name a couple of examples. Refer
to the documentation for information on valid options.

A comparison shell script is also included. It will run your natively
installed 'fdupes' or 'fdupes-jody' with the directories and extra
installed 'jdupes' or 'jdupes' with the directories and extra
options you specify and compare the run times and output a 'diff' of
the two program outputs. Unless the core algorithm or sort behavior is
changed, both programs should produce identical outputs and the 'diff'
output shouldn't appear at all. To use it, type:

./ [options]
./ [options]

@@ -39,7 +39,7 @@ CFLAGS_CONFIG += -DNO_FLOAT
# PROGRAM_NAME determines the installation name and manual page name

# BIN_DIR indicates directory where program is to be installed.
@@ -48,7 +48,7 @@ PROGRAM_NAME=fdupes-jody

# MAN_DIR indicates directory where the fdupes-jody man page is to be
# MAN_DIR indicates directory where the jdupes man page is to be
# installed. Suggested value is "$(PREFIX)/man/man1"
MAN_BASE_DIR = $(PREFIX)/share/man
@@ -98,24 +98,24 @@ INSTALL_DATA = $(INSTALL) -c -m 0644

OBJECT_FILES += fdupes-jody.o jody_hash.o $(ADDITIONAL_OBJECTS)
OBJECT_FILES += jdupes.o jody_hash.o $(ADDITIONAL_OBJECTS)

# no need to modify anything beyond this point #

all: fdupes-jody
all: jdupes

fdupes-jody: $(OBJECT_FILES)
$(CC) $(CFLAGS) -o fdupes-jody $(OBJECT_FILES)
jdupes: $(OBJECT_FILES)
$(CC) $(CFLAGS) -o jdupes $(OBJECT_FILES)


install: fdupes-jody installdirs
install: jdupes installdirs

$(RM) $(OBJECT_FILES) fdupes-jody fdupes-jody.exe *~
$(RM) $(OBJECT_FILES) jdupes jdupes.exe *~
Oops, something went wrong.

2 comments on commit 67c69e2


This comment has been minimized.

Copy link

rr- replied Dec 24, 2015

Does that mean we should expect rewrite to Java soon? 🍰


This comment has been minimized.

Copy link

jbruchon replied Dec 24, 2015

That's an excellent idea for a horror film!

Please sign in to comment.