Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use minimatch to exclude files #122

Merged
merged 1 commit into from
Nov 13, 2017

Conversation

NumminorihSF
Copy link
Contributor

@NumminorihSF NumminorihSF commented Nov 13, 2017

There is a problem with excluding. jscpd resolves exclude patterns
but there may be cases then include and exclude options have
no overlap. Also now it calculate difference between 2 lists but it's
slow in case of 2 big lists. This commit use minimatch module (it's
always installed cause it's glob's dependency) for filtering mathed
files. It allow prevent extra sync interaction with fs that should
make jscpd much faster on slow hdd.


This change is Reviewable

There is a problem with excluding. jscpd resolves `exclude` patterns
but there may be cases then `include` and `exclude` options have
no overlap. Also now it calculate difference between 2 lists but it's
slow in case of 2 big lists. This commit use minimatch module (it's
always installed cause it's glob's dependency) for filtering mathed
files. It allow prevent extra sync interaction with fs that should
make jscpd much faster on slow hdd.
@coveralls
Copy link

coveralls commented Nov 13, 2017

Coverage Status

Coverage decreased (-0.02%) to 91.748% when pulling 62e9d7e on NumminorihSF:feature/fast-excluding into a5ed788 on kucherenko:master.

@NumminorihSF
Copy link
Contributor Author

For example, in my case current version works too long:

time ./jscpd -p ./packages -g javascript -e "**/node_modules/**/*"
info:    jscpd - copy/paste detector for programming source code, developed by Andrey Kucherenko
info:    Preprocessors running time: durationMs=94186
info:    Scanning 23 files for duplicates...
info:    Scanning for duplicates time: durationMs=143
info:    Scanning... done!

info:    Start report generation...

info:    Found 0 exact clones with 0 duplicated lines in 0 files
 

 0.00% (0 lines) duplicated lines out of 1105 total lines of code.

warn:    output file is not provided
info:    Generate report time: durationMs=1
info:    All time: durationMs=94334

real    1m34,589s
user    1m32,656s
sys     0m0,832s

But updated version is much faster (I have ssd, it's just time for calculation in both cases):

time ../jscpd/bin/jscpd -p ./packages -g javascript -e "**/node_modules/**/*"
info:    jscpd - copy/paste detector for programming source code, developed by Andrey Kucherenko
info:    Preprocessors running time: durationMs=3059
info:    Scanning 23 files for duplicates...
info:    Scanning for duplicates time: durationMs=116
info:    Scanning... done!

info:    Start report generation...

info:    Found 0 exact clones with 0 duplicated lines in 0 files
 

 0.00% (0 lines) duplicated lines out of 1105 total lines of code.

warn:    output file is not provided
info:    Generate report time: durationMs=1
info:    All time: durationMs=3179

real    0m3,470s
user    0m3,268s
sys     0m0,164s

@kucherenko kucherenko merged commit f716358 into kucherenko:master Nov 13, 2017
@kucherenko
Copy link
Owner

Thank you for PR, merged and published. Changes available at jscpd@0.6.17

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants