Not respecting exclude_paths?

[jdelStrother]

Hi there,
When I run codeclimate locally, the duplication engine always times out. lsof suggests it's spending quite a while reading the contents of my node_modules and tmp directories, which are excluded in exclude_paths. (Those directories appear to be correctly excluded in other engines)

Here's the debug output:

Running duplication: |[DEBUG] engine config: "{\"enabled\":true,\"config\":{\"languages\":[\"ruby\"]},\"exclude_fingerprints\":[\"2250d1464b24f34abf02cbbe872059fe\",\"f0db7ce1dd3e0c6999d697643ef396e7\"],\"include_paths\":[\".autotest\",\".bundle/\",\".cap_tasks\",\".codeclimate.yml\",\".DS_Store\",\".eslintrc\",\".eslintrc-standard\",\".eslintrc-standard-react\",\".gitattributes\",\".gitignore\",\".jshintrc\",\".powenv\",\".powrc\",\".preferred_ips\",\".projectile\",\".rspec\",\".rspec_parallel\",\".rubocop.yml\",\".rubocop_todo.yml\",\".ruby-gemset\",\".ruby-version\",\".sass-cache/\",\".simplecov\",\".spin.rb\",\"app/.DS_Store\",\"app/assets/.DS_Store\",\"app/assets/javascripts/.DS_Store\",\"app/assets/javascripts/.gitignore\",\"app/assets/javascripts/application.js\",\"app/assets/javascripts/embed.js\",\"app/assets/javascripts/embed_player.js\",\"app/assets/javascripts/embed_player_square_ad.js\",\"app/assets/javascripts/extras/\",\"app/assets/javascripts/popup_button.js\",\"app/assets/javascripts/preload.js.erb\",\"app/assets/javascripts/react/\",\"app/assets/stylesheets/\",\"app/controllers/.#payments_controller.rb\",\"app/controllers/.DS_Store\",\"app/controllers/admin/\",\"app/controllers/admin.rb\",\"app/controllers/analytics_controller.rb\",\"app/controllers/api/\",\"app/controllers/api.rb\",\"app/controllers/application_controller.rb\",\"app/controllers/audio_clips_controller.rb\",\"app/controllers/audio_comments_controller.rb\",\"app/controllers/base_controller.rb\",\"app/controllers/blogger_controller.rb\",\"app/controllers/categories_controller.rb\",\"app/controllers/channel_invites_controller.rb\",\"app/controllers/channels_controller.rb\",\"app/controllers/comments_controller.rb\",\"app/controllers/concerns/\",\"app/controllers/facebook_permissions_controller.rb\",\"app/controllers/flipboard_controller.rb\",\"app/controllers/follows_controller.rb\",\"app/controllers/images_controller.rb\",\"app/controllers/messages_controller.rb\",\"app/controllers/notices_controller.rb\",\"app/controllers/page_items_controller.rb\",\"app/controllers/payments_controller.rb\",\"app/controllers/permissions_controller.rb\",\"app/controllers/playlist_memberships_controller.rb\",\"app/controllers/playlists_controller.rb\",\"app/controllers/publishing_controller.rb\",\"app/controllers/ratings_controller.rb\",\"app/controllers/regions_controller.rb\",\"app/controllers/search_controller.rb\",\"app/controllers/services_controller.rb\",\"app/controllers/sessions_controller.rb\",\"app/controllers/sources_controller.rb\",\"app/controllers/static_controller.rb\",\"app/controllers/stats_controller.rb\",\"app/controllers/streams_controller.rb\",\"app/controllers/submissions_controller.rb\",\"app/controllers/tags_controller.rb\",\"app/controllers/text_comments_controller.rb\",\"app/controllers/transcriptions_controller.rb\",\"app/controllers/tumblr_controller.rb\",\"app/controllers/twitter/\",\"app/controllers/uploads_controller.rb\",\"app/controllers/user_assets_controller.rb\",\"app/controllers/users_controller.rb\",\"app/controllers/welcome_controller.rb\",\"app/controllers/widgets_controller.rb\",\"app/controllers/zendesk_session_controller.rb\",\"app/decorators/\",\"app/helpers/\",\"app/indices/\",\"app/mailers/\",\"app/models/\",\"app/presenters/\",\"app/reports/\",\"app/services/\",\"app/views/.DS_Store\",\"app/views/admin/\",\"app/views/analytics/\",\"app/views/api/\",\"app/views/audio_clips/\",\"app/views/audio_comments/\",\"app/views/banners/\",\"app/views/categories/\",\"app/views/channel_invites/\",\"app/views/channel_mailer/\",\"app/views/channels/\",\"app/views/comments/\",\"app/views/flipboard/\",\"app/views/follows/\",\"app/views/layouts/\",\"app/views/messages/\",\"app/views/notices/\",\"app/views/oauth/\",\"app/views/page_header/\",\"app/views/panels/\",\"app/views/payments/\",\"app/views/player/\",\"app/views/playlist_memberships/\",\"app/views/playlists/\",\"app/views/promos/\",\"app/views/publishing/\",\"app/views/ratings/\",\"app/views/regions/\",\"app/views/search/\",\"app/views/services/\",\"app/views/sessions/\",\"app/views/shared/\",\"app/views/sources/\",\"app/views/static/\",\"app/views/stats_mailer/\",\"app/views/streams/\",\"app/views/submissions/\",\"app/views/subscriptions/\",\"app/views/support_mailer/\",\"app/views/tags/\",\"app/views/text_comments/\",\"app/views/twitter/\",\"app/views/uploads/\",\"app/views/user_assets/\",\"app/views/user_mailer/\",\"app/views/users/\",\"app/views/welcome/\",\"app/views/wip/\",\"app/workers/\",\"bin/\",\"Capfile\",\"cert/\",\"clock.rb\",\"config/\",\"config.ru\",\"db/.DS_Store\",\"db/.gitignore\",\"db/amazon_ip_ranges.json\",\"db/geo_ip.dat\",\"db/migrate/\",\"db/schema.rb.orig\",\"db/seeds.rb\",\"db/sphinx/\",\"deploy/\",\"doc/\",\"files_archive/\",\"Gemfile\",\"Gemfile.lock\",\"lib/\",\"Procfile.dev\",\"profiles/\",\"public/.DS_Store\",\"public/.gitignore\",\"public/403.html\",\"public/404.html\",\"public/422.html\",\"public/500.html\",\"public/502.html\",\"public/api/\",\"public/assets-static/\",\"public/crossdomain.xml\",\"public/empxml.html\",\"public/errors/\",\"public/favicon.ico\",\"public/flash/\",\"public/googlef073f5b5cafab306.html\",\"public/i/\",\"public/icons/\",\"public/images/\",\"public/java/\",\"public/jwplayer/\",\"public/nokia_callback.html\",\"public/osd.xml\",\"public/pinterest-5b088.html\",\"public/robots-staging.txt\",\"public/robots.txt\",\"public/tests/\",\"public/wb_1979ce46ecf9652f.txt\",\"public/wordpressres/\",\"Rakefile\",\"report/\",\"script/\",\"scripts/\",\"TAGS\",\"test/\",\"webpack/.DS_Store\",\"webpack/.gitignore\",\"webpack/assets/\",\"webpack/index.html\",\"webpack/karma.conf.js\",\"webpack/npm-shrinkwrap.json\",\"webpack/package.json\",\"webpack/readme\",\"webpack/scripts/\",\"webpack/test/\",\"webpack/test.js\",\"webpack/vendor/README\",\"webpack/webpack-records.json\",\"webpack/webpack.config.base.js\",\"webpack/webpack.config.js\",\"webpack/webpack.production.config.js\",\"webpack/webpack.test.config.js\"]}"
[DEBUG] docker run: ["docker", "run", "--name", "cc-engines-duplication-2f5dc76e-6feb-4b5d-b290-ffd8cff1d662", "--cap-drop", "all", "--label", "com.codeclimate.label=b6a220ac-84c2-42e5-b08b-10ac2ca92021", "--memory", "512000000", "--memory-swap", "-1", "--net", "none", "--rm", "--volume", "/Users/jon/Developer/web:/code:ro", "--volume", "/tmp/cc/02b04a60-58b9-406d-975c-1925ec647418:/config.json:ro", "--user", "9000:9000", "codeclimate/codeclimate-duplication"]

While it's running, lsof suggests it's spending quite a while parsing the contents of, eg, /code/webpack/node_modules/babel-preset-es2015/. Given the include_paths getting passed to the engine above, I'm surprised it's touching those paths at all.

If I temporarily delete my webpack/node_modules directory and rerun the duplication engine, it completes within a minute or so.

Anything else I can try?

[ABaldwinHunter]

@jdelStrother Sorry for the trouble there and thanks for the report!

First off, could you make sure that everything is up to date by running

docker pull codeclimate/codeclimate && docker pull codeclimate/codeclimate-duplication

and let me know if the behavior is the same? If so, sounds like a bug in the duplication engine.

[jdelStrother]

Yeah, already up to date I'm afraid.

[adamnengland]

👍 I've confirmed this problem on the latest code as well.

[pbrisbin]

Hi all. #108 I think will fix this. I'll report back here when the image is released to Docker Hub.

Sorry for the early close: I mentioned that it "should" fix this issue in the commit message, so GH auto-closed. We can re-open if this turns out to not be the real fix.

[pbrisbin]

OK, the latest image has been updated on docker hub. Please give it a try and see if the problem persists.

[jdelStrother]

Seems much better, thanks