Exclude node_modules folders from Spotlight indexing #6453

as-com · 2018-09-30T17:16:41Z

Do you want to request a feature or report a bug?
Feature

What is the current behavior?
node_modules folders are indexed by Spotlight on macOS

What is the expected behavior?
node_modules folders are (possibly optionally) excluded from indexing

This can be done by putting a .metadata_never_index in node_modules whenever the folder is created.

This may not work (it didn't work for me?)
Another way is to create node_modules.noindex, and symlink node_modules to it. Alternatively, figure out a way to programmatically add to the list of Spotlight exclusions. Looks like I was holding it wrong.

The text was updated successfully, but these errors were encountered:

burkeshartsis · 2018-10-12T14:53:17Z

It seems like this only just became an issue when updating macOS to Mojave 🤷‍♂️

as-com · 2018-10-12T14:54:49Z

@burkeshartsis I thought node_modules polluting spotlight search results and high CPU usage from indexing always been an issue.

tcrossland · 2018-10-30T14:18:27Z

See also npm/npm#15346

jrz · 2019-03-11T13:29:12Z

15346 says the "true" solution is add folder names to the index. This is pure nonsense. It should be possible to add the metadata_never_index, OR to set a file attribute.
It's either this, or having a global repository and better module loading (and only have spotlight index the files once).

chrisbull · 2019-03-25T04:10:29Z

*** UPDATE *** .metadata_never_index doesn't work on Majave or Catalina

Found this quick script to add .metadat_never_index to your node_modules folders.

find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;

I added it as a bash alias, so I can use apply quickly.

alias fix-spotlight='find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;'

Just thought I'd share here, for anyone else that is looking. :)

kiddyfurby · 2019-05-09T09:50:31Z

I am sorry but will yarn consider this feature?

dicksonhk · 2019-06-03T04:43:45Z

Based on @chrisbull 's idea, this script does the same thing faster by not going inside any node_modules directories found (coz the whole node_modules directory had already been ignored)

find . -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \;

For myself, I run it on the home directory, so I exclude hidden directories and Library directory to speed things up

find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print

I also use the -print option so I know which folder is affected

justin808 · 2019-06-14T02:14:09Z

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

dicksonhk · 2019-06-25T11:21:25Z

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

@justin808 You're right! I edited my answer to add . -type d and it should work fine!
Also, if you are running the command in the home directory like me you should take a look at the second command which excludes unnecessary directories :)

joaomnb · 2019-06-29T22:51:02Z

My Spotlight seems to ignore these .metadata_never_index files, every node_modules folder is still indexed... I'm running macOS Mojave. Anyone had the same problem?

rolandleth · 2019-08-01T15:08:49Z

Until this gets added — if it will — another approach would be to add a postinstall script:

"postinstall": "touch ./node_modules/.metadata_never_index"

Also, I didn't manage to make @dicksonhk's script to work (maybe because I'm using fish, dunno), but I did find another handy little script: https://github.com/Willian-Zhang/Ignore-node_modules-in-Spotlight.

0x80 · 2019-09-12T08:24:45Z

@rolandleth I think it makes more sense to use "preinstall", so spotlight doesn't start picking up the files while the install process is ongoing.

But now I tried it is seems that postinstall and preinstall both don't add the file to my node_modules. Even if I delete the folder and run install again 🤔

rolandleth · 2019-09-12T09:26:38Z

There is no node_modules folder at preinstall, though, if you come from a clean slate.

Be sure to show hidden files when checking if it's there or not:

furudean · 2019-09-20T11:39:33Z

@rolandleth

There is no node_modules folder at preinstall, though, if you come from a clean slate.

"preinstall": "mkdir -p node_modules; touch ./node_modules/.metadata_never_index"

rolandleth · 2019-09-20T13:03:57Z

Ha, nice call. Sadly, the file gets deleted after yarn finishes, no idea why 🤷🏻‍♂️

MariuzM · 2019-11-17T16:26:37Z

Guys does adding .metadata_never_index really work? Because it's still indexing node_modules and subfolders. I've ran
sudo find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print
And then I cleared my index by using

sudo mdutil -i off /
sudo rm -rf /.Spotlight*
sudo mdutil -i on /
sudo mdutil -E /

And noticed that it still indexing node folders/subfolders

shanedg · 2019-11-17T17:53:30Z

I didn't see any change with .metadata_never_index (caveat that I don't think I cleared my index after adding) but adding my projects folders to to the Spotlight privacy list does seem to have helped. I haven't noticed high resource usage by Spotlight processes since (Mojave and Catalina).

joaomnb · 2019-11-18T00:39:22Z

Same. As I had said before, .metadata_never_index does not work on my system. It didn't with macOS Mojave, and still doesn't now with Catalina.

tsujp · 2019-11-27T07:38:57Z

I've read that .noindex might be an option for Spotlight? Do you see any results with that @joaomnb ?

joaomnb · 2019-11-27T23:16:31Z

No, doesn't work for me either. Spotlight still indexes everything...

joaomnb · 2019-11-27T23:49:45Z

I did some research on that .noindex thing and found this.

This works in a different way. If you want the folder test/ to be ignored by Spotlight, you can rename it to test.noindex/. While this does work, renaming folders is obviously not an acceptable solution...

nemosmithasf · 2019-12-04T10:51:18Z

.metadata_never_index doesn't work for me either on Catalina\

Potential workaround is to put all projects in a single folder, black-list that folder via spotlight privacy settings and then create a VScode workspace outside that folder which points to the project folder internally.

Strajk · 2020-07-07T12:23:48Z

I've glued together solutions found here and around the internets and automated it:

Get current Spotlight Exclusions
Find all node_modules dirs in a specified directory
Add new ones to Spotlight Exclusions
Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

wiegell · 2020-12-26T20:17:16Z

I've glued together solutions found here and around the internets and automated it:

Get current Spotlight Exclusions

Find all node_modules dirs in a specified directory

Add new ones to Spotlight Exclusions

Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

Is this actually working for anyone?? Find candidates are not evaluated??

Strajk · 2020-12-30T06:55:40Z

@wiegell still working for me :) can you clarify what do you mean by not being evaluated?

I've written the script intentionally in a very simple, step-by-step style so it's easy to walk-through and debug. I recommend running it in debug mode (I use WebStorm) and checking what each line does :)

cblims · 2021-01-21T10:09:37Z

You are missing execSync on line 21:

const candidates = cleanArray((`find ${DIR} -type d -name 'node_modules' -prune\`))

This works for me:

const candidates = cleanArray(execSync(`find ${DIR} -type d -name 'node_modules' -prune`))

kuncevic · 2021-02-12T07:42:51Z

@Strajk just tried to give it to go and got an error

Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

jasonbarry · 2021-02-12T07:51:11Z

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

kuncevic · 2021-02-12T07:59:44Z

@jasonbarry
✅ done

However I wonder how to fix the error above 🤪

jeffhuys · 2021-03-04T22:46:34Z

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

You don't know how many times I've been saved by spotlight searching a snippet of code that I KNEW I had somewhere, but didn't know where. Also, webstorm is much more of an energy hog than spotlight is (after indexing, that is).

v-kiniv · 2021-07-26T18:48:04Z

@Strajk just tried to give it to go and got an error
Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist
I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

I had the same issue and resolved it by granting Full Disk Access permission to Terminal.app(or any other app you're using to run the script):

JamesLIAOHJ · 2022-08-11T05:14:48Z

Base on @chrisbull and @dicksonhk ， those find command is still create too mouch file on my macOS Big Sur 11.6. so I modified the command

find -E /Volumes/MacData/workspace -type d -name 'node_modules' -not -regex ".+/node_modules/.+/node_modules" -exec touch '{}/.metadata_never_index' \; -print

this command affect folders that have no /node_modules/ in the middle but /node_modules at the end.

iloveitaly · 2022-08-16T19:31:52Z

@JamesLIAOHJ I thought apple removed support for metadata_never_index (ref). Are you finding that this magic file works for you?

fengerzh · 2023-07-24T01:50:23Z

No, the only way to stop it is to add your whole project folder into "Privacy of spotlight". Evil Apple.

ghost assigned rally25rs Sep 30, 2018

ghost added the triaged label Sep 30, 2018

rally25rs added cat-feature and removed triaged labels Sep 30, 2018

DanielRuf mentioned this issue Jan 15, 2020

[FEATURE] Prevent Spotlight on macOS from indexing the node_modules directory npm/cli#549

Closed

emendelski mentioned this issue Jan 30, 2020

Added postinstall hook to prevent indexing node_modules emendelski/webpack-starter-kit#3

Closed

zmoshansky mentioned this issue Mar 5, 2020

Failing to install big sized module #6221

Closed

Pezmc mentioned this issue Mar 2, 2021

Fix the spotlight script Strajk/setup#35

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exclude node_modules folders from Spotlight indexing #6453

Exclude node_modules folders from Spotlight indexing #6453

as-com commented Sep 30, 2018 •

edited

Loading

burkeshartsis commented Oct 12, 2018

as-com commented Oct 12, 2018

tcrossland commented Oct 30, 2018

jrz commented Mar 11, 2019

chrisbull commented Mar 25, 2019 •

edited

Loading

kiddyfurby commented May 9, 2019

dicksonhk commented Jun 3, 2019 •

edited

Loading

justin808 commented Jun 14, 2019

dicksonhk commented Jun 25, 2019 •

edited

Loading

joaomnb commented Jun 29, 2019

rolandleth commented Aug 1, 2019

0x80 commented Sep 12, 2019 •

edited

Loading

rolandleth commented Sep 12, 2019 •

edited

Loading

furudean commented Sep 20, 2019 •

edited

Loading

rolandleth commented Sep 20, 2019

MariuzM commented Nov 17, 2019

shanedg commented Nov 17, 2019

joaomnb commented Nov 18, 2019

tsujp commented Nov 27, 2019 •

edited

Loading

joaomnb commented Nov 27, 2019

joaomnb commented Nov 27, 2019

nemosmithasf commented Dec 4, 2019 •

edited

Loading

Strajk commented Jul 7, 2020

wiegell commented Dec 26, 2020 •

edited

Loading

Strajk commented Dec 30, 2020

cblims commented Jan 21, 2021

kuncevic commented Feb 12, 2021 •

edited

Loading

jasonbarry commented Feb 12, 2021

kuncevic commented Feb 12, 2021

jeffhuys commented Mar 4, 2021

v-kiniv commented Jul 26, 2021

JamesLIAOHJ commented Aug 11, 2022

iloveitaly commented Aug 16, 2022

fengerzh commented Jul 24, 2023

Exclude node_modules folders from Spotlight indexing #6453

Exclude node_modules folders from Spotlight indexing #6453

Comments

as-com commented Sep 30, 2018 • edited Loading

burkeshartsis commented Oct 12, 2018

as-com commented Oct 12, 2018

tcrossland commented Oct 30, 2018

jrz commented Mar 11, 2019

chrisbull commented Mar 25, 2019 • edited Loading

kiddyfurby commented May 9, 2019

dicksonhk commented Jun 3, 2019 • edited Loading

justin808 commented Jun 14, 2019

dicksonhk commented Jun 25, 2019 • edited Loading

joaomnb commented Jun 29, 2019

rolandleth commented Aug 1, 2019

0x80 commented Sep 12, 2019 • edited Loading

rolandleth commented Sep 12, 2019 • edited Loading

furudean commented Sep 20, 2019 • edited Loading

rolandleth commented Sep 20, 2019

MariuzM commented Nov 17, 2019

shanedg commented Nov 17, 2019

joaomnb commented Nov 18, 2019

tsujp commented Nov 27, 2019 • edited Loading

joaomnb commented Nov 27, 2019

joaomnb commented Nov 27, 2019

nemosmithasf commented Dec 4, 2019 • edited Loading

Strajk commented Jul 7, 2020

wiegell commented Dec 26, 2020 • edited Loading

Strajk commented Dec 30, 2020

cblims commented Jan 21, 2021

kuncevic commented Feb 12, 2021 • edited Loading

jasonbarry commented Feb 12, 2021

kuncevic commented Feb 12, 2021

jeffhuys commented Mar 4, 2021

v-kiniv commented Jul 26, 2021

JamesLIAOHJ commented Aug 11, 2022

iloveitaly commented Aug 16, 2022

fengerzh commented Jul 24, 2023

as-com commented Sep 30, 2018 •

edited

Loading

chrisbull commented Mar 25, 2019 •

edited

Loading

dicksonhk commented Jun 3, 2019 •

edited

Loading

dicksonhk commented Jun 25, 2019 •

edited

Loading

0x80 commented Sep 12, 2019 •

edited

Loading

rolandleth commented Sep 12, 2019 •

edited

Loading

furudean commented Sep 20, 2019 •

edited

Loading

tsujp commented Nov 27, 2019 •

edited

Loading

nemosmithasf commented Dec 4, 2019 •

edited

Loading

wiegell commented Dec 26, 2020 •

edited

Loading

kuncevic commented Feb 12, 2021 •

edited

Loading