Exclude node_modules folders from Spotlight indexing #6453

as-com · 2018-09-30T17:16:41Z

Do you want to request a feature or report a bug?
Feature

What is the current behavior?
node_modules folders are indexed by Spotlight on macOS

What is the expected behavior?
node_modules folders are (possibly optionally) excluded from indexing

This can be done by putting a .metadata_never_index in node_modules whenever the folder is created.

This may not work (it didn't work for me?)
Another way is to create node_modules.noindex, and symlink node_modules to it. Alternatively, figure out a way to programmatically add to the list of Spotlight exclusions. Looks like I was holding it wrong.

The text was updated successfully, but these errors were encountered:

burkeshartsis · 2018-10-12T14:53:17Z

It seems like this only just became an issue when updating macOS to Mojave 🤷‍♂️

as-com · 2018-10-12T14:54:49Z

@burkeshartsis I thought node_modules polluting spotlight search results and high CPU usage from indexing always been an issue.

tcrossland · 2018-10-30T14:18:27Z

See also npm/npm#15346

jrz · 2019-03-11T13:29:12Z

15346 says the "true" solution is add folder names to the index. This is pure nonsense. It should be possible to add the metadata_never_index, OR to set a file attribute.
It's either this, or having a global repository and better module loading (and only have spotlight index the files once).

chrisbull · 2019-03-25T04:10:29Z

*** UPDATE *** .metadata_never_index doesn't work on Majave or Catalina

Found this quick script to add .metadat_never_index to your node_modules folders.

find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;

I added it as a bash alias, so I can use apply quickly.

alias fix-spotlight='find . -type d -name "node_modules" -exec touch "{}/.metadata_never_index" \;'

Just thought I'd share here, for anyone else that is looking. :)

kiddyfurby · 2019-05-09T09:50:31Z

I am sorry but will yarn consider this feature?

dicksonhk · 2019-06-03T04:43:45Z

Based on @chrisbull 's idea, this script does the same thing faster by not going inside any node_modules directories found (coz the whole node_modules directory had already been ignored)

find . -type d  -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \;

For myself, I run it on the home directory, so I exclude hidden directories and Library directory to speed things up

find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print

I also use the -print option so I know which folder is affected

justin808 · 2019-06-14T02:14:09Z

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

dicksonhk · 2019-06-25T11:21:25Z

For those using @dicksonhk's answer (thanks!) on MacOS, you have to have a dot or other directory after the command "find".

Maybe the same for other linux.

I agree that yarn should automatically add this file, although this is MacOS specific. Maybe some hook is better?

@justin808 You're right! I edited my answer to add . -type d and it should work fine!
Also, if you are running the command in the home directory like me you should take a look at the second command which excludes unnecessary directories :)

joaomnb · 2019-06-29T22:51:02Z

My Spotlight seems to ignore these .metadata_never_index files, every node_modules folder is still indexed... I'm running macOS Mojave. Anyone had the same problem?

rolandleth · 2019-08-01T15:08:49Z

Until this gets added — if it will — another approach would be to add a postinstall script:

"postinstall": "touch ./node_modules/.metadata_never_index"

Also, I didn't manage to make @dicksonhk's script to work (maybe because I'm using fish, dunno), but I did find another handy little script: https://github.com/Willian-Zhang/Ignore-node_modules-in-Spotlight.

0x80 · 2019-09-12T08:24:45Z

@rolandleth I think it makes more sense to use "preinstall", so spotlight doesn't start picking up the files while the install process is ongoing.

But now I tried it is seems that postinstall and preinstall both don't add the file to my node_modules. Even if I delete the folder and run install again 🤔

rolandleth · 2019-09-12T09:26:38Z

There is no node_modules folder at preinstall, though, if you come from a clean slate.

Be sure to show hidden files when checking if it's there or not:

furudean · 2019-09-20T11:39:33Z

@rolandleth

There is no node_modules folder at preinstall, though, if you come from a clean slate.

"preinstall": "mkdir -p node_modules; touch ./node_modules/.metadata_never_index"

rolandleth · 2019-09-20T13:03:57Z

Ha, nice call. Sadly, the file gets deleted after yarn finishes, no idea why 🤷🏻‍♂️

MariuzM · 2019-11-17T16:26:37Z

Guys does adding .metadata_never_index really work? Because it's still indexing node_modules and subfolders. I've ran
sudo find . -type d -path './.*' -prune -o -path './Pictures*' -prune -o -path './Library*' -prune -o -path '*node_modules/*' -prune -o -type d -name 'node_modules' -exec touch '{}/.metadata_never_index' \; -print
And then I cleared my index by using

sudo mdutil -i off /
sudo rm -rf /.Spotlight*
sudo mdutil -i on /
sudo mdutil -E /

And noticed that it still indexing node folders/subfolders

shanedg · 2019-11-17T17:53:30Z

I didn't see any change with .metadata_never_index (caveat that I don't think I cleared my index after adding) but adding my projects folders to to the Spotlight privacy list does seem to have helped. I haven't noticed high resource usage by Spotlight processes since (Mojave and Catalina).

joaomnb · 2019-11-18T00:39:22Z

Same. As I had said before, .metadata_never_index does not work on my system. It didn't with macOS Mojave, and still doesn't now with Catalina.

tsujp · 2019-11-27T07:38:57Z

I've read that .noindex might be an option for Spotlight? Do you see any results with that @joaomnb ?

joaomnb · 2019-11-27T23:16:31Z

No, doesn't work for me either. Spotlight still indexes everything...

joaomnb · 2019-11-27T23:49:45Z

I did some research on that .noindex thing and found this.

This works in a different way. If you want the folder test/ to be ignored by Spotlight, you can rename it to test.noindex/. While this does work, renaming folders is obviously not an acceptable solution...

nemosmithasf · 2019-12-04T10:51:18Z

.metadata_never_index doesn't work for me either on Catalina\

Potential workaround is to put all projects in a single folder, black-list that folder via spotlight privacy settings and then create a VScode workspace outside that folder which points to the project folder internally.

Strajk · 2020-07-07T12:23:48Z

I've glued together solutions found here and around the internets and automated it:

Get current Spotlight Exclusions
Find all node_modules dirs in a specified directory
Add new ones to Spotlight Exclusions
Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

wiegell · 2020-12-26T20:17:16Z

I've glued together solutions found here and around the internets and automated it:

Get current Spotlight Exclusions

Find all node_modules dirs in a specified directory

Add new ones to Spotlight Exclusions

Restart system service to apply changes

It's definitely not a code that I would be proud of, but it works :)

https://github.com/Strajk/setup/blob/master/programs/prevent-spotlight-from-indexing-node-modules.js

Is this actually working for anyone?? Find candidates are not evaluated??

Strajk · 2020-12-30T06:55:40Z

@wiegell still working for me :) can you clarify what do you mean by not being evaluated?

I've written the script intentionally in a very simple, step-by-step style so it's easy to walk-through and debug. I recommend running it in debug mode (I use WebStorm) and checking what each line does :)

cblims · 2021-01-21T10:09:37Z

You are missing execSync on line 21:

const candidates = cleanArray((`find ${DIR} -type d -name 'node_modules' -prune\`))

This works for me:

const candidates = cleanArray(execSync(`find ${DIR} -type d -name 'node_modules' -prune`))

kuncevic · 2021-02-12T07:42:51Z

@Strajk just tried to give it to go and got an error

Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

jasonbarry · 2021-02-12T07:51:11Z

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

kuncevic · 2021-02-12T07:59:44Z

@jasonbarry
✅ done

However I wonder how to fix the error above 🤪

jeffhuys · 2021-03-04T22:46:34Z

Sorry, it's not clear to me and it hasn't been brought up yet – why not just put all your JS projects into ~/Sites, and drag the Sites directory into the Spotlight privacy pane? Just exclude all your web projects from being Spotlight indexed in general since you can ⌘P in your IDE to find what you want. Then npm/yarn installs won't trigger an mds_stores reindex.

You don't know how many times I've been saved by spotlight searching a snippet of code that I KNEW I had somewhere, but didn't know where. Also, webstorm is much more of an energy hog than spotlight is (after indexing, that is).

v-kiniv · 2021-07-26T18:48:04Z

@Strajk just tried to give it to go and got an error
Candidates: 11, Existing: 0, To add: 11
child_process.js:669
    throw err;
    ^

Error: Command failed: printf "MYPWD" | sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist
I am on the latest Big Sur 11.2.1

UPD:

If I just run this command sudo -S /usr/libexec/PlistBuddy -c "Add :Exclusions: string /Users/User/Work/frontendwatch/service/node_modules" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist getting an error Error Reading File: /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist

Tried to get manually to this folder, seems like no permissions to get in to folder /System/Volumes/Data/.Spotlight-V100/

I had the same issue and resolved it by granting Full Disk Access permission to Terminal.app(or any other app you're using to run the script):

JamesLIAOHJ · 2022-08-11T05:14:48Z

Base on @chrisbull and @dicksonhk ， those find command is still create too mouch file on my macOS Big Sur 11.6. so I modified the command

find -E /Volumes/MacData/workspace -type d -name 'node_modules' -not -regex ".+/node_modules/.+/node_modules" -exec touch '{}/.metadata_never_index' \; -print

this command affect folders that have no /node_modules/ in the middle but /node_modules at the end.

iloveitaly · 2022-08-16T19:31:52Z

@JamesLIAOHJ I thought apple removed support for metadata_never_index (ref). Are you finding that this magic file works for you?

fengerzh · 2023-07-24T01:50:23Z

No, the only way to stop it is to add your whole project folder into "Privacy of spotlight". Evil Apple.

kenandalda · 2024-12-15T14:34:10Z

I use one of the LLMs to complete the shell verion of the script based on @Strajk , and it worked!

#!/bin/bash

# Function to exclude a path
exclude_from_spotlight() {
  local folder=$1
  if [ -d "$folder" ]; then
    echo "Excluding: $folder"
    # sudo mdutil -i off "$folder" > /dev/null 2>&1
    # sudo mdutil -E "$folder" > /dev/null 2>&1
    sudo /usr/libexec/PlistBuddy -c "Add :Exclusions: string $folder" /System/Volumes/Data/.Spotlight-V100/VolumeConfiguration.plist
  else
    echo "Path does not exist or is invalid: $folder"
  fi
}

# Function to find top-level node_modules and .git folders
find_top_level_folders() {
  local folder_name=$1
  local search_path=$2
  find "$search_path" -type d -name "$folder_name" -prune 2>/dev/null
}

# Main logic
if [ -z "$1" ]; then
  echo "Please enter the folder path to search. Example: ./exclude_spotlight.sh /Users/your_username"
  exit 1
fi

search_path=$1

echo "Starting to exclude top-level node_modules and .git folders in path $search_path..."

# Exclude all top-level node_modules folders
find_top_level_folders "node_modules" "$search_path" | while read -r folder; do
  exclude_from_spotlight "$folder"
done

# Exclude all top-level .git folders
find_top_level_folders ".git" "$search_path" | while read -r folder; do
  exclude_from_spotlight "$folder"
done

echo "All specified folders have been excluded from Spotlight."

Just save the code to exclude_spotlight.sh, and run it like:

./exclude_spotlight.sh /Users/your_username

ghost assigned rally25rs Sep 30, 2018

ghost added the triaged label Sep 30, 2018

rally25rs added cat-feature and removed triaged labels Sep 30, 2018

DanielRuf mentioned this issue Jan 15, 2020

[FEATURE] Prevent Spotlight on macOS from indexing the node_modules directory npm/cli#549

Closed

emendelski mentioned this issue Jan 30, 2020

Added postinstall hook to prevent indexing node_modules emendelski/webpack-starter-kit#3

Closed

zmoshansky mentioned this issue Mar 5, 2020

Failing to install big sized module #6221

Closed

Pezmc mentioned this issue Mar 2, 2021

Fix the spotlight script Strajk/setup#35

Open

Exclude node_modules folders from Spotlight indexing #6453

Exclude node_modules folders from Spotlight indexing #6453

Comments

as-com commented Sep 30, 2018 • edited Loading

burkeshartsis commented Oct 12, 2018

as-com commented Oct 12, 2018

tcrossland commented Oct 30, 2018

jrz commented Mar 11, 2019

chrisbull commented Mar 25, 2019 • edited Loading

kiddyfurby commented May 9, 2019

dicksonhk commented Jun 3, 2019 • edited Loading

justin808 commented Jun 14, 2019

dicksonhk commented Jun 25, 2019 • edited Loading

joaomnb commented Jun 29, 2019

rolandleth commented Aug 1, 2019

0x80 commented Sep 12, 2019 • edited Loading

rolandleth commented Sep 12, 2019 • edited Loading

furudean commented Sep 20, 2019 • edited Loading

rolandleth commented Sep 20, 2019

MariuzM commented Nov 17, 2019

shanedg commented Nov 17, 2019

joaomnb commented Nov 18, 2019

tsujp commented Nov 27, 2019 • edited Loading

joaomnb commented Nov 27, 2019

joaomnb commented Nov 27, 2019

nemosmithasf commented Dec 4, 2019 • edited Loading

Strajk commented Jul 7, 2020

wiegell commented Dec 26, 2020 • edited Loading

Strajk commented Dec 30, 2020

cblims commented Jan 21, 2021

kuncevic commented Feb 12, 2021 • edited Loading

jasonbarry commented Feb 12, 2021

kuncevic commented Feb 12, 2021

jeffhuys commented Mar 4, 2021

v-kiniv commented Jul 26, 2021

JamesLIAOHJ commented Aug 11, 2022

iloveitaly commented Aug 16, 2022

fengerzh commented Jul 24, 2023

kenandalda commented Dec 15, 2024 • edited Loading

as-com commented Sep 30, 2018 •

edited

Loading

chrisbull commented Mar 25, 2019 •

edited

Loading

dicksonhk commented Jun 3, 2019 •

edited

Loading

dicksonhk commented Jun 25, 2019 •

edited

Loading

0x80 commented Sep 12, 2019 •

edited

Loading

rolandleth commented Sep 12, 2019 •

edited

Loading

furudean commented Sep 20, 2019 •

edited

Loading

tsujp commented Nov 27, 2019 •

edited

Loading

nemosmithasf commented Dec 4, 2019 •

edited

Loading

wiegell commented Dec 26, 2020 •

edited

Loading

kuncevic commented Feb 12, 2021 •

edited

Loading

kenandalda commented Dec 15, 2024 •

edited

Loading