Unindexed app detection is slow for nonstandard apps

[tplobo]

Configuration

mas ▁▁▁▁ 5.2.0
slice ▁▁ arm64
slices ▁ arm64
dist ▁▁▁ homebrew/core/mas
origin ▁ https://github.com/mas-cli/mas.git
rev ▁▁▁▁ e84c0658e1dfff2fd1eaf0fc8ef338a2a99b8f67
swift ▁▁ 6.2.3 (swift-6.2.3-RELEASE)
driver ▁ unknown
store ▁▁ DE
region ▁ DE
macos ▁▁ 14.8.4 (23J319)
mac ▁▁▁▁ Mac15,9
cpu ▁▁▁▁ Apple M3 Max
arch ▁▁▁ arm64

mas ▁▁▁▁ 5.2.0
slice ▁▁ arm64
slices ▁ arm64
dist ▁▁▁ homebrew/core/mas
origin ▁ https://github.com/mas-cli/mas.git
rev ▁▁▁▁ e84c0658e1dfff2fd1eaf0fc8ef338a2a99b8f67
swift ▁▁ 6.2.3 (swiftlang-6.2.3.3.21 clang-1700.6.3.2)
driver ▁ 1.127.14.1
store ▁▁ DE
region ▁ DE
macos ▁▁ 26.3 (25D125)
mac ▁▁▁▁ Mac16,11
cpu ▁▁▁▁ Apple M4 Pro
arch ▁▁▁ arm64

Bug description

This is a branching of comments made in #1207.

For some time now, mas has started taking a long time to output anything (e.g. ~2min to start running upgrade, >1min to run list). Especially confusing when taking long to run mas upgrade even if it outputs nothing (i.e. nothing to update). This might have started after v5 was released, but this still needs to be confirmed. I'll try to install other versions with homebrew to see if I can isolate the version in which slowness sets in.

Issue is common for two machines running different macOS versions, so two outputs for mas config are given. In the same order as above, here follow the lists of apps (25 and 30 respectively) being used in each computer:

1617089333  AVIF Converter              (2.0.2)
 425264550  Blackmagic Disk Speed Test  (3.4.2)
1381004916  Discovery                   (2.1.0)
 971826370  Extractor                   (1.6)
1512570906  Flowchart Designer 3        (1.2.10)
1483172210  Free Ruler                  (2.0.6)
 682658836  GarageBand                  (10.4.13)
 668208984  GIPHY CAPTURE               (4.2)
1444383602  Goodnotes                   (7.0.36)
 408981434  iMovie                      (10.4.3)
 409183694  Keynote                     (14.5)
 462058435  Microsoft Excel             (16.106.3)
 784801555  Microsoft OneNote           (16.106.3)
 985367838  Microsoft Outlook           (16.106.3)
 462062816  Microsoft PowerPoint        (16.106.3)
 462054704  Microsoft Word              (16.106.3)
 409203825  Numbers                     (14.5)
 409201541  Pages                       (14.5)
 593341977  PDF Attributes              (2.1.2)
 545519333  Prime Video                 (10.120.1)
1477595241  QrScan                      (4.0.2)
1497435571  Simple Comic                (1.9.9)
 425424353  The Unarchiver              (4.3.9)
1295203466  Windows App                 (11.3.3)
1451685025  WireGuard                   (1.0.16)

1168254295  AmorphousDiskMark           (4.0.1)
6748853585  AppGrid                     (1.31)
1617089333  AVIF Converter              (2.0.2)
 425264550  Blackmagic Disk Speed Test  (3.4.2)
1381004916  Discovery                   (2.1.0)
 446243721  Disk Space Analyzer         (6.0)
 971826370  Extractor                   (1.6)
1512570906  Flowchart Designer 3        (1.2.10)
1483172210  Free Ruler                  (2.0.6)
 682658836  GarageBand                  (10.4.13)
 668208984  GIPHY CAPTURE               (4.2)
1444383602  Goodnotes                   (7.0.36)
 408981434  iMovie                      (10.4.4)
 409183694  Keynote                     (14.5)
 784801555  Microsoft OneNote           (16.106.3)
 985367838  Microsoft Outlook           (16.106.3)
 462062816  Microsoft PowerPoint        (16.106.3)
 462054704  Microsoft Word              (16.106.3)
 409203825  Numbers                     (14.5)
 823766827  OneDrive                    (26.002.0105)
 409201541  Pages                       (14.5)
 593341977  PDF Attributes              (2.1.2)
 545519333  Prime Video                 (10.120.1)
1497435571  Simple Comic                (1.9.9)
 425424353  The Unarchiver              (4.3.9)
 310633997  WhatsApp                    (26.8.75)
1295203466  Windows App                 (11.3.3)
1451685025  WireGuard                   (1.0.16)
 497799835  Xcode                       (26.3)
 562211012  Yomu                        (3.16.0)

Steps to reproduce

Run mas upgrade or mas list.

[tplobo]

P.S.: would it be possible to set up homebrew formulas for milestone mas versions, other than the latest?

This would make these tests significantly quicker with (for example):

> brew search mas
mas✔      mas@4.1.2     mas@3.1.0

> brew install mas@4.1.2
...

> brew switch mas 4.1.2

to run mas at specific versions.

[rgoldberg]

I could set up versioned formulae on the custom tap, but it might take some time as I've never done it.

I cannot imagine that Homebrew Core would accept versioned formulae for mas.

[tplobo]

I see... indeed, the use of different versions must be justified with number of downloads, so it might be difficult to do so...

I thought that the mas-cli itself already had different versions in the repo homebrew-tap... Is it just a matter of setting formulas for the different versions?

Do you think this would be beneficial for project management and debugging? It would make it significantly easier to compare performance and behavior between versions.

[rgoldberg]

I compare performance between the current & upcoming version on my Mac before I release, but I only have my setup, and thus cannot test or know all possible things affecting performance.

Can you run the following before running mas update?

export MAS_NO_AUTO_INDEX=1

That prevents searching /Applicationa for probable App Store apps to auto-index. That's a recent feature, so maybe disabling it via the environment variable will fix your performance issue.

[tplobo]

hi again @rgoldberg,

look, I tested calling mas list on different terminal windows, both with and without calling export MAS_NO_AUTO_INDEX=1 beforehand on both computers. Without calling export, even on repeated calls in the same terminal, listing all installed apps always takes 1-2 min. With export, the calls take ~3s.

This might not be the only reason for the performance drop, but it is definitely a culprit here.

I can test it with mas update or mas upgrade as well, if you want.

[rgoldberg]

@tplobo Thanks for the info, it's very helpful.

Do you have a ton of iOS/iPadOS apps installed on you Mac?

I just remembered that iOS/iPadOS apps have a Wrapper folder instead of Contents.

Optimizations for unindexed app detection rely on apps having a Contents folder.

They also rely on you not having tons of non-app items in the file system under /Applications (or under the large App Store app external drive /Applications folder, if you've set one up).

Do you have a ton of non-apps in either of those folders?

Your Macs are new enough that their storage should be fast.

The place in the code that handles unindexed apps that should take the longest is when it searches for such apps on the file system.

The other code should run fast, unless maybe you saw output that begins like the following in your stderr when you run without exporting the environment variable:

Found a likely App Store app that is not indexed in Spotlight in

Did you see any of that?

The file system searching code iterates through the file system rooted from your /Applications folder (& possibly a configured /Applications on an external drive for large App Store apps). All normal apps I've seen have a Contents folder directly under the app root folder. To speed things up, whenever the code sees a Contents folder, it stops traversing the file tree, then looks for a _MASReceipt/receipt file.

Otherwise, I don't know why your system would be exceptionally slow with this.

[tplobo]

OK, so let me answer one thing at a time.

---

Do you have a ton of iOS/iPadOS apps installed on you Mac?

Not that I know of. I added above the lists of apps I have in both computers, that were installed by the AppStore, and I think they are all macOS-native versions.

---

They also rely on you not having tons of non-app items in the file system under /Applications or external drive... Do you have a ton of non-apps in either of those folders?

Maybe:

• I have some apps stored in sub-folders inside the /Applications folder.
• In the M4 mac (where I have more apps) I have 173 items there, and 16 of those are folders that contain multiple apps or files added by the installers of software I am forced to use for work.
• I have 59 apps installed as homebrew casks, which are not listed by mas since I do not download them from the AppStore.

---

... unless maybe you saw output that begins with "Found a likely App Store app that..." in your stderr.

I ran mas list 1>stdout.txt 2>stderr.txt without exporting the variable, and stdout contains the list as expected, while stderr is empty.

---

Let me know if you need any extra info!

[rgoldberg]

The slowdown shouldn't come from any apps listed by mas list; it should come from other folders under your Applications folders.

Can you run the following 2 commands, then reply with both of their outputs?

time find -E /Volumes/*/Applications -type d -regex '.*/(Contents|Wrapper)' -prune -print
time find -E /Volumes/*/Applications -type d -regex '.*/(Contents|Wrapper)' -prune -o -type d -print

The first lists all of the folders that are named Contents or Wrapper.

The second lists all the folders that aren't named either Contents or Wrapper, and that aren't descendants of folders that are named Contents or Wrapper.

They will search the /Applications folder on each of your volumes (drives), in case you have a large-app folder set up.

The time at the start is just to see how long the find commands take to run, to see if they run slowly. They should be fast unless you have tons of file system items under the searched Applications folders that aren't normally configured macOS, iOS, or iPadOS apps, at least to my knowledge.

[tplobo]

Here you go:

• ... -prune -print: 2.42s user 44.02s system 28% cpu 2:44.88 total
• ... -prune -o -type d -print: 2.62s user 47.32s system 30% cpu 2:46.18 total

Most of the time they took to run was spent on MATLAB files. I have several versions installed (R2020b, R2021a, R2022b, R2023a...).

[rgoldberg]

@tplobo thanks for the info. It's very helpful.

I assume that the listed MATLAB files are not under any folder named Contents or Wrapper, right?

Can you provide the roots of the trees of output folders, so I can know how to avoid unnecessarily walking certain folder trees? I.e., provide all the shortest non-trivial common path prefixes from the output of the second command; e.g., /Volumes/*/Applications is trivial, so don't give that as a shortest common path prefix.

How did you install your MATLAB installs?