Asset browser becomes extremely slow with tens of thousands of assets

[mfischer-wfm]

Bug description

We use the local driver for our assets. The metadata of the images are in the asstes_meta table of the database and we have also executed the command php please eloquent:sync-assets.

With tens of thousands of assets (~ 300.000), the asset browser in the cp becomes extremely slow. Every single interaction with the assets (open asset browser, open asset to edit meta data, assign asset to a collection entry) takes 10 - 20 seconds.

With the help of xdebug, we were able to narrow down the problem to the Flysystem\LocalFilesystemAdapter.

The adapter's listContents method loops through the entire content of the specified folder/asset container - even if only 25 items per page are to be displayed in the cp.

If we simply insert a return; in the foreach loop of the listContents method (foreach ($iterator as $fileInfo) {}) after 25 iterations, we get the desired results in ~200 ms instead of ~10 seconds.

Update: There is a similar problem in the listContents method in the Flysystem\AwsS3V3Adapter. Without a limit, the method loops over all assets in the specified bucket.

How to reproduce

• create asset container with local disk

• upload or import tens of thousands of files into the local directory

Logs

Environment

Environment
Application Name: elektroniknet
Laravel Version: 11.44.2
PHP Version: 8.3.17
Composer Version: 2.8.3
Environment: local
Debug Mode: ENABLED
URL: 127.0.0.1:8000
Maintenance Mode: OFF
Timezone: UTC
Locale: de

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: redis
Database: mariadb
Logs: stack / single
Mail: log
Queue: sync
Session: redis

Statamic
Addons: 6
Sites: 1
Stache Watcher: Disabled
Static Caching: Disabled
Version: 5.56.0 PRO

Statamic Addons
alt-design/alt-seo: 1.3.0
daun/statamic-loupe: 1.6.0
jacksleight/statamic-bonus-routes: 0.3.2
statamic-rad-pack/meilisearch: 3.4.0
statamic-rad-pack/runway: 8.5.0
statamic/eloquent-driver: 4.23.0

Statamic Eloquent Driver
Asset Containers: eloquent
Assets: eloquent
Blueprints: file
Collection Trees: file
Collections: file
Entries: eloquent
Forms: eloquent
Global Sets: eloquent
Global Variables: eloquent
Navigation Trees: file
Navigations: file
Revisions: file
Sites: eloquent
Taxonomies: eloquent
Terms: eloquent
Tokens: eloquent

Installation

Fresh statamic/statamic site via CLI

Additional details

filesystems.php

    'disks' => [

        'assets' => [
            'driver' => 'local',
            'serve' => true,
            'root' => public_path('assets/images'),
            'url' => env('CDN_URL_ASSETS'),
            'disable_asserts' => true,
            'visibility' => 'public',
            'throw' => false,
        ],

        'svg' => [
            'driver' => 'local',
            'root' => public_path('assets/svg'),
            'url' => '/assets/svg',
            'visibility' => 'public',
            'throw' => false,
        ],

    ],

[ryanmitchell]

If you are using eloquent driver, then listContents should only be called to build the initial folder list and then cached. By any chance do you still have statamic.stache.watcher set to true?

I can see from your environment its false, but thats the most obvious reason I can think of for it rebuilding the cache every time.

Did XDebug give the stack trace for what was calling listContents on the eloquent driver / statamic core side of things?

[mfischer-wfm]

The statamic.stache.watcher is disabled.

I haven't noticed anything on the eloquent side in xdebug yet, since the loop through the asset order only starts after the return statement in the folder action in the BrowserController (or the show action in the AssetController). If we output the result arrays before the result statement with dd(..), we get the results after a few ms.

If it helps, I can create a new asset container with a few hundred images, run an xdebug trace and attach the logfile.

[ryanmitchell]

I understand that but xdebug should give you a slow method warning or something similar with a full stack trace? We need to know where has initiated the call to listcontents() as it should be being cached by eloquent driver.

[mfischer-wfm]

During the step debug with xdebug I could not detect any problems with the eloquent driver.

Everything runs smoothly until the return statement of BrowserController->folder from the position

    return [
        'data' => [
            'assets' => FolderAsset::collection($assets ?? collect())->resolve(),
            'folder' => array_merge((new Folder($folder))->resolve(), [
                'folders' => Folder::collection($folders->values()),
            ]),
        ],
        ...

the loop through the complete asset folder content starts.

Apart from the attached CallGraph, I am currently unable to provide any further information about the error.

Until we have found another solution, we will use our own FilesystemAdapter based on the LocalFilesystemAdapter, in which we abort the loop in listContents() after n iterations with a return;. Undoubtedly a dirty solution, but it allows us to continue with the project.

The issue can probably be closed with this.

[ryanmitchell]

No I think its legitimate.

It's hitting AssetContainer::assetFolders() which is then calling eloquent AssetContainer::folders() which is hitting the disk() (and therefore filesystem issue you're experiencing). It should be going through AssetContainerContents instead to get our cached folder list.

If you comment out these lines, is it any faster?

eloquent-driver/src/Assets/AssetContainer.php

Lines 131 to 135 in b64654e

	public function folders($folder = '/', $recursive = false)
	{
	return $this->disk()->getFolders($folder, $recursive);
	}

This looks like an eloquent issue is you want to transfer it @duncanmcclean

[mfischer-wfm]

Yes, by commenting out these lines I get the result much faster (~150 ms instead of ~6 s).

[ryanmitchell]

Great. I’ll work on a PR and get that into a release next week.