Strapi media library very slow or even fails to load when files are linked to multiple entities

[weidox]

Describe the bug
When files are linked to tens of entities, it taken very long to open media library.

Steps to reproduce the behavior

1. Upload a single image to Strapi library.
2. Create a simple collection definition, having just single "file" property of type Media.
3. Fill collection with 50 records, all referencing the same file in Media Library.
4. Try to open Media Library.
5. See how long it takes to open media library, in my case it was 13 seconds.
   debug GET /upload/files?_limit=10&_start=0&_sort=updatedAt:DESC (12788 ms) 200
6. Add 50 more records, see now 87 seconds.
   debug GET /upload/files?_limit=10&_start=0&_sort=updatedAt:DESC (86799 ms) 200
7. Add 100 more records, go to make coffee, when you return if you're lucky you will have some number. I wasn't lucky and could not wait seemingly forever.

Expected behavior
Expected to be able to open Media Library under any circumstances, and in reasonable amount of time.

Screenshots
Not needed.

Code snippets
To quickly create 50 additional records on restart, may use this in bootstrap.js:

  const media = await strapi.query('file', 'upload').findOne({}, []);
  for (let i = 0; i < 50; i++) {
    await strapi.services['your-collection-with-file-property'].create({ file: media.id });
  }

System

• Node.js version: v12.18.3
• NPM version: 6.14.6
• Strapi version: 3.1.4
• Database: MongoDB
• Operating system: both Linux and Windows

Additional context
As I figured out, each media item keeps track of linked entities in "related" property, and if this property gets loaded (by default, unless suppressed with populate parameter in query), it works unacceptably slowly.

[alexandrebodin]

Hello, Please update the issue to match issue templates. Closing this issue until it gets updated

[weidox]

Updated.

[mukeshsolanki]

Yes i agree im facing this exact issue as well

[zambawamba]

@derrickmehaffy I'm having the same issue and I would say severity is definitely higher than low. At the moment I'm completely blocked from finishing work because the media library is essentially non-functional.

[sercanov]

I can also confirm that Media Library performance degraded so bad after 2k files.
I've enabled debug mode for Mongoose and it seems Strapi tries to find every uploaded file in 500 id batches.
Then again, only related file ID's

Here are the logs;

Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("60534986ee0f0900b43867fa"), ObjectId("605349ddee0f0900b4386804"), ObjectId("60534a31ee0f0900b438680f"), ObjectId("60534ae4ee0f0900b438681a"), ObjectId("60534b4eee0f0900b4386825"), ObjectId("60534bc7ee0f0900b4386830"), ObjectId("60534c25ee0f0900b438683b"), ObjectId("60534ca5ee0f0900b4386846"), ObjectId("60534cf3ee0f0900b4386851"), ObjectId("60534d7bee0f0900b438685c"), ObjectId("60534f39ee0f0900b4386869"), ObjectId("60534f8bee0f0900b4386874"), ObjectId("60534fdfee0f0900b438687f"), ObjectId("60535034ee0f0900b438688a"), ObjectId("60535085ee0f0900b4386895"), ObjectId("605350d7ee0f0900b43868a0"), ObjectId("6053511eee0f0900b43868ab"), ObjectId("60535168ee0f0900b43868b6"), ObjectId("605351e0ee0f0900b43868c1"), ObjectId("605355f0ee0f0900b43868cc"), ObjectId("6053563cee0f0900b43868d7"), ObjectId("60535687ee0f0900b43868e2"), ObjectId("605356d3ee0f0900b43868ed"), ObjectId("60535779ee0f0900b43868f8"), ObjectId("605357d2ee0f0900b4386903"), ObjectId("605396d9ee0f0900b4386913"), ObjectId("60539795ee0f0900b438691d"), ObjectId("6053988eee0f0900b4386928"), ObjectId("605398ffee0f0900b4386933"), ObjectId("60539978ee0f0900b438693e"), ObjectId("605399e9ee0f0900b4386949"), ObjectId("60539a66ee0f0900b4386954"), ObjectId("60539addee0f0900b438695f"), ObjectId("60539b3aee0f0900b438696a"), ObjectId("60549931b7ebdb00b7af20f5"), ObjectId("60549944b7ebdb00b7af2100"), ObjectId("60549953b7ebdb00b7af210b"), ObjectId("6054995eb7ebdb00b7af2116"), ObjectId("60549970b7ebdb00b7af2121"), ObjectId("6054a7ae426ce500b5748a2f"), ObjectId("6054a7ba426ce500b5748a3a"), ObjectId("6054a7c6426ce500b5748a45"), ObjectId("6054a7d6426ce500b5748a50"), ObjectId("6054a7e2426ce500b5748a5b"), ObjectId("6054a7f0426ce500b5748a66"), ObjectId("6054a7fc426ce500b5748a71"), ObjectId("6054a808426ce500b5748a7c"), ObjectId("6054a816426ce500b5748a87"), ObjectId("6054a824426ce500b5748a92"), ObjectId("6054a834426ce500b5748a9d"), ObjectId("6054a843426ce500b5748aa8"), ObjectId("6054a850426ce500b5748ab3"), ObjectId("6054a85d426ce500b5748abe"), ObjectId("6054a86f426ce500b5748ac9"), ObjectId("6054a881426ce500b5748ad4"), ObjectId("6054a88e426ce500b5748adf"), ObjectId("6054a89c426ce500b5748aea"), ObjectId("6054a8aa426ce500b5748af5"), ObjectId("6054a8b9426ce500b5748b00"), ObjectId("6054a8c7426ce500b5748b0b"), ObjectId("6054a8d2426ce500b5748b16"), ObjectId("6054a8e2426ce500b5748b21"), ObjectId("6054a8ef426ce500b5748b2c"), ObjectId("6054a8fc426ce500b5748b37"), ObjectId("6054a907426ce500b5748b42"), ObjectId("6054a913426ce500b5748b4d"), ObjectId("6054a922426ce500b5748b58"), ObjectId("6054a92e426ce500b5748b63"), ObjectId("6054a939426ce500b5748b6e"), ObjectId("6054a947426ce500b5748b79"), ObjectId("6054a95b426ce500b5748b84"), ObjectId("6054a96b426ce500b5748b8f"), ObjectId("6054a979426ce500b5748b9a"), ObjectId("6054a985426ce500b5748ba5"), ObjectId("6054a995426ce500b5748bb0"), ObjectId("6054a9a1426ce500b5748bbb"), ObjectId("6054a9ad426ce500b5748bc6"), ObjectId("6054a9bb426ce500b5748bd1"), ObjectId("6054a9cc426ce500b5748bdc"), ObjectId("6054a9da426ce500b5748be7"), ObjectId("6054a9e7426ce500b5748bf2"), ObjectId("6054a9f3426ce500b5748bfd"), ObjectId("6054a9fe426ce500b5748c08"), ObjectId("6054aa0e426ce500b5748c13"), ObjectId("6054aa1d426ce500b5748c1e"), ObjectId("6054aa29426ce500b5748c29"), ObjectId("6054aa35426ce500b5748c34"), ObjectId("6054aa45426ce500b5748c3f"), ObjectId("6054aa51426ce500b5748c4a"), ObjectId("6054aa5c426ce500b5748c55"), ObjectId("6054aa69426ce500b5748c60"), ObjectId("6054aa77426ce500b5748c6b"), ObjectId("6054aa86426ce500b5748c76"), ObjectId("6054aa93426ce500b5748c81"), ObjectId("6054aa9e426ce500b5748c8c"), ObjectId("6054aaa9426ce500b5748c97"), ObjectId("6054aab6426ce500b5748ca2"), ObjectId("6054aac2426ce500b5748cad"), ObjectId("6054aacc426ce500b5748cb8"), ObjectId("6054aada426ce500b5748cc3"), ... 404 more items ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("60534972ee0f0900b43867f9"), ObjectId("605349f1ee0f0900b4386805"), ObjectId("60534a46ee0f0900b4386810"), ObjectId("60534b05ee0f0900b438681b"), ObjectId("60534b61ee0f0900b4386826"), ObjectId("60534bdeee0f0900b4386831"), ObjectId("60534c35ee0f0900b438683c"), ObjectId("60534cbaee0f0900b4386847"), ObjectId("60534d04ee0f0900b4386852"), ObjectId("60534eadee0f0900b4386867"), ObjectId("60534f4aee0f0900b438686a"), ObjectId("60534fa2ee0f0900b4386875"), ObjectId("60534ff0ee0f0900b4386880"), ObjectId("6053503dee0f0900b438688b"), ObjectId("6053508cee0f0900b4386896"), ObjectId("605350e1ee0f0900b43868a1"), ObjectId("6053512eee0f0900b43868ac"), ObjectId("60535170ee0f0900b43868b7"), ObjectId("605351f2ee0f0900b43868c2"), ObjectId("605355fdee0f0900b43868cd"), ObjectId("6053564eee0f0900b43868d8"), ObjectId("60535696ee0f0900b43868e3"), ObjectId("605356e6ee0f0900b43868ee"), ObjectId("60535783ee0f0900b43868f9"), ObjectId("605357ddee0f0900b4386904"), ObjectId("605396c1ee0f0900b4386912"), ObjectId("605397a7ee0f0900b438691e"), ObjectId("605398a4ee0f0900b4386929"), ObjectId("6053990fee0f0900b4386934"), ObjectId("60539999ee0f0900b438693f"), ObjectId("60539a01ee0f0900b438694a"), ObjectId("60539a7cee0f0900b4386955"), ObjectId("60539aebee0f0900b4386960"), ObjectId("60539b4cee0f0900b438696b"), ObjectId("6054992fb7ebdb00b7af20f2"), ObjectId("60549941b7ebdb00b7af20fd"), ObjectId("60549950b7ebdb00b7af2108"), ObjectId("6054995cb7ebdb00b7af2113"), ObjectId("6054996db7ebdb00b7af211e"), ObjectId("6054a7aa426ce500b5748a2c"), ObjectId("6054a7b8426ce500b5748a37"), ObjectId("6054a7c4426ce500b5748a42"), ObjectId("6054a7d4426ce500b5748a4d"), ObjectId("6054a7e0426ce500b5748a58"), ObjectId("6054a7ee426ce500b5748a63"), ObjectId("6054a7fa426ce500b5748a6e"), ObjectId("6054a806426ce500b5748a79"), ObjectId("6054a813426ce500b5748a84"), ObjectId("6054a821426ce500b5748a8f"), ObjectId("6054a832426ce500b5748a9a"), ObjectId("6054a83f426ce500b5748aa5"), ObjectId("6054a84e426ce500b5748ab0"), ObjectId("6054a85a426ce500b5748abb"), ObjectId("6054a86c426ce500b5748ac6"), ObjectId("6054a87f426ce500b5748ad1"), ObjectId("6054a88c426ce500b5748adc"), ObjectId("6054a899426ce500b5748ae7"), ObjectId("6054a8a7426ce500b5748af2"), ObjectId("6054a8b6426ce500b5748afd"), ObjectId("6054a8c4426ce500b5748b08"), ObjectId("6054a8d0426ce500b5748b13"), ObjectId("6054a8e0426ce500b5748b1e"), ObjectId("6054a8ed426ce500b5748b29"), ObjectId("6054a8fa426ce500b5748b34"), ObjectId("6054a905426ce500b5748b3f"), ObjectId("6054a911426ce500b5748b4a"), ObjectId("6054a920426ce500b5748b55"), ObjectId("6054a92c426ce500b5748b60"), ObjectId("6054a937426ce500b5748b6b"), ObjectId("6054a944426ce500b5748b76"), ObjectId("6054a957426ce500b5748b81"), ObjectId("6054a968426ce500b5748b8c"), ObjectId("6054a976426ce500b5748b97"), ObjectId("6054a982426ce500b5748ba2"), ObjectId("6054a993426ce500b5748bad"), ObjectId("6054a99e426ce500b5748bb8"), ObjectId("6054a9aa426ce500b5748bc3"), ObjectId("6054a9b8426ce500b5748bce"), ObjectId("6054a9c5426ce500b5748bd9"), ObjectId("6054a9d7426ce500b5748be4"), ObjectId("6054a9e4426ce500b5748bef"), ObjectId("6054a9f1426ce500b5748bfa"), ObjectId("6054a9fc426ce500b5748c05"), ObjectId("6054aa0c426ce500b5748c10"), ObjectId("6054aa1b426ce500b5748c1b"), ObjectId("6054aa27426ce500b5748c26"), ObjectId("6054aa32426ce500b5748c31"), ObjectId("6054aa40426ce500b5748c3c"), ObjectId("6054aa4e426ce500b5748c47"), ObjectId("6054aa5a426ce500b5748c52"), ObjectId("6054aa66426ce500b5748c5d"), ObjectId("6054aa75426ce500b5748c68"), ObjectId("6054aa84426ce500b5748c73"), ObjectId("6054aa91426ce500b5748c7e"), ObjectId("6054aa9c426ce500b5748c89"), ObjectId("6054aaa7426ce500b5748c94"), ObjectId("6054aab4426ce500b5748c9f"), ObjectId("6054aac0426ce500b5748caa"), ObjectId("6054aacb426ce500b5748cb5"), ObjectId("6054aad8426ce500b5748cc0"), ... 405 more items ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: merchants.find({ '$and': [ { merchant_id: 'adae0132-6ddd-4809-9447-4316cbd7e876' } ] }, { publicationState: undefined, session: null, skip: 0, limit: 1, projection: {}})
Mongoose: organizations.find({ _id: { '$in': [ ObjectId("6034e1836e4f4b00b3e3c327") ] } }, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: products.find({ merchant: { '$in': [ ObjectId("604f5536882efa00b6c52287") ] } }, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: strapi_administrator.find({ _id: { '$in': [ ObjectId("6040df5620d78300b2eb0fb5") ] } }, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: strapi_administrator.find({ _id: { '$in': [ ObjectId("6040df5620d78300b2eb0fb5") ] } }, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("6054b82a426ce500b574948a"), ObjectId("6054b836426ce500b5749495"), ObjectId("6054b842426ce500b57494a0"), ObjectId("6054b853426ce500b57494ab"), ObjectId("6054b85d426ce500b57494b6"), ObjectId("6054b869426ce500b57494c1"), ObjectId("6054b875426ce500b57494cc"), ObjectId("6054b881426ce500b57494d7"), ObjectId("6054b890426ce500b57494e2"), ObjectId("6054b89c426ce500b57494ed"), ObjectId("6054b8a7426ce500b57494f8"), ObjectId("6054b8b5426ce500b5749503"), ObjectId("6054b8c6426ce500b574950e"), ObjectId("6054b8dd426ce500b5749519") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("6054b81c426ce500b574947c"), ObjectId("6054b828426ce500b5749487"), ObjectId("6054b834426ce500b5749492"), ObjectId("6054b840426ce500b574949d"), ObjectId("6054b84e426ce500b57494a8"), ObjectId("6054b85c426ce500b57494b3"), ObjectId("6054b867426ce500b57494be"), ObjectId("6054b873426ce500b57494c9"), ObjectId("6054b87e426ce500b57494d4"), ObjectId("6054b88e426ce500b57494df"), ObjectId("6054b89a426ce500b57494ea"), ObjectId("6054b8a4426ce500b57494f5"), ObjectId("6054b8b3426ce500b5749500"), ObjectId("6054b8c4426ce500b574950b"), ObjectId("6054b8db426ce500b5749516") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: products.find({ '$and': [ { merchant: ObjectId("604f5536882efa00b6c52287") }, { organization: ObjectId("6034e1836e4f4b00b3e3c327") } ]}, { publicationState: undefined, session: null, skip: 0, limit: 100, projection: {}})
Mongoose: models.find({ product: { '$in': [ ObjectId("6054b81e426ce500b574947d"), ObjectId("6054b829426ce500b5749488"), ObjectId("6054b836426ce500b5749493"), ObjectId("6054b841426ce500b574949e"), ObjectId("6054b852426ce500b57494a9"), ObjectId("6054b85d426ce500b57494b4"), ObjectId("6054b868426ce500b57494bf"), ObjectId("6054b875426ce500b57494ca"), ObjectId("6054b880426ce500b57494d5"), ObjectId("6054b890426ce500b57494e0"), ObjectId("6054b89b426ce500b57494eb"), ObjectId("6054b8a6426ce500b57494f6"), ObjectId("6054b8b4426ce500b5749501"), ObjectId("6054b8c6426ce500b574950c"), ObjectId("6054b8dd426ce500b5749517") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("6054b82a426ce500b574948a"), ObjectId("6054b836426ce500b5749495"), ObjectId("6054b842426ce500b57494a0"), ObjectId("6054b853426ce500b57494ab"), ObjectId("6054b85d426ce500b57494b6"), ObjectId("6054b869426ce500b57494c1"), ObjectId("6054b875426ce500b57494cc"), ObjectId("6054b881426ce500b57494d7"), ObjectId("6054b890426ce500b57494e2"), ObjectId("6054b89c426ce500b57494ed"), ObjectId("6054b8a7426ce500b57494f8"), ObjectId("6054b8b5426ce500b5749503"), ObjectId("6054b8c6426ce500b574950e"), ObjectId("6054b8dd426ce500b5749519") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: strapi_administrator.find({ _id: { '$in': [ ObjectId("5fed98b71167c6ad3b3e6858") ] } }, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("6054b81c426ce500b574947c"), ObjectId("6054b828426ce500b5749487"), ObjectId("6054b834426ce500b5749492"), ObjectId("6054b840426ce500b574949d"), ObjectId("6054b84e426ce500b57494a8"), ObjectId("6054b85c426ce500b57494b3"), ObjectId("6054b867426ce500b57494be"), ObjectId("6054b873426ce500b57494c9"), ObjectId("6054b87e426ce500b57494d4"), ObjectId("6054b88e426ce500b57494df"), ObjectId("6054b89a426ce500b57494ea"), ObjectId("6054b8a4426ce500b57494f5"), ObjectId("6054b8b3426ce500b5749500"), ObjectId("6054b8c4426ce500b574950b"), ObjectId("6054b8db426ce500b5749516") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
Mongoose: upload_file.find({ _id: { '$in': [ ObjectId("6054b823426ce500b5749483"), ObjectId("6054b823426ce500b5749484"), ObjectId("6054b82e426ce500b574948d"), ObjectId("6054b82f426ce500b5749490"), ObjectId("6054b83a426ce500b5749498"), ObjectId("6054b83c426ce500b574949b"), ObjectId("6054b845426ce500b57494a3"), ObjectId("6054b848426ce500b57494a6"), ObjectId("6054b857426ce500b57494ae"), ObjectId("6054b858426ce500b57494b1"), ObjectId("6054b861426ce500b57494b9"), ObjectId("6054b863426ce500b57494bc"), ObjectId("6054b86d426ce500b57494c4"), ObjectId("6054b870426ce500b57494c7"), ObjectId("6054b879426ce500b57494cf"), ObjectId("6054b879426ce500b57494d2"), ObjectId("6054b886426ce500b57494da"), ObjectId("6054b887426ce500b57494dd"), ObjectId("6054b894426ce500b57494e5"), ObjectId("6054b895426ce500b57494e8"), ObjectId("6054b8a0426ce500b57494f0"), ObjectId("6054b8a3426ce500b57494f3"), ObjectId("6054b8aa426ce500b57494fb"), ObjectId("6054b8ab426ce500b57494fe"), ObjectId("6054b8b9426ce500b5749506"), ObjectId("6054b8ba426ce500b5749509"), ObjectId("6054b8cc426ce500b5749511"), ObjectId("6054b8cd426ce500b5749514"), ObjectId("6054b8e5426ce500b574951c"), ObjectId("6054b8e7426ce500b5749520") ] }}, { skip: undefined, limit: undefined, perDocumentLimit: undefined, publicationState: undefined, projection: {}})
[2021-03-22T12:57:46.967Z] debug GET /merchant/adae0132-6ddd-4809-9447-4316cbd7e876/products (3294 ms) 200

I'm using MongoDB Atlas M0 in the same AWS region as API

[andriichern]

We faced the same issue in production. We can not add any items because Media Library count request makes the whole system frozen. In our case it takes the whole CPU resources and the only possible (and) temporary solution is to reload the whole strapi & mongo services. It definately should not work this way

[andriichern]

@alexandrebodin After porforming some investigation and trying to find temporary work around solution now I can tell you the reason of this issue. Moreover I can tell for sure that it does not matter what DB you use: MongoDB, PostgreSQl or other.

As far as this issue arised for our production users this was very blocking for us. Firstly, we decide to change DB from mongo to Postgres. Secondly, we decided to made our own load testing of strapi system with the new DB. So we put 20000 records for our collection, (yes, 20000, and yes - we expect to work with such a huge amounts of records in the future and even more). We also put a few images to Media Library, just 4 items, because we use them very actively in our strapi data collection. Each item of those 20000 was connected to 1 specific image from Media Library. So in the end we had 20000 items and 4 images, and every image is linked to ~5000 items.

When we opend Media Library page from the strapi admin panel we saw the same situation: images were fetching too long. Not so long as with the mongo, but on our testing system (which is in the cloud) it was ~10 second. And this delay was caused by the same request: GET /upload/files?_limit=10&_start=0&_sort=updatedAt:DESC.

Querying this request with Postman gives us the first understanding of the root cause. The resulted response contains ALL linked collection items for EVERY image presented in the response. In our case it was array of 4 media objects (images) and every media object contains ~5000 linked collection items. All this data is present under related field in the madia object. The response json was about 9.5 MB (!!!) of size 😳 and contained more than 300000 lines 😞. Even Postman feels bad displaying that response to us 😣

Going through the source code of a strapi-plugin-upload proved our consideration. in the File.settings.json model definition there is a field related:

"related": {
    "collection": "*",
        "filter": "field",
        "configurable": false
}

Thi is that same related field which we saw in the response cntained those huge amounts of data. And this is the field causing the slowdown for the whole Media Library.

P.S. To me it makes no sense to get all related items from all collections for media objects. Is it really necessary and what the purpose of it?

[alexandrebodin]

@alexandrebodin After porforming some investigation and trying to find temporary work around solution now I can tell you the reason of this issue. Moreover I can tell for sure that it does not matter what DB you use: MongoDB, PostgreSQl or other.

As far as this issue arised for our production users this was very blocking for us. Firstly, we decide to change DB from mongo to Postgres. Secondly, we decided to made our own load testing of strapi system with the new DB. So we put 20000 records for our collection, (yes, 20000, and yes - we expect to work with such a huge amounts of records in the future and even more). We also put a few images to Media Library, just 4 items, because we use them very actively in our strapi data collection. Each item of those 20000 was connected to 1 specific image from Media Library. So in the end we had 20000 items and 4 images, and every image is linked to ~5000 items.

When we opend Media Library page from the strapi admin panel we saw the same situation: images were fetching too long. Not so long as with the mongo, but on our testing system (which is in the cloud) it was ~10 second. And this delay was caused by the same request: GET /upload/files?_limit=10&_start=0&_sort=updatedAt:DESC.

Querying this request with Postman gives us the first understanding of the root cause. The resulted response contains ALL linked collection items for EVERY image presented in the response. In our case it was array of 4 media objects (images) and every media object contains ~5000 linked collection items. All this data is present under related field in the madia object. The response json was about 9.5 MB (!!!) of size 😳 and contained more than 300000 lines 😞. Even Postman feels bad displaying that response to us 😣

Going through the source code of a strapi-plugin-upload proved our consideration. in the File.settings.json model definition there is a field related:

"related": {
    "collection": "*",
        "filter": "field",
        "configurable": false
}

Thi is that same related field which we saw in the response cntained those huge amounts of data. And this is the field causing the slowdown for the whole Media Library.

P.S. To me it makes no sense to get all related items from all collections for media objects. Is it really necessary and what the purpose of it?

Have you tried extending the file model and adding autoPopulate: false ? Not sure this will work in this direction of the relation though but if not making a PR to allow this should be fairly easy

[andriichern]

@alexandrebodin After porforming some investigation and trying to find temporary work around solution now I can tell you the reason of this issue. Moreover I can tell for sure that it does not matter what DB you use: MongoDB, PostgreSQl or other.
As far as this issue arised for our production users this was very blocking for us. Firstly, we decide to change DB from mongo to Postgres. Secondly, we decided to made our own load testing of strapi system with the new DB. So we put 20000 records for our collection, (yes, 20000, and yes - we expect to work with such a huge amounts of records in the future and even more). We also put a few images to Media Library, just 4 items, because we use them very actively in our strapi data collection. Each item of those 20000 was connected to 1 specific image from Media Library. So in the end we had 20000 items and 4 images, and every image is linked to ~5000 items.
When we opend Media Library page from the strapi admin panel we saw the same situation: images were fetching too long. Not so long as with the mongo, but on our testing system (which is in the cloud) it was ~10 second. And this delay was caused by the same request: GET /upload/files?_limit=10&_start=0&_sort=updatedAt:DESC.
Querying this request with Postman gives us the first understanding of the root cause. The resulted response contains ALL linked collection items for EVERY image presented in the response. In our case it was array of 4 media objects (images) and every media object contains ~5000 linked collection items. All this data is present under related field in the madia object. The response json was about 9.5 MB (!!!) of size flushed and contained more than 300000 lines disappointed. Even Postman feels bad displaying that response to us persevere
Going through the source code of a strapi-plugin-upload proved our consideration. in the File.settings.json model definition there is a field related:

"related": {
    "collection": "*",
        "filter": "field",
        "configurable": false
}

Thi is that same related field which we saw in the response cntained those huge amounts of data. And this is the field causing the slowdown for the whole Media Library.
P.S. To me it makes no sense to get all related items from all collections for media objects. Is it really necessary and what the purpose of it?

Have you tried extending the file model and adding autoPopulate: false ? Not sure this will work in this direction of the relation though but if not making a PR to allow this should be fairly easy

@alexandrebodin autoPopulate: false option for file model did the job. Applying it removes related field from response and now Media Library is loading very fast

[andriichern]

@alexandrebodin I've added a PR for this fix

[derrickmehaffy]

That is breaking changes, it's unlikely we would merge that by default in the v3 as we already plan to disable all population in the v4. For now this would likely need to remain on the user's side to implement as you have done but I'll let @alexandrebodin confirm.