Listing metadata for retreival takes too long

[Irritator]

Hi! I have an issue with metadata retrieval:
If I uncheck "subscribed only" in the retrieval menu, metadata listing takes about 5 minutes to complete, regardless I have cached metadata for that connection.

The project has a lot of metadata, but it still looks like a bug.

Please tell me if I need to share any logs

Also it would be nice to be able to retrieve newly created metadata without refreshing the whole metadata cache. I'm currently retrieving it by name with sf CLI, I think IC2 could have an additional retrieve button that does the same.

Thanks

[kaboomkazoom]

We also have a massive org with a lot of metadata. I also experience the same issue. It takes a lot of time for these operations.

Opening Configure Module or module content subscription editor brings up the "Finding subscribed-only metadata" dialog which takes a few minutes.
Clicking on "Update now" to refresh cached metadata is even slower.

Both these behaviours happen every time, not just the first time.

[SCWells72]

Hi. I'd probably need to see a debug log from when IC is enumerating org metadata. Can you please add the following to Help | Diagnostic Tools | Debug Log Settings:

com.illuminatedcloud.intellij.metadata.configCache.FilePropertiesCache

then refresh the metadata cache, and finally email the resulting idea.log file to me for review?

[SCWells72]

I've attached a new build here:

https://github.com/RoseSilverSoftware/IlluminatedCloud/releases/tag/2979

Please install it, re-enable .forceignore processing in settings, and let me know how things look. IC should be resolving .forceignore files via an index instead of a search, and it should be caching the .forceignore files that apply to each file so that it only has to resolve them once until/unless the set of .forceignore files for the containing module changes (not the file contents; the actual set of potentially-applicable files).

[kaboomkazoom]

This build did not fix the issue.

I installed the new build, restarted the IDE a couple of times, re-enable .forceignore processing in settings. It still keeps finding subscribed-only metadata, every time.

[SCWells72]

I'll need to see a new debug log with the following classes enabled for debugging:

com.illuminatedcloud.intellij.builder.subscription.settings.module.ModuleSubscriptionEditor
com.illuminatedcloud.intellij.sfdx.forceignore.ForceIgnoreUtil

[kaboomkazoom]

New log files sent to the support email

[SCWells72]

New build attached that should avoid lookup and caching in the simple/standard case of a single .forceignore file in the project root directory. Note that none of this helps with refreshing the metadata cache from the org, only with building the metadata selection tree locally in the subscription editor.

[SCWells72]

I've attached a pre-release test build here:

https://github.com/RoseSilverSoftware/IlluminatedCloud/releases/tag/2979

It is basically the most recent released build with additional diagnostic/timing logging in the area that seems to be taking up the remaining time.

Please download it and install it using Settings | Plugins | Install plugin from disk (under the gear drop-down menu). Change the contents of Help | Diagnostic Tools | Debug Log Settings to be only:

com.illuminatedcloud.intellij.builder.subscription.settings.module.ModuleSubscriptionEditor

Then reproduce the behavior and send over the resulting idea*.log files for review.

Also please let me know if disabling .forceignore processing makes any difference whatsoever.

[SCWells72]

Well, if you're using .forceignore to filter out actual metadata types/objects, those will still show up and be processed by deployment/retrieval/etc. operations. And I'm still going to try to tune this more, but I'm guessing further improvements will be smaller since applying a .forceignore filter against hundreds of thousands of metadata objects like in this project just takes time.

[SCWells72]

New build attached to the same link. I'm not actually expecting much of an improvement from this, but I've changed the data structure used for the metadata index to be a bit simpler since that was the main remaining hotspot in the last provided snapshot:

I'll be curious to hear whether or not it actually makes a difference. I'd love to see a new snapshot if you don't mind.

[kaboomkazoom]

hanks for the new build. I've performed the same steps as before. A new CPU profiling snapshot has been emailed.

[SCWells72]

Thanks for the new build. I've performed the same steps as before. A new CPU profiling snapshot has been emailed.

Appreciate it. Not much of a difference...could just be noise between samples. The good news is that the optimizations that have been implemented do make a noticeable difference overall; the bad news is that I'm not seeing much more opportunity for improvement in an org this large. I'll keep thinking about other ways to optimize it. The existing changes will go into this Thursday's build pending testing.

[kaboomkazoom]

Perfect! Thanks.

[SCWells72]

All discussed changes delivered in 2.4.1.4.

[kaboomkazoom]

Hello Scott, I can confirm that the 2.4.1.4 release is significantly better than the pre-release test build.

With the pre-release build, the subscription editor was taking 20-25 seconds, however, with 2.4.1.4 it is taking 5-7 seconds only. That makes a huge difference.

Thanks again!

[SCWells72]

With the pre-release build, the subscription editor was taking 20-25 seconds, however, with 2.4.1.4 it is taking 5-7 seconds only. That makes a huge difference.

That's wonderful! There were a few other seemingly small optimizations that I worked in since the last test build I posted. Sounds like they were more than just "seemingly small"! Thanks very much for all the diagnostic information -- and more importantly, time -- that you provided to help characterize these issues.