Eclipse becomes unresponsive when trying to refresh a viewer with more than 30000 elements on one level

[raghucssit]

Description:
When we have a large number of files inside a folder in an eclipse workspace refreshing the folder makes the UI freeze.
Large number may depend on machine configuration. As per my test if we have ~30000 files generated on the disk and if we try to refresh the folder under project eclipse freezes. It takes more than 30 seconds to populate tree items in the Package Explorer.

The reason behind this is viewer takes all the time to create Items.

Solution proposed: We will limit the number of Items created and add an Expandable Node. This Expandable Node on click populate next block of Limited elements with another Expandable Node. This way we can keep the Package Explorer responsive.

The goal is to minimize the effort client need to use this solution. So client can implement ILimitBasedContentProvider with the existing content provider and tell us the number items to show in the viewer. WrapperContentProvider is an example and can be used to wrap existing Content Provider.

Below is the screenshot how does it look if we have items limit = 10 and number of files on the file system is 30000.

[iloveeclipse]

Note: the main goal here is to provide a small extension to viewers / content providers API that would allow every existing viewer implementation (Package / Project Explorer, JUnit, Search view etc) that can have a huge number of elements to deal with that huge number of elements without freezing UI and without lot of internal code rework.

As the end result we (Advantest) plan to provide patches for all affected SDK views that we are aware of and where we saw freezes.

[iloveeclipse]

Sure, we can set the preference to zero to disable the feature for M3 nd mark API as provisional.
@raghucssit : please prepare PR.

[raghucssit]

Sure, we can set the preference to zero to disable the feature for M3 nd mark API as provisional. @raghucssit : please prepare PR.

ok. i will work on it.

[iloveeclipse]

Update: so far all known bugs were fixed and by default feature will be not enabled in M3 (preference need to be manually changed by used to enable the feature).

We (Advantest) will continue working on enabling this feature in 4.29 and on implementing reported enhancement requests. Everyone is welcome to help of course :-)

[Phillipus]

Thank-you everyone involved for your work and for listening to the community. :-)

[vogella]

Could you also add it to the N&N of 4.29 / 4.30 and describe how to enable it in the preferences? Otherwise it is impossible for users to find out of this new feature.

[vogella]

Ping for N&N entry. Without this entry it is almost for other projects to leverage this improvement in their viewers.

[vogella]

If I create a new project with 30 000 and open it in the project explorer it still freezes for a very long time.
@iloveeclipse you said it is already implemented in project explorer but as @raghucssit never responded to the request to document this feature in the N&N I don't see how to activate it.

Here is my test handler to generate example project.

package com.vogella.resources.handlers;

import java.io.ByteArrayInputStream;
import java.util.Random;

import org.eclipse.core.filesystem.EFS;
import org.eclipse.core.filesystem.IFileStore;
import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IFolder;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.resources.IResource;
import org.eclipse.core.resources.IWorkspace;
import org.eclipse.core.resources.IWorkspaceRoot;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.e4.core.di.annotations.Execute;
import org.eclipse.ui.IWorkbenchPage;

public class CreateLargeProjectHandler {
	private final Random random = new Random();

	private static final String CHARSFORCREATION = "abcdefghijklmnopqrstuvwxyz";

	@Execute
	public void execute(IWorkbenchPage page) {
		IWorkspace workspace = ResourcesPlugin.getWorkspace();
		IWorkspaceRoot root = workspace.getRoot();
		IProject project = root.getProject("performancetest");
		try {
			project.create(new NullProgressMonitor());
			project.open(null);
			for (int i = 0; i < 1; i++) {
				IFolder folder = project.getFolder("test" + i);
				folder.create(true, true, null);
				for (int j = 0; j < 30000; j++) {
					IFile file = folder.getFile(createString(10));
					file.create(new ByteArrayInputStream(createBytes(5000)), IResource.NONE, null);
					IFileStore fileStore = EFS.getLocalFileSystem().getStore(file.getFullPath());
					if (!fileStore.fetchInfo().isDirectory()) {
//						try {
//							IDE.openEditorOnFileStore(page, fileStore);
//						} catch (PartInitException e) {
//							/* some code */
//						}
					}
				}
			}
		} catch (CoreException e) {
			// nothing to do
		}
	}

	private byte[] createBytes(int length) {
		byte[] bytes = new byte[length];
		random.nextBytes(bytes);
		return bytes;
	}

	private String createString(int length) {
		StringBuilder buf = new StringBuilder(length);
		// fill the string with random characters up to the desired length
		for (int i = 0; i < length; i++) {
			buf.append(CHARSFORCREATION.charAt(random.nextInt(CHARSFORCREATION.length())));
		}
		return buf.toString();
	}

}

[iloveeclipse]

If I create a new project with 30 000 and open it in the project explorer it still freezes for a very long time.

See #1425, the feature was disabled by default.

[iloveeclipse]

N&N for 4.31 were added via eclipse-platform/www.eclipse.org-eclipse#104 .
With that and #1425 we are done for the main part of the ticket.
Whatever enhancement requests or bugs can be found, should go to dedicated tickets.