Ensure Folder Async hits the Sharepoint Limit of 5000

[ChrTall]

Category

• Bug

Describe the bug

If you use EnsureFolderAsync for a Folder in a large document Library that has more than 5000 SubFolders a SPQueryThrottledException is thrown, because we hit the 5000 ListItem Limit of Sharepoint Online, which can not be increased, can it?

Steps to reproduce

You need a large document Library with lots of SubFolders, in order to hit the 5000 ListItem Limit.
Load the Folder into a variable and execute EnsureFolderAsync("TestFolder/Test/Test").

1. See code var site = await targetSiteContext.Web.Lists.GetByTitleAsync("VeryBigLib"); var rootFolder = await site.RootFolder.GetAsync(); await rootFolder.EnsureFolderAsync("/FolderWithManySubFolders/TEST/TEST/TEST");

Expected behavior

It would be great if this could be implemented in a way that you can rely on it even when your library grows over time and eventually will hit the limit.

Environment details (development & target environment)

• SDK version: 1.5.0
• OS: Windows 10
• SDK used in: Console App and Asp.Net Core
• Framework: .NET 6
• Browser(s): --
• Tooling: Jetbrains Rider

[MathijsVerbeeck]

@jansenbe I've replicated this issue and the error happens in the function "EnsureFolderAsync" in Folder.cs on line 324 when the subfolders are being loaded. The error being thrown is that the list view item limit has been exceeded. What we might do is check the value of "ItemCount" of the currentfolder and when this exceeds 5000 we'll have to make an adjustment to load the subfolders on a different way than using the REST API call.

[Ironbell]

@MathijsVerbeeck @ChrTall as we had this problem as well with the current inefficient implementation of EnsureFolderAsync, see my workaround here:

private static async Task<IFolder> EnsureFolderAsync(string directoryPath, IFolder rootFolder)
        {
            var currentFolder = rootFolder;

            var childFolderNames = directoryPath.Split(new[] { '/' }, StringSplitOptions.RemoveEmptyEntries);

            var context = rootFolder.PnPContext;

            await rootFolder.LoadAsync(p => p.ServerRelativeUrl);

            var currentUrl = rootFolder.ServerRelativeUrl;
            var currentFolderWasCreated = false;

            foreach (var folderName in childFolderNames)
            {
                currentUrl = $"{currentUrl}/{folderName}";

                if (!currentFolderWasCreated)
                {
                    try
                    {
                        var nextFolder = await context.Web.GetFolderByServerRelativeUrlAsync(currentUrl);

                        currentFolder = nextFolder;
                    }
                    catch (PnP.Core.SharePointRestServiceException)
                    {
                        currentFolderWasCreated = true;
                        currentFolder = await currentFolder.AddFolderAsync(folderName);
                    }
                }
                else
                {
                    currentFolder = await currentFolder.AddFolderAsync(folderName);
                }
            }

            return currentFolder;
        }

Hope it helps!
Cheers,
Isabelle

[jansenbe]

@Ironbell : excellent approach! I'll update our implementation with your code. Thanks for sharing the fix :-)

[jansenbe]

@Ironbell : just integrated and tested your code, works like a charm in our test cases! @ChrTall : the updated implementation will be part of the next nightly release (version 1.5.99 or higher). Will be closing this issue now, but don't hesitate to re-open in case you still see issues with the fix.