Skip to content
This repository has been archived by the owner on Jan 19, 2021. It is now read-only.

Get-PnPRecycleBinItem exceeds the list view threshold #2315

Open
4 tasks
DanielSanIT opened this issue Oct 16, 2019 · 59 comments · Fixed by #2760
Open
4 tasks

Get-PnPRecycleBinItem exceeds the list view threshold #2315

DanielSanIT opened this issue Oct 16, 2019 · 59 comments · Fixed by #2760
Assignees
Labels
status:fixed-next-drop Issue will be fixed in upcoming release.

Comments

@DanielSanIT
Copy link

DanielSanIT commented Oct 16, 2019

Reporting an Issue or Missing Feature

"Get-PnPRecycleBinItem" command on the recyclebin with a lot of items fails with "the list view threshold" error.

I have tried to filter results by following commands such as "Get-PnPRecycleBinItem | select -last 10" and "Get-PnPRecycleBinItems | ? DeletedDate -gt $today" however, these commands did not help. I got the same error.

This worked for recycle bin with more than 40k items some time ago.

Is there any workaround solution?
Is it possible to fix this trouble in PnP-PowerShell module?

Expected behavior

Restored files/files list

Actual behavior

Error message:
"Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by
the administrator.
At line:1 char:2

  • (Get-PnPRecycleBinItem).count
  •  + CategoryInfo          : WriteError: (:) [Get-PnPRecycleBinItem], ServerException
     + FullyQualifiedErrorId : EXCEPTION,SharePointPnP.PowerShell.Commands.RecycleBin.GetRecycleBinItems"
    
    

Steps to reproduce behavior

Connect-PnPOnline -Url https://...
(Get-PnPRecycleBinItem).count

Which version of the PnP-PowerShell Cmdlets are you using?

  • PnP PowerShell for SharePoint 2013
  • PnP PowerShell for SharePoint 2016
  • [ x] PnP PowerShell for SharePoint Online

What is the version of the Cmdlet module you are running?

3.1.1809.0

How did you install the PnP-PowerShell Cmdlets?

  • MSI Installed downloaded from GitHub
  • [ x] Installed through the PowerShell Gallery with Install-Module
  • Other means
@ghost
Copy link

ghost commented Oct 16, 2019

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

@ghost ghost added the Needs: Triage 🔍 label Oct 16, 2019
@CollinChaffin
Copy link

Is this being looked at yet? It's been a month and a half and this module is simply 100% broken and can no longer even so much as get a COUNT of recycle bin items etc.

@KrystianNiepsuj
Copy link

Hello @CollinChaffin ,
as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.

That said, i added a parameter to allow you to limit the amount of returned results, which you should be able to use to retrieve items without the service timing out for you. Just use the -RowLimit parameter for this function. I recommend 5000 to 10000 items to return, that seems to be the sweet spot - depending on how the tenant is doing you may be able to retrieve more (but, again, performance will vary tenant by tenant).

Hope it helps!

@DanielSanIT
Copy link
Author

Hi @MrDoNotBreak ,
Thanks for your fix.

Two questions for you.

Do you know when we will get the update to see this fix?

As I understand, we just will get top 5000-10000 results. Is there any pagination, to get all result by 5-10k elements on page?

@6079-Winston-Smith
Copy link

@MrDoNotBreak this problem also seems to apply to Restore-PnPRecycleBinItem, for which there is no limiting parameter. For example, if I try:

Get-PnPRecycleBinItem -SecondStage -RowLimit 4500 | ? {($_.DeletedByEmail -eq 'user@domain.com')} | Restore-PnPRecycledBinItem -Force

Then errors are thrown:
Restore-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

@Brendenawit
Copy link

Hello @CollinChaffin ,
as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.

@MrDoNotBreak thank you for putting the restriction on the get-pnprecyclebinitem. Unfortunately I am having the same issue as @6079-Winston-Smith where a client has accidentally deleted a massive amount of files and the restore function fails because of the 5000 limit, regardless of how few files are piped to it. Even if I have to restore items 5000 at a time it would be a big help to have -RowLimit on the restore function. The only other alternative is to do a full library restore and lose the day's work.

@pmatthews05
Copy link

pmatthews05 commented Feb 14, 2020

@Brendenawit I have come across the same limitation recently. It seems to be that the Restore-PnpRecycleBinItem does a call to get-pnpRecycleBinItem in the code, but this code doesn't have a Row Limit on it.
Until someone at PNP fixes this issue, (not meaning to promote myself, but it should help) I wrote a blog on this yesterday., with code to restore.

Professr pushed a commit to Professr/PnP-PowerShell that referenced this issue Jun 24, 2020
@Professr
Copy link

Clear-PnPRecycleBinItem has the same problem, so I've added support for the RowLimit parameter to it. Would this be useful to make a PR for?

@KoenZomers
Copy link
Collaborator

All recycle bin cmdlets should have the RowLimit parameter in the July 2020 release.

@PrzemyslawKlys
Copy link

PrzemyslawKlys commented Jun 30, 2020

And when can we expect that release? I am just trying to restore files and hitting this error.

@KoenZomers
Copy link
Collaborator

@PrzemyslawKlys, here's a custom build which includes this fix already, if you want to test it out already:

SharePointPnPPowerShellOnline.zip

Official July 2020 release will be somewhere between next Friday and the end of next week. Exact date yet to be defined.

@KoenZomers
Copy link
Collaborator

July 13th will be the release date

@PrzemyslawKlys
Copy link

Thank you!

@Vindicator007
Copy link

Vindicator007 commented Jun 30, 2020

Perhaps if I have to ask this question I shouldn't be testing your custom build, but if I've installed the module from the gallery, how do I incorporate the changes from your zip file above? I've tried copying the folder over to C:\Program Files\WindowsPowerShell\Modules but I still don't see the -RowLimit option on Restore-PnPRecycleBinItem. TIA

@KoenZomers
Copy link
Collaborator

Questions are always welcome @Vindicator007 . As I think more people will have the same question, I have just written a wiki page outlining how you can make use of this or other custom builds which we share in the issue topics:

https://github.com/pnp/PnP-PowerShell/wiki/Using-custom-PnP-PowerShell-builds

@Vindicator007
Copy link

@KoenZomers Thanks for the wiki, I was off by one directory, but after uninstall and following your post it still doesn't seem to be working for me. It acts like an untrusted script whenever I attempt to use tab completion, prompting me to allow it to run even though I tried setting execution policy to unrestricted. Maybe it's me, but I thought I would let you know.

@PrzemyslawKlys
Copy link

@Vindicator007 You may need to right-click on the file in explorer, properties, and check if you need to unblock them. It's often like that when downloaded from internet directly.

@PrzemyslawKlys
Copy link

@KoenZomers any reason why pipeline doesn't work

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Files = $Stage | Where-Object { $_.DirName -like "$Test*"} #| Out-HtmlView -ScrollX -DisablePaging

#foreach ($File in $Files) {
#    $File | Restore-PnPRecycleBinItem -RowLimit 150000
#}

$Files | Restore-PnPRecycleBinItem -RowLimit 150000
Restore-PnPRecycleBinItem : The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
At line:1 char:10
+ $Files | Restore-PnPRecycleBinItem -RowLimit 150000
+          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (Microsoft.Share....RecycleBinItem:PSObject) [Restore-PnPRecycleBinItem], ParameterBindingException
    + FullyQualifiedErrorId : InputObjectNotBound,SharePointPnP.PowerShell.Commands.RecycleBin.RestoreRecycleBinItem

It doesn't say it accepts pipeline like in other cmdlets, but examples show it's normally used that way.

get-help Restore-PnPRecycleBinItem -Full


NAME
    Restore-PnPRecycleBinItem

SYNOPSIS
    Restores the provided recycle bin item to its original location


SYNTAX
    Restore-PnPRecycleBinItem [-Force <SwitchParameter>] [-RowLimit <Int32>] [-Connection <PnPConnection>] [<CommonParameters>]

    Restore-PnPRecycleBinItem -Identity <RecycleBinItemPipeBind> [-Force <SwitchParameter>] [-Connection <PnPConnection>] [<CommonParameters>]


DESCRIPTION


PARAMETERS
    -Force [<SwitchParameter>]
        If provided, no confirmation will be asked to restore the recycle bin item

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -Identity <RecycleBinItemPipeBind>
        Id of the recycle bin item or the recycle bin item object itself to restore

        Required?                    true
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -RowLimit [<Int32>]
        Limits restoration to specified number of items

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    -Connection [<PnPConnection>]
        Optional connection to be used by the cmdlet. Retrieve the value for this parameter by either specifying -ReturnConnection on Connect-PnPOnline or by executing Get-PnPConnection.

        Required?                    false
        Position?                    named
        Default value
        Accept pipeline input?
        Accept wildcard characters?

    <CommonParameters>
        This cmdlet supports the common parameters: Verbose, Debug,
        ErrorAction, ErrorVariable, WarningAction, WarningVariable,
        OutBuffer, PipelineVariable, and OutVariable. For more information, see
        about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216).

INPUTS

OUTPUTS




    ------------------EXAMPLE 1---------------------

    PS:> Restore-PnpRecycleBinItem -Identity 72e4d749-d750-4989-b727-523d6726e442

    Restores the recycle bin item with Id 72e4d749-d750-4989-b727-523d6726e442 to its original location

    ------------------EXAMPLE 2---------------------

    PS:> Get-PnPRecycleBinItem | ? -Property LeafName -like "*.docx" | Restore-PnpRecycleBinItem

    Restores all the items in the first and second stage recycle bins to their original location of which the filename ends with the .docx extension

    ------------------EXAMPLE 3---------------------

    PS:> Restore-PnPRecycleBinItem -All -RowLimit 10000

    Permanently restores up to 10,000 items in the recycle bin


RELATED LINKS
    SharePoint Developer Patterns and Practices: http://aka.ms/sppnp

Did something change?

@PrzemyslawKlys
Copy link

Ok, so the reason for the above is, RowLimit was added, but only if you don't use Identity. This makes no sense :-(

@Vindicator007
Copy link

@PrzemyslawKlys Unblock did the trick, thanks. Now I'm getting the same thing you are, that it can't take pipeline input.

@KoenZomers
Copy link
Collaborator

@PrzemyslawKlys that sounds exactly like what I said:

Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem? In that case it would take -Identity as the parameter and that indeed does not allow a -RowLimit as it would only retrieve that single item anyway. Also that seems to be working fine here.

Can you share an exact PnP PowerShell line which returns an error for you?

@KoenZomers
Copy link
Collaborator

I now notice there have been more replies. The sample you're sharing is the one I used and worked for me. I'll give it another try. Waiting now for my script to complete which deletes the 5050 items into the recycle bin so I can try it again.

@PrzemyslawKlys
Copy link

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
foreach ($File in $Stage) {
       $File | Restore-PnPRecycleBinItem -RowLimit 150000
}

Or

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
foreach ($File in $Stage) {
      Restore-PnPRecycleBinItem -RowLimit 150000 -Identity $File
}

But you can easily see the problem by simply typing the command and trying to find -Identity parameter while RowLimit is set.

image

@PrzemyslawKlys
Copy link

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem

I shouldn't need RowLimit for Restore-PnPRecycleBin at all in this case, as I am restoring just 5 elements, that I already know/have, as Get command provided them to me.

@KoenZomers
Copy link
Collaborator

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem

I shouldn't need RowLimit for Restore-PnPRecycleBin at all in this case, as I am restoring just 5 elements, that I already know/have, as Get command provided them to me.

You would need it on the Get, but not on the Restore, so this script is exactly as you would like to see it, correct? I agree that this script should work. Still waiting for the deletes to be done to try it out.

@PrzemyslawKlys
Copy link

None of the options I provided worked without RowLimit. With old version there's RowLimit on Get, and there's no RowLimit on Restore and I wasn't able to restore 70 items that sit between 22000 and 24000. In new version RowLimit is there but it removes the Identity parameter once it's used.

Make sure it works this way as well, as it's valid use case to first limit output, verify it and then pass to restore.

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5 
$Limited | Restore-PnPRecycleBinItem

Optionally if possible that would be good as well as I often skip pipelines

$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5 
Restore-PnPRecycleBinItem -Identity $Limited

@KoenZomers
Copy link
Collaborator

@PrzemyslawKlys Got a filled up recycle bin again :) Just curious, does this throw an exception for you already?

image

@PrzemyslawKlys
Copy link

Yes

image

@KoenZomers
Copy link
Collaborator

Your test is also working fine here:

image

This will not be possible:

Restore-PnPRecycleBinItem -Identity $Limited

As that would mean it would need to accept an array, which would be different from all other cmdlets. So we will have to stick with the pipe which basically for-eaches through the collection.

Strange that it doesn't work for you. Are you sure you're using my custom build?

@KoenZomers
Copy link
Collaborator

Some thresholds are only starting to kick in at 20,000 items. I'm wondering if that may be making the difference here as you mentioned to have a huge recycle bin?

@PrzemyslawKlys
Copy link

Yes, your custom build for sure.

PS C:\Support\GitHub\O365Essentials> get-module SharePointPnPPowerShellOnline | fl



Name              : SharePointPnPPowerShellOnline
Path              : C:\Support\GitHub\O365Essentials\SharePointPnPPowerShellOnline\SharePointPnP.PowerShell.Online.Commands.dll
Description       : SharePoint Patterns and Practices PowerShell Cmdlets for SharePoint Online
ModuleType        : Binary
Version           : 3.22.2006.2
NestedModules     : {}
ExportedFunctions :
ExportedCmdlets   : {Add-PnPAlert, Add-PnPApp, Add-PnPApplicationCustomizer, Add-PnPClientSidePage...}
ExportedVariables : WebRequestCounter
ExportedAliases   : {Connect-PnPHubSite, Disconnect-PnPHubSite, Ensure-PnPFolder, Execute-PnPQuery...}

Most likely your tenant has higher thresholds limit then mine, that's why you don't get the error. I am not sure how I can change the view. I don't even know what size of recycle bin I have, but at 150k items in 2nd stage i was still able to get more.

@PrzemyslawKlys
Copy link

What if you would try doing something like this:

$Stage = Get-PnPRecycleBinItem -RowLimit 15000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem -RowLimit 150

Surely it should give you an error because of RowLimit and Identity at the same time.

@Vindicator007
Copy link

Vindicator007 commented Jul 2, 2020

In my case, the first stage recycle bin has about 120000 items and I'm attempting to restore about about 107000 of those. I'm able to filter and retrieve the items, but when I try to pass that to the restore command it fails.

$BinItems = Get-PnPRecycleBinItem -RowLimit 120000 -FirstStage |
Where-Object -Property Leafname -Like -Value "." |
Where-Object -Property DeletedDateLocalFormatted -Like -Value "6/26/2020*" |
Where-Object -Property Dirname -Like -Value "sites/MY-Site/Shared Documents/*" |
Where-Object -Property DeletedByEmail -EQ –Value John.Doe@contoso.com
$BinItems | Restore-PnPRecycleBinItem -Force

I've tried several different variations at this point for the restore with different for, foreach, etc. I'm losing track. lol For example, I tried:
foreach ($Item in $BinItems) {Restore-PnPRecycleBinItem -Identity $Item.Id -Force}

With your new version I tried adding the -RowLimit to the Restore-PnPRecycleBinItem in a few different scenarios, but still no luck.

@Brendenawit
Copy link

Confirming that the o365 sharepoint view list threshold is 5000. The difference in results we're seeing here would be neatly explained if @KoenZomers is using a sharepoint server with a higher limit.

@KoenZomers
Copy link
Collaborator

I'm also testing against a normal multi tenant, so there should be no difference in threshold levels. I've just generated another 16.000 documents in my library and am running a script now to throw all of them into the recycle bin. I did already get an error running just Get-PnPListItem now regarding the threshold, so perhaps that explains why it didn't give me any issues before. Will run the before mentioned tests on the recycle bin here once it's done throwing everything into the bin.

@KoenZomers
Copy link
Collaborator

Passed the 20.000 items in my recycle bin and it still works fine

image

image

I'm starting to think my custom build perhaps doesn't contain the proper updates. I have compiled a new build. Can you guys try this version and see if the results are different for you with this build?

SharePointPnPPowerShellOnline.zip

@PrzemyslawKlys
Copy link

Nope

image

@Professr
Copy link

Professr commented Jul 3, 2020

The RowLimit implementation seems to have paging built in, so Get-PnPRecycleBinItem -RowLimit 99999999 would theoretically work where Get-PnPRecycleBinItem would not. I don't know what magic happens in the RowLimit code to get single-item operations to work within its paged loop, but that magic is definitely not in place in these specific cmdlets - when the Identity parameter is specified, they just use the standard sharepoint single-item commands which do not work in environments over the list view threshold.
I think the code is important to include in the release even as-is, because without the ability to do bulk operations via RowLimit there's literally no way to bring a recycle bin back under that 5,000-item threshold for O365 tenants - not through the web interface, not through powershell.
The ability to work with individual items in an environment over that threshold is important, but less hair-on-fire IMO. Whoever understands how RowLimit does its thing (not me, I could barely get this far) will need to take that strategy and apply it to the single-item commands so that Identity will work too.

@PrzemyslawKlys
Copy link

The RowLimit implementation seems to have paging built in, so Get-PnPRecycleBinItem -RowLimit 99999999 would theoretically work where Get-PnPRecycleBinItem would not. I don't know what magic happens in the RowLimit code to get single-item operations to work within its paged loop, but that magic is definitely not in place in these specific cmdlets - when the Identity parameter is specified, they just use the standard sharepoint single-item commands which do not work in environments over the list view threshold.

I think the code is important to include in the release even as-is, because without the ability to do bulk operations via RowLimit there's literally no way to bring a recycle bin back under that 5,000-item threshold for O365 tenants - not through the web interface, not through powershell.

The ability to work with individual items in an environment over that threshold is important, but less hair-on-fire IMO. Whoever understands how RowLimit does its thing (not me, I could barely get this far) will need to take that strategy and apply it to the single-item commands so that Identity will work too.

But Get command is already released with RoWLimit. Restore is broken and releasing it as is makes no difference right? You can't specify RowLimit and Identity at same time. This means you can restore all of it or none of it, unless you are under 5000 limit. I dont understand why would you create 2 parameter sets where you need it in all cases.

@Professr
Copy link

Professr commented Jul 3, 2020

The release updates Clear-PnPRecycleBinItem, which would not work at all. That's crucial to letting people clear recycle bins over 5000 items. It also adds RowLimit to the other related cmdlets like Restore-PnPRecycleBinItem even though it doesn't support single-item operations yet, so before this release you couldn't do anything with a large bin except list the items in it and after this release you can delete or restore items from large bins wholesale.
Sure, there's more we need to do re: single-item operations, but that's no reason to not release an update that fixes a blocking bug with a core O365 user story :)

@PrzemyslawKlys
Copy link

By all means, I am not blocking. Release away. You can release daily as far as I am concerned (that's what I do with my PowerShell modules).

@KoenZomers
Copy link
Collaborator

I don't think anyone is trying to block this release. I'm just struggling with why it doesn't run into thresholds on my environment which makes it hard for me to fix what you guys are running into. Can't fix what's not broken. Can any of you by chance file a PR where it is fixed in the way you envision it needs fixing?

@bierlyt
Copy link

bierlyt commented Jul 17, 2020

I am running into the exact same issue with Restore-PnPRecycleBinItem, just like PrzemyslawKlys.

We have a SharePoint Online customer with over 107,000 items in the first stage recycle bin, and we need to restore over 24,000 of them that were deleted yesterday. I cannot run Get-PnPRecycleBinItem without the -RowLimit parameter or I get the error:

Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

I can use the -RowLimit parameter to successfully perform the Get command. Here's some simple commands that illustrate this.

$restoreSet = Get-PnPRecycleBinItem -RowLimit 150000
$restoreSet.Count ## 107540 results
$restoreDavid = $restoreSet | ?{ $_.DeletedByEmail -eq "Davidm@<domain>.com" -and $_.DeletedDate -gt "7/14/2020" }
$restoreDavid.count ## 24325 results

However, I cannot find any way to get the Restore-PnPRecycleBinItem to work. Either I get the "list view threshold" error, or I get a pipeline input error if I try to use -RowLimit at the same time. Here's 3 attempts, 2 with the pipeline and one passing in Identity.

$restoreDavid | Restore-PnPRecycleBinItem -Force ##Results in "list view threshold" error

$restoreDavid | Restore-PnPRecycleBinItem -Force -RowLimit 500 ##Results in a "pipeline input" error

$foreach($r in $restoreDavid){ Restore-PnPRecycleBinItem -Identity $r.Id.Guid -Force} ## Results in "list view threshold" error

I don't understand why the Restore-PnPRecycleBinItem cmdlet runs into the "list view threshold" error when simply trying to restore a single file that was passed in by Identity. But it appears that if Get-PnPRecycleBinItem without -RowLimit gets an error, then Restore-PnPRecycleBinItem without -RowLimit does too, and if there's enough items in the recycle bin, there's no way to restore a specific list of files passed in from the Get-PnPRecycleBinItem cmdlet.

Is there a fix since the discussion stopped 11 days ago?

@bierlyt
Copy link

bierlyt commented Jul 17, 2020

I found an alternative way to restore the recycle bin items in a case like I'm running into where there are over 100K items in the recycle bin. The author in this blog post figured out the API calls that SharePoint makes when restoring a file from the web and found a way to replicate it using the Invoke-PnPSPRestMethod cmdlet in the SharePointPnPPowerShellOnline module.

https://cann0nf0dder.wordpress.com/2020/02/13/viewing-restoring-and-removing-items-from-the-sharepoint-recycle-bin-the-attempted-operation-is-prohibited-because-it-exceeds-the-list-view-threshold-enforced-by-the-administrator/

image

I was able to pass in the Id's from my list of files to restore that I had retrieved using the Get-PnPRecycleBinItem' into the Restore-RecycleBinItem` function in the screenshot above, and it restored the files successfully.

@pmatthews05
Copy link

@bierlyt glad you found my blog post. You can also make a single call and have the body contain a collection of Ids. The blog post is just an example, but in the production environment where I'm running a similar piece of code to this, I am passing in 200 Ids at a time.

So the body would just have the collection of Deleted Object Id's. Example of 5 shown below.

$body = "{""ids"":[""Id1"",""Id2"",""Id3"",""Id4"",""Id5""]}"  

@bierlyt
Copy link

bierlyt commented Jul 17, 2020

pmatthews05, thanks for the information! You're a life-saver. I didn't even realize the blog post was linked from this thread or was written by you. I found it after hours of Googling and trying other CSOM attempts that ran into the same error message the Restore-PnPRecycleBinItem cmdlet runs into. Is it a lot faster passing 200 Ids instead of passing 1 Id at a time? It seems to take 2-3 seconds per file to restore 1 Id at a time. I have two PowerShell processes running restores from my sorted list, one from the top down and the other from the bottom up, but it will still probably take about 10 hours to finish. If it's faster, do you have a bit of code that demonstrates passing in an array of ID's and processing 200 Ids at a time? I could write some code to do it, but if you have a snippet available that'd be awesome. Thanks!

@pmatthews05
Copy link

@bierlyt It is a lot quicker when batching.

I've put together the following Gist. https://gist.github.com/pmatthews05/83e3fb4471a1e82187d4f8d648068190 Please note, I haven't tested it, (although it looks ok) as I've taken bits of production code and my Restore-RecycleBinItems.ps1 code. The original production code I have was to clear the recycle bin, leaving x amount of days left in the recycle bin.

@bierlyt
Copy link

bierlyt commented Jul 17, 2020

Wow, it's a LOT faster when batching 200 at a time. Here's the code I whipped up on the fly to do it.

$siteUrl = "https://<tenant>.sharepoint.com"
Connect-PnPOnline -Url $siteUrl
$restoreSet = Get-PnPRecycleBinItem -RowLimit 150000
$restoreSet.Count ## 107540 results
$restoreDavid = $restoreSet | ?{ $_.DeletedByEmail -eq "Davidm@<domain>.com" -and $_.DeletedDate -gt "7/14/2020" }
$restoreDavid.count ## 24325 results
# I restored all matching deleted folders first, and then restored files as shown below
$restoreDavidFileSorted = $restoreDavid | ?{$_.ItemType -eq "File"} | sort DirName, LeafName
$restoreDavidFileSorted.Count  ## 20761 results

# Batch restore up to 200 at a time
$restoreList = $restoreDavidFileSorted | select Id, ItemType, LeafName, DirName
$apiCall = $siteUrl + "/_api/site/RecycleBin/RestoreByIds"
$restoreListCount = $restoreList.count
$start = 0
$leftToProcess = $restoreListCount - $start
while($leftToProcess -gt 0){
    If($leftToProcess -lt 200){$numToProcess = $leftToProcess} Else {$numToProcess = 200}
    Write-Host -ForegroundColor Yellow "Building statement to restore the following $numToProcess files"
    $body = "{""ids"":["
    for($i=0; $i -lt $numToProcess; $i++){
        $cur = $start + $i
        $curItem = $restoreList[$cur]
        $Id = $curItem.Id
        Write-Host -ForegroundColor Green "Adding ", $curItem.ItemType, ": ", $curItem.DirName, "//", $curItem.LeafName
        $body += """" + $Id + """"
        If($i -ne $numToProcess - 1){ $body += "," }
    }
    $body += "]}"
    Write-Host -ForegroundColor Yellow $body
    Write-Host -ForegroundColor Yellow "Performing API Call to Restore items from RecycleBin..."
    try {
        Invoke-PnPSPRestMethod -Method Post -Url $apiCall -Content $body | Out-Null
    }
    catch {
        Write-Error "Unable to Restore"     
    }
    $start += 200
    $leftToProcess = $restoreListCount - $start
}

@PrzemyslawKlys
Copy link

You know you can convert hashtable to JSON and skip this ugly bodybuilding right?

$Ids = @(youridsgohere)
$MyCode = @{
    ids = @($Ids)
} 
$Body = $MyCode | ConvertTo-Json -Depth 5

@Conda86
Copy link

Conda86 commented Oct 21, 2020

Any fix to this yet? Still struggeling (with the clear-pnpR) here.

PS C:\WINDOWS\system32> Get-PnPRecycleBinItem -RowLimit 10 | where {$_.DirName -like "l157"} | Clear-PnPRecycleBinItem

Confirm
Permanently delete file '170102_52alene_UTM32_NN2000_20191106' from the recycle bin?
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y
Clear-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.
At line:1 char:74

  • ... imit 10 | where {$_.DirName -like "l157"} | Clear-PnPRecycleBinItem
  •                                               ~~~~~~~~~~~~~~~~~~~~~~~
    
    • CategoryInfo : WriteError: (:) [Clear-PnPRecycleBinItem], ServerException
    • FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.RecycleBin.ClearRecycleBinItem

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status:fixed-next-drop Issue will be fixed in upcoming release.
Projects
None yet