-
Notifications
You must be signed in to change notification settings - Fork 662
Get-PnPRecycleBinItem exceeds the list view threshold #2315
Comments
Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible. |
Is this being looked at yet? It's been a month and a half and this module is simply 100% broken and can no longer even so much as get a COUNT of recycle bin items etc. |
Hello @CollinChaffin , That said, i added a parameter to allow you to limit the amount of returned results, which you should be able to use to retrieve items without the service timing out for you. Just use the -RowLimit parameter for this function. I recommend 5000 to 10000 items to return, that seems to be the sweet spot - depending on how the tenant is doing you may be able to retrieve more (but, again, performance will vary tenant by tenant). Hope it helps! |
Hi @MrDoNotBreak , Two questions for you. Do you know when we will get the update to see this fix? As I understand, we just will get top 5000-10000 results. Is there any pagination, to get all result by 5-10k elements on page? |
@MrDoNotBreak this problem also seems to apply to Restore-PnPRecycleBinItem, for which there is no limiting parameter. For example, if I try: Get-PnPRecycleBinItem -SecondStage -RowLimit 4500 | ? {($_.DeletedByEmail -eq 'user@domain.com')} | Restore-PnPRecycledBinItem -Force Then errors are thrown: |
@MrDoNotBreak thank you for putting the restriction on the get-pnprecyclebinitem. Unfortunately I am having the same issue as @6079-Winston-Smith where a client has accidentally deleted a massive amount of files and the restore function fails because of the 5000 limit, regardless of how few files are piped to it. Even if I have to restore items 5000 at a time it would be a big help to have -RowLimit on the restore function. The only other alternative is to do a full library restore and lose the day's work. |
@Brendenawit I have come across the same limitation recently. It seems to be that the Restore-PnpRecycleBinItem does a call to get-pnpRecycleBinItem in the code, but this code doesn't have a Row Limit on it. |
Clear-PnPRecycleBinItem has the same problem, so I've added support for the RowLimit parameter to it. Would this be useful to make a PR for? |
All recycle bin cmdlets should have the RowLimit parameter in the July 2020 release. |
And when can we expect that release? I am just trying to restore files and hitting this error. |
@PrzemyslawKlys, here's a custom build which includes this fix already, if you want to test it out already: SharePointPnPPowerShellOnline.zip Official July 2020 release will be somewhere between next Friday and the end of next week. Exact date yet to be defined. |
July 13th will be the release date |
Thank you! |
Perhaps if I have to ask this question I shouldn't be testing your custom build, but if I've installed the module from the gallery, how do I incorporate the changes from your zip file above? I've tried copying the folder over to C:\Program Files\WindowsPowerShell\Modules but I still don't see the -RowLimit option on Restore-PnPRecycleBinItem. TIA |
Questions are always welcome @Vindicator007 . As I think more people will have the same question, I have just written a wiki page outlining how you can make use of this or other custom builds which we share in the issue topics: https://github.com/pnp/PnP-PowerShell/wiki/Using-custom-PnP-PowerShell-builds |
@KoenZomers Thanks for the wiki, I was off by one directory, but after uninstall and following your post it still doesn't seem to be working for me. It acts like an untrusted script whenever I attempt to use tab completion, prompting me to allow it to run even though I tried setting execution policy to unrestricted. Maybe it's me, but I thought I would let you know. |
@Vindicator007 You may need to right-click on the file in explorer, properties, and check if you need to unblock them. It's often like that when downloaded from internet directly. |
@KoenZomers any reason why pipeline doesn't work
It doesn't say it accepts pipeline like in other cmdlets, but examples show it's normally used that way.
Did something change? |
Ok, so the reason for the above is, RowLimit was added, but only if you don't use Identity. This makes no sense :-( |
@PrzemyslawKlys Unblock did the trick, thanks. Now I'm getting the same thing you are, that it can't take pipeline input. |
@PrzemyslawKlys that sounds exactly like what I said:
Can you share an exact PnP PowerShell line which returns an error for you? |
I now notice there have been more replies. The sample you're sharing is the one I used and worked for me. I'll give it another try. Waiting now for my script to complete which deletes the 5050 items into the recycle bin so I can try it again. |
Or
But you can easily see the problem by simply typing the command and trying to find -Identity parameter while RowLimit is set. |
I shouldn't need RowLimit for Restore-PnPRecycleBin at all in this case, as I am restoring just 5 elements, that I already know/have, as Get command provided them to me. |
You would need it on the Get, but not on the Restore, so this script is exactly as you would like to see it, correct? I agree that this script should work. Still waiting for the deletes to be done to try it out. |
None of the options I provided worked without RowLimit. With old version there's RowLimit on Get, and there's no RowLimit on Restore and I wasn't able to restore 70 items that sit between 22000 and 24000. In new version RowLimit is there but it removes the Identity parameter once it's used. Make sure it works this way as well, as it's valid use case to first limit output, verify it and then pass to restore. $Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5
$Limited | Restore-PnPRecycleBinItem Optionally if possible that would be good as well as I often skip pipelines $Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5
Restore-PnPRecycleBinItem -Identity $Limited |
@PrzemyslawKlys Got a filled up recycle bin again :) Just curious, does this throw an exception for you already? |
Your test is also working fine here: This will not be possible: Restore-PnPRecycleBinItem -Identity $Limited As that would mean it would need to accept an array, which would be different from all other cmdlets. So we will have to stick with the pipe which basically for-eaches through the collection. Strange that it doesn't work for you. Are you sure you're using my custom build? |
Some thresholds are only starting to kick in at 20,000 items. I'm wondering if that may be making the difference here as you mentioned to have a huge recycle bin? |
Yes, your custom build for sure. PS C:\Support\GitHub\O365Essentials> get-module SharePointPnPPowerShellOnline | fl
Name : SharePointPnPPowerShellOnline
Path : C:\Support\GitHub\O365Essentials\SharePointPnPPowerShellOnline\SharePointPnP.PowerShell.Online.Commands.dll
Description : SharePoint Patterns and Practices PowerShell Cmdlets for SharePoint Online
ModuleType : Binary
Version : 3.22.2006.2
NestedModules : {}
ExportedFunctions :
ExportedCmdlets : {Add-PnPAlert, Add-PnPApp, Add-PnPApplicationCustomizer, Add-PnPClientSidePage...}
ExportedVariables : WebRequestCounter
ExportedAliases : {Connect-PnPHubSite, Disconnect-PnPHubSite, Ensure-PnPFolder, Execute-PnPQuery...} Most likely your tenant has higher thresholds limit then mine, that's why you don't get the error. I am not sure how I can change the view. I don't even know what size of recycle bin I have, but at 150k items in 2nd stage i was still able to get more. |
What if you would try doing something like this: $Stage = Get-PnPRecycleBinItem -RowLimit 15000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem -RowLimit 150 Surely it should give you an error because of RowLimit and Identity at the same time. |
In my case, the first stage recycle bin has about 120000 items and I'm attempting to restore about about 107000 of those. I'm able to filter and retrieve the items, but when I try to pass that to the restore command it fails. $BinItems = Get-PnPRecycleBinItem -RowLimit 120000 -FirstStage | I've tried several different variations at this point for the restore with different for, foreach, etc. I'm losing track. lol For example, I tried: With your new version I tried adding the -RowLimit to the Restore-PnPRecycleBinItem in a few different scenarios, but still no luck. |
Confirming that the o365 sharepoint view list threshold is 5000. The difference in results we're seeing here would be neatly explained if @KoenZomers is using a sharepoint server with a higher limit. |
I'm also testing against a normal multi tenant, so there should be no difference in threshold levels. I've just generated another 16.000 documents in my library and am running a script now to throw all of them into the recycle bin. I did already get an error running just |
Passed the 20.000 items in my recycle bin and it still works fine I'm starting to think my custom build perhaps doesn't contain the proper updates. I have compiled a new build. Can you guys try this version and see if the results are different for you with this build? |
The RowLimit implementation seems to have paging built in, so |
But Get command is already released with RoWLimit. Restore is broken and releasing it as is makes no difference right? You can't specify RowLimit and Identity at same time. This means you can restore all of it or none of it, unless you are under 5000 limit. I dont understand why would you create 2 parameter sets where you need it in all cases. |
The release updates |
By all means, I am not blocking. Release away. You can release daily as far as I am concerned (that's what I do with my PowerShell modules). |
I don't think anyone is trying to block this release. I'm just struggling with why it doesn't run into thresholds on my environment which makes it hard for me to fix what you guys are running into. Can't fix what's not broken. Can any of you by chance file a PR where it is fixed in the way you envision it needs fixing? |
I am running into the exact same issue with We have a SharePoint Online customer with over 107,000 items in the first stage recycle bin, and we need to restore over 24,000 of them that were deleted yesterday. I cannot run
I can use the -RowLimit parameter to successfully perform the Get command. Here's some simple commands that illustrate this.
However, I cannot find any way to get the
I don't understand why the Is there a fix since the discussion stopped 11 days ago? |
I found an alternative way to restore the recycle bin items in a case like I'm running into where there are over 100K items in the recycle bin. The author in this blog post figured out the API calls that SharePoint makes when restoring a file from the web and found a way to replicate it using the I was able to pass in the Id's from my list of files to restore that I had retrieved using the |
@bierlyt glad you found my blog post. You can also make a single call and have the body contain a collection of Ids. The blog post is just an example, but in the production environment where I'm running a similar piece of code to this, I am passing in 200 Ids at a time. So the body would just have the collection of Deleted Object Id's. Example of 5 shown below. $body = "{""ids"":[""Id1"",""Id2"",""Id3"",""Id4"",""Id5""]}" |
pmatthews05, thanks for the information! You're a life-saver. I didn't even realize the blog post was linked from this thread or was written by you. I found it after hours of Googling and trying other CSOM attempts that ran into the same error message the |
@bierlyt It is a lot quicker when batching. I've put together the following Gist. https://gist.github.com/pmatthews05/83e3fb4471a1e82187d4f8d648068190 Please note, I haven't tested it, (although it looks ok) as I've taken bits of production code and my Restore-RecycleBinItems.ps1 code. The original production code I have was to clear the recycle bin, leaving x amount of days left in the recycle bin. |
Wow, it's a LOT faster when batching 200 at a time. Here's the code I whipped up on the fly to do it.
|
You know you can convert hashtable to JSON and skip this ugly bodybuilding right? $Ids = @(youridsgohere)
$MyCode = @{
ids = @($Ids)
}
$Body = $MyCode | ConvertTo-Json -Depth 5 |
Any fix to this yet? Still struggeling (with the clear-pnpR) here. PS C:\WINDOWS\system32> Get-PnPRecycleBinItem -RowLimit 10 | where {$_.DirName -like "l157"} | Clear-PnPRecycleBinItem Confirm
|
Reporting an Issue or Missing Feature
"Get-PnPRecycleBinItem" command on the recyclebin with a lot of items fails with "the list view threshold" error.
I have tried to filter results by following commands such as "Get-PnPRecycleBinItem | select -last 10" and "Get-PnPRecycleBinItems | ? DeletedDate -gt $today" however, these commands did not help. I got the same error.
This worked for recycle bin with more than 40k items some time ago.
Is there any workaround solution?
Is it possible to fix this trouble in PnP-PowerShell module?
Expected behavior
Restored files/files list
Actual behavior
Error message:
"Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by
the administrator.
At line:1 char:2
Steps to reproduce behavior
Connect-PnPOnline -Url https://...
(Get-PnPRecycleBinItem).count
Which version of the PnP-PowerShell Cmdlets are you using?
What is the version of the Cmdlet module you are running?
3.1.1809.0
How did you install the PnP-PowerShell Cmdlets?
The text was updated successfully, but these errors were encountered: