[ BUG ] Get-FalconVulnerability fails with 401: access denied, invalid bearer token when retrieving large numbers of results

[Minty123]

Describe the bug
Hello! I'm trying to fetch a large amount of vulnerability data via Get-FalconVulnerability, enriched with facet data. The verbose output shows many Invoke-Loops, so pagination seems to work well. At one point, X-Ratelimit-Limit jumps from 6000 to 15. After that, the script fails with an error code 401, access denied, invalid bearer token. The inner exception says "results limited by API.

To Reproduce

1. Run Get-FalconVulnerability -Filter "status:!'closed'+host_info.groups:['$ID']" -Detailed -Verbose -All -Limit 1000 -Facet host_info, cve, remediation, where $ID is the ID of a host group.
2. In this example, there are 800k results. (I know...), so it takes a while.
3. At one point a certain limit is reached and the command fails:

...
VERBOSE: [Invoke-Loop] 484000 of 832056
VERBOSE: [Write-Result] Transfer-Encoding=chunked, Connection=keep-alive, Strict-Transport-Security=max-age=15724800; includeSubDomains max-age=31536000; includeSubDomains, X-Cs-Regi
on=eu-1, X-Cs-Traceid=249b14a1-63a3-4c01-a4a0-4c409a552682, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5989, Date=Thu, 27 Jan 2022 15:35:51 GMT, Server=nginx, meta.query_time=0.28
9215588, meta.pagination.limit=1000, meta.pagination.total=832056, meta.pagination.after=WyI2OWZmY2EzYzY5NGE0MzhmYmM0Nj...yOWYzYTU0YmU2MmY4NmY0MjJiMDAzNSJd,
 meta.powered_by=spapi
VERBOSE: [Invoke-Loop] 485000 of 832056
VERBOSE: [Write-Result] Transfer-Encoding=chunked, Connection=keep-alive, Strict-Transport-Security=max-age=15724800; includeSubDomains max-age=31536000; includeSubDomains, X-Cs-Regi
on=eu-1, X-Cs-Traceid=76d578c6-f7e5-4d1e-ac89-07140c6d6f5c, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5988, Date=Thu, 27 Jan 2022 15:35:55 GMT, Server=nginx, meta.query_time=0.13
7268172, meta.pagination.limit=1000, meta.pagination.total=832056, meta.pagination.after=WyI2OWZmY2EzYzY5NGE0MzhmYmM0Nj...yOWYzYTU0YmU2MmY4NmY0MjJiMDAzNSJd,
 meta.powered_by=spapi
VERBOSE: [Invoke-Loop] 486000 of 832056
VERBOSE: [Write-Result] Transfer-Encoding=chunked, Connection=close, Strict-Transport-Security=max-age=15724800; includeSubDomains max-age=31536000; includeSubDomains, X-Cs-Region=eu
-1, X-Cs-Traceid=65e2364a-47c6-4cc4-91b1-db5c6ee67f75, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5988, Date=Thu, 27 Jan 2022 15:35:58 GMT, Server=nginx, meta.query_time=0.3373669
13, meta.pagination.limit=1000, meta.pagination.total=832056, meta.pagination.after=WyI2OWZmY2EzYzY5NGE0MzhmYmM0Nj...yOWYzYTU0YmU2MmY4NmY0MjJiMDAzNSJd, meta
.powered_by=spapi
VERBOSE: [Invoke-Loop] 487000 of 832056
VERBOSE: [Write-Result] Connection=keep-alive, X-Content-Type-Options=nosniff, X-Ratelimit-Limit=15, X-Ratelimit-Remaining=14, Strict-Transport-Security=max-age=31536000; includeSubD
omains, Date=Thu, 27 Jan 2022 15:36:03 GMT, Server=nginx, meta.query_time=1.28E-07, meta.powered_by=crowdstrike-api-gateway
Write-Result : [{"code":401,"message":"access denied, invalid bearer token"}]
At C:\Program Files\WindowsPowerShell\Modules\PSFalcon\2.1.6\Private\Private.ps1:625 char:27
+                 $Result = Write-Result $Request
+                           ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (System.Threadin...esponseMessage]:Task`1) [Write-Result], Exception
    + FullyQualifiedErrorId : Write-Result
 
Invoke-Loop : [Invoke-Loop] Results limited by API '/spotlight/combined/vulnerabilities/v1' (487000 of 832056).
At C:\Program Files\WindowsPowerShell\Modules\PSFalcon\2.1.6\Private\Private.ps1:565 char:33
+                                 Invoke-Loop @Param
+                                 ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Loop

Expected behavior
Even though there's a large number of results, I'd expect it to run through, given enough time. Not sure if the error message is really correct here, as error 429 would indicate "The allowed rate limit has been reached". With 401 I assume that the bearer token expires in the meantime and is not properly refreshed?

Environment (please complete the following information):

• OS: Server 2016
• PowerShell: 5.1.14393.4583
• PSFalcon: 2.1.6

Additional context
Trying different values for the limit param didn't help. Getting the vulnerability IDs only works. I'd have to get info about hosts, remediations, etc. too, so I thought the facet param would be a good idea. Might not be the best way to go about it, but I'm just curious about the error.

Any help or hint is appreciated. Thanks again for all your time and effort in this project! :-)

[bk-cs]

I agree, it sounds like the error is due to the expiration of the bearer token. I'm guessing it's because the amount of time between the check of whether or not the bearer token is expired and the finishing of the request is longer than the default amount of time (15 seconds).

In v2.1.7 (the release I just published), I upped the time of that to 60 seconds. Could you retry with v2.1.7 and let me know if that solves the problem?

[Minty123]

Unfortunately with 2.1.7 the issue still exists. :-( I thought about a potential workaround by just getting the vulnerability IDs and pipe it to Get-FalconVulnerability:
Get-FalconVulnerability -Filter "status:!'closed'+host_info.groups:['$GroupID']" -Verbose -All -Limit 400 | ForEach-Object {Get-FalconVulnerability -Ids $_}

This however takes forever to finish. Maybe I could get around this by slicing the IDs to have workable batches, though I would still need to try that.

[bk-cs]

Would it work if we greatly increased the token renewal time? Try changing this line in Private\Private.ps1 under the process block in the Invoke-Falcon (around line 524) from

if ($Script:Falcon.Expiration -le (Get-Date).AddSeconds(60)) {

to

if ($Script:Falcon.Expiration -le (Get-Date).AddSeconds(600)) {

If that doesn't solve it, maybe break it into two steps? Note that you'll likely need to mess with the output to get readable content in the resulting CSV, but I don't think Export-FalconReport will work properly in this context (because it won't append after the first round of exports).

$Ids = Get-FalconVulnerability -Filter "status:!'closed'+host_info.groups:['$GroupID']" -All -Verbose
for ($i = 0; $i -lt ($Ids | Measure-Object).Count; $i += 500) {
    Get-FalconVulnerability -Ids $Ids[$i..($i + 499)] | Export-Csv .\Vulnerabilities.csv -NoTypeInformation -Append
}

This might hint that the -Detailed functionality might deserve a redesign for really large result sets.

[Minty123]

Interesting! After updating to PSFalcon 2.1.7 and increasing the expiration to 600 seconds, it still leads to the same error. Your second snippet throws an error when Get-FalconVulnerability is used without -Detailed:

Write-Result : [{"code":400,"message":"5000 is an invalid page size, must be between 1 and 400"}]
At C:\Program Files\WindowsPowerShell\Modules\PSFalcon\2.1.7\Private\Private.ps1:547 char:35
+                         $Result = Write-Result -Request $Request
+                                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (System.Threadin...esponseMessage]:Task`1) [Write-Result], Exception
    + FullyQualifiedErrorId : f3cf1b44-eebc-4564-84c4-2cafd679ed9e,Write-Result

When adding a -Limit 400, I can get the IDs just fine. Not sure if this is an API limitation or part of PSFalcon.

I really like the "two-step" approach though as pulling everything at once is hard on memory consumption anyway, with this large set of a data set. I'll play around with it a bit and report back later. :-)

Edit:

Finally some good news! With the two-step approach I can get all 800k entries into a CSV.
Started at: 02 Feb 2022 12:08:59 GMT
Ended at: 02 Feb 2022 12:49:50 GMT

Memory usage is still high, but at least it's finishing in an acceptable time.

[bk-cs]

-All checks the command you're running for the maximum limit and automatically inserts it. It seems like that's not working properly for Get-FalconVulnerability, so I can take a look.

I'm glad it worked for you! I'll see if I can think of a way to break up the large result sets or make them work differently than how -Detailed usually works (get the max number of ids, then submit them for details in groups of 500 and repeat).

[Minty123]

Alright, great! If you want we can close the issue for now, as I have a good workaround now. If you'd like me to test anything for you, just let me know. Thanks so much for your work on this project!

[bk-cs]

This issue will also manifest when using Find-FalconDuplicate or Get-FalconHost if your token expires during the retrieval of large result sets. I expect that this is fixed in the v2.2.2 release with changes that were made to Invoke-Falcon and Invoke-Loop.

[bk-cs]

Closing issue--resolved in 2.2.2 release.