Extraction of all the endpoints and hosts is giving a 400 status error after retrieving 10000 assets.

[sivanand15]

I have used the following method:

falcon = Hosts(client_id=CLIENT_ID, client_secret=CLIENT_SECRET)
# Set the initial limit and offset
limit = 500
offset = 0
# Initialize an empty list to store all endpoints
all_endpoints = []
while True:
    # Query endpoints (devices) with the specified limit and offset
    response = falcon.query_devices_by_filter(limit=limit, offset=offset)
    # Check if any endpoints were retrieved in the response
    if 'resources' in response['body']:
        # Append the current batch of endpoints to the list
        all_endpoints.extend(response['body']['resources'])
    else:
        # No endpoints found, break the loop
        break
        # Update the offset to retrieve the next batch
        offset += limit
        # Save all endpoints to a JSON file
        with open('./json/all_endpoints.json', 'w') as outfile:
            json.dump(all_endpoints, outfile, indent=4)

Error after some 10000 odd records:

{
	'status_code': 400,
	'headers': {
		'Server': 'nginx',
		'Date': 'Fri, 16 Jun 2023 10:09:58 GMT',
		'Content-Type': 'application/json',
		'Content-Length': '206',
		'Connection': 'keep-alive',
		'Content-Encoding': 'gzip',
		'Strict-Transport-Security': 'max-age=15724800; includeSubDomains, max-age=31536000; includeSubDomains',
		'xxxxxx': 'xxxxxx',
		'X-Cs-Traceid': 'xxxxxx',
		'X-Ratelimit-Limit': '6000',
		'X-Ratelimit-Remaining': '5982'
	},
	'body': {
		'meta': {
			'query_time': 0.02051825,
			'powered_by': 'device-api',
			'trace_id': 'xxxxxx'
		},
		'resources': [],
		'errors': [{
			'code': 400,
			'message': 'limit + offset must be less than 10000'
		}]
	}
}

[crowdstrikedcs]

Hi @sivanand15 thanks for the question!

When using the query_devices_by_filter operation we are limited to paginating through the first 10,000 (150000 for US1) results that the API returns. I covered this in a bit of detail in another discussion here

It is recommended instead to use the query_devices_by_filter_scroll operation which allows for pagination using a token. This has a bit different of a pagination approach

//set these to initialize the loop
new_offset = ""
total_records = 1

while total_records > 0:
    response_from_falcon = falcon.query_devices_by_filter_scroll(offset=new_offset, limit=2500, sort="status.desc")
    device_ids= response_from_falcon["body"]["resources"]
    new_offset = response_from_falcon["body"]["meta"]["pagination"]["offset"]
    total_records = response_from_falcon["body"]["meta"]["pagination"]["total"]
    device_details = falcon.get_device_details(device_ids)

When the last page is returned, total_records becomes 0 and the loop ends.

Check out using this new operation and let us know with any questions.

[Rohan-Kar]

@crowdstrikedcs Thanks for the information. But I have couple of question.

1. i checked with operation the offset pointer which is expire after 2 minutes so if the data will large and unable to fetch in 2 minutes that scenario what we do.

2. one more question is try to get all asset using get_host it will only return the 100 of asset how to get the total number of asset .
   got the error "powered_by": "discover-api", "trace_id": "jjsdfbjfwefejfjsdjcsdnvksdgwegweg959" }, "errors": [ { "code": 400, "message": "200 is an invalid limit, must be between 1 and 100." }

Thank you