gcs_list_objects seems limited to return 1000 rows max

[G3rtjan]

Calling gcs_list_objects() on my Google Cloud bucket returns exactly 1000 rows, even after adding some additional files just to make sure that here is more in there than a 1000 files.

Maybe the maxResults parameter in one of the API calls you use can be used:
https://cloud.google.com/storage/docs/json_api/v1/objects/list

Perhaps this could be implemented as a parameter in the function :)

[MarkEdmondson1234]

@G3rtjan thanks, thats a bug. I'll look to add paging ASAP

[G3rtjan]

@MarkEdmondson1234 I've created a workaround for myself for the time being and in doing so I found out that:

• The maxResults parameter can only limit results BELOW 1000, it doesn't enable you to get more than 1000 rows ...
• However, within the first set of (1000) results a nextPageToken is included (within content$nextPageToken), which can be used in the next call for the pageToken parameter to get the next set of (1000) results
• You can continue this within a loop until the nextPageToken is NULL, which means you have all available results
• This way, I was able to get all +/- 40k results from my Google Cloud storage bucket

I hope this helps :)

[MarkEdmondson1234]

Thanks @G3rtjan , I will handle it in a very similar manner to what you outline above.

[MarkEdmondson1234]

@G3rtjan I have published a fix for this, if you have time please test it let me know if it works as expected.

[MarkEdmondson1234]

Ok it works, but if you have a lot of objects it takes ages. Going to add a limit to it

[MarkEdmondson1234]

No limit as the order it comes out is random so doesn't make much sense, but use the new prefix and delimiter instead to limit what is downloaded ( #68 )