getBucket() looks like there is a limit of 999 files

[capsandiego]

I tried to use getBucket() to read the content of a folder into S3 Bucket.
If there are more than 999 files, the function can't return and hangs indefinitely. I tried also with empty files. Same issue.
Is there a limitation of number of files getBucket() can list?

[capsandiego]

I can read 1000 files if I specify 1001 as 4th parameter in the getBucket method. But how can I know how many files are in a bucket upfront? If I pass a higher number of (number of files in the folder) +1 it won't work

$this->connector->getBucket($this->bucket, $this->folder.'/', null, 1001);

[nikosdion]

If you do not set $maxKeys it will cause the S3 call to run recursively until S3 reports there are no more entries with the given prefix. It does not hang indefinitely, it takes time to list thousands of files.

If you do set $maxKeys to a non–null value it will only list that many items and stop. This values has to be less than or equal to 1000 because that's the Amazon S3 API limit.

What you want (list all files without spending more time listing the files) is not possible. The Amazon S3 API will ONLY return up to 1000 files. If you want to list more files you have to do ceil(n / 1000) requests to S3. If you have 99,999 files you need to do 100 requests. Each request takes around 2–3 seconds so you'd have to wait about 5 minutes to get the full listing.

[capsandiego]

Could you please provide an example of how reading more than 1001 files could be done? How do you know in advance how many files are in a folder in order to do ceil(n / 1000) ?

[nikosdion]

Look, man. I can tell you how things work, I cannot understand it for you. I already told you that you need to leave $maxKeys as null. I also explained that in this case the library itself will AUTOMATICALLY make the required number of requests. I also explained that even though you said it “hangs indefinitely” when you do that it actually doesn't, it makes this many number of requests which take time.

This is the second issue you are filing because you seem unable to read the docblock headers and keep on asking for someone to understand it for you. Sorry, this doesn't work like that. You really need to understand it yourself. The library is very simple and open source, you can read the code and follow it with a debugger. I will have to lock this conversation since there's only so many ways I can say the same, simple thing.