ListObjects V2 causes endless loop when > 1000 items

[half-life666]

I use below code on a prefix which have > 1000 objects, it will loop forever on first 1000 objects:

	url := "myurl"
	creds := credentials.NewStaticV4("xxx", "xxxx", "")
	bucket := "my-bucket"

	s3Client, err := minio.New(url, &minio.Options{
		Creds: creds,
		Secure: true,
	})
	if err != nil {
		log.Fatalln(err)
	}

	opts := minio.ListObjectsOptions{
		UseV1:     false,
		Prefix:    "my-prefix",
		Recursive: true,
	}

	for object := range s3Client.ListObjects(context.TODO(), bucket, opts) {
		if object.Err != nil {
			fmt.Println(object.Err)
			return
		}
		fmt.Println(object)
	}

I saw there is #719 but it seems only add a validation and does not seem fix this issue.
On java sdk there is minio/minio-java#580 which seems fixed the issue for v2, but go sdk still have this problem.

listobject v1 works fine though.

[harshavardhana]

@half-life666 what is your server implementation here?

[half-life666]

oh, it's huawei obs, do you think that's the problem?

[harshavardhana]

oh, it's huawei obs, do you think that's the problem?

yes they do not implement ListObjectsV2 API or they do and implement it incorrectly - they need to send back ContinuationToken if they do not we have no way to know.

This is not our SDK bug. However we can try to error out when such broken implementations are seen in the wild.

[half-life666]

Thanks for pointing that out!
We could use ListObjects v1 for our side to fix our problem. But I think error out in this case is also somehow necessary, there are various s3 server side implementations, you can't expect them to be same as aws s3.

[harshavardhana]

But I think error out in this case is also somehow necessary, there are various s3 server side implementations, you can't expect them to be same as aws s3.

This SDK is only meant to work with AWS S3 and its compatible implementations such as MinIO. It is out of our scope for us to support obscure implementations such as Huawei OBS.

But to catch such actors we will error out when necessary conditions are not met.