fix: oss list bucket return all records

[shuangkun]

when list a bucket over 100 object, the response can only return 100 records, so wrote a loop to return all records.

Fixes #TODO

Motivation

Modifications

Verification

[terrytangyuan]

Thank you. Can you add some notes in the code? Is there any reference to the 100 records limit?

[shuangkun]

Thank you. Can you add some notes in the code? Is there any reference to the 100 records limit?

https://help.aliyun.com/zh/oss/user-guide/list-objects-18?spm=a2c4g.11186623.0.0.275f7711UTgi3H
there are some offical examples.

    continueToken := ""
    for {
         lsRes, err := bucket.ListObjectsV2(oss.ContinuationToken(continueToken))
         if err != nil {
             HandleError(err)
          }
          // 打印列举结果。默认情况下，一次返回100条记录。
          for _, object := range lsRes.Objects {
              fmt.Println(object.Key, object.Type, object.Size, object.ETag, object.LastModified, object.StorageClass)
          }
          if lsRes.IsTruncated {
              continueToken = lsRes.NextContinuationToken
          } else {
              break
          }
      }

[terrytangyuan]

Should we also bump max-keys to 1000?

[shuangkun]

Should we also bump max-keys to 1000?

Maybe we need a configuration to limit it, because some customers' oss objects are indeed relatively large, exceeding 1000.

[shuangkun]

/CI

[terrytangyuan]

oss.Prefix is removed accidentally?

[shuangkun]

i forget it. thanks.

[terrytangyuan]

We should check whether it's truncated before accessing next continuation token

[shuangkun]

Yes.

[terrytangyuan]

Add a link to https://help.aliyun.com/zh/oss/user-guide/list-objects-18

[shuangkun]

ok.

[terrytangyuan]

Can you address my comments and then run some tests if you have OSS available?

[terrytangyuan]

Can you test it against actual OSS environment and see if everything is working correctly?

[shuangkun]

Can you test it against actual OSS environment and see if everything is working correctly?

OK. I'll run some tests later.

[shuangkun]

Can you test it against actual OSS environment and see if everything is working correctly?

I made a test use the file. It works well.
After fixing this problem, more than 100 steps can be triggered when the directory to be listed has more than 100 files.（There were only 100 before）

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-transformations-
  labels:
    workflows.argoproj.io/test: "true"
  annotations:
    workflows.argoproj.io/description: |
      This workflow demonstrates using a data template to list in an S3 bucket
      and then process those log files.
    workflows.argoproj.io/version: ">= 3.1.0"
spec:
  entrypoint: data-transformations
  templates:
    - name: data-transformations
      steps:
        - - name: list-log-files
            template: list-log-files
        - - name: process-logs
            template: process-logs
            withParam: "{{steps.list-log-files.outputs.result}}"
            arguments:
              artifacts:
                - name: file
                  oss:
                    endpoint: http://oss-cn-hangzhou.aliyuncs.com
                    bucket: bucket-workflow-artifect
                    accessKeySecret:
                      name: bucket-workflow-artifect-credentials
                      key: accessKey
                    secretKeySecret:
                      name: bucket-workflow-artifect-credentials
                      key: secretKey
                    key: "{{item}}"

              parameters:
                - name: file-name
                  value: "{{item}}"

    - name: list-log-files
      data:
        source:
          artifactPaths:
            name: test-bucket
            oss:
              endpoint: http://oss-cn-hangzhou.aliyuncs.com
              bucket: bucket-workflow-artifect
              key: artifact-passing-wsl88 # this is path in the bucket
              accessKeySecret:
                name: bucket-workflow-artifect-credentials
                key: accessKey
              secretKeySecret:
                name: bucket-workflow-artifect-credentials
                key: secretKey

        transformation:
          - expression: "filter(data, {# startsWith \"arti\"})"
      outputs:
        artifacts:
          - name: file
            path: /tmp

    - name: process-logs
      inputs:
        parameters:
          - name: file-name
        artifacts:
          - name: file
            path: /tmp
      container:
        image:  argoproj/argosay:v2
        command: [ sh, -c ]
        args:
          - |
            echo {{inputs.parameters.file-name}}
            head /tmp

[terrytangyuan]

Thanks!