cannot find aws s3 file?

[hengck23]

The following code return the error:
[09:38:25] src/io/input_split_base.cc:163: Check failed: files_.size() != 0U (0 vs. 0) Cannot find any files that matches the URI patternz s3://deep-learn-data/imagenet1k-val.rec

What did I go wrong?

path_imgrec = r's3://deep-learn-data/imagenet1k-val.rec'
data = mx.io.ImageRecordIter(
        path_imgrec=path_imgrec,
        label_width=1,
        mean_r=rgb_mean[0],
        mean_g=rgb_mean[1],
        mean_b=rgb_mean[2],
        preprocess_threads=4,
        batch_size=batch_size,
        data_shape=(3,224,224),
        rand_crop=False,
        rand_mirror=False)

---

However, if i use boto3 to verify the file, it is ok:

   s3 = boto3.resource('s3')
    exists = False

    try:
        s3.Object('deep-learn-data', 'imagenet1k-val.rec').load() 
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            exists = False
        else:
            raise
    else:
        exists = True

    print(exists)

## this print true

Lastly, by using aws cli, i verify that file exists:

    export path="s3://deep-learn-data/imagenet1k-val.rec"
    count=$(aws s3 ls $path | wc -l)
    echo $count

## this print 1

[mli]

have you set the s3 credential correctly, namely set env var: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

[hengck23]

@mli yes i have set them correctly. The boto3 code will verify correct setting. Also for wrong keys, the error will be different.

[hengck23]

@mli not sure if there is a bug in mxnet source code. If i remove the hypens, it can find my s3 file. For example, the below link is ok

 path_imgrec = r's3://deepdatas/imagenet1kval'

(however, doing so will lead to another redirect error "Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests." which i think is a separate issue, see #1178)

[asadoughi]

I encountered this issue as well. The way the code works today it will only find data that are in S3 folders, see https://github.com/dmlc/dmlc-core/blob/f1f366bafbae30dcacbe5a1407370b05f2aabef3/src/io/input_split_base.cc#L109-L111.

[phunterlau]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!