aws: imds_v2 credential has some problem

[yufan022]

Databend failed to start after removing access_key_id and secret_access_key .

[storage.s3]
bucket = "xx"
endpoint_url = "https://s3.ap-northeast-1.amazonaws.com"
#access_key_id = "xx"
#secret_access_key = "xx"

{"timestamp":"2022-12-12T16:44:44.716618Z","level":"WARN","fields":{"message":"load_via_profile_shared_credentials_file failed: No such file or directory (os error 2)","log.target":"reqsign::aws::config","log.module_path":"reqsign::aws::config","log.file":"/opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/reqsign-0.6.9/src/aws/config.rs","log.line":174},"target":"reqsign::aws::config"}
{"timestamp":"2022-12-12T16:44:44.716640Z","level":"WARN","fields":{"message":"load_via_profile_config_file failed: No such file or directory (os error 2)","log.target":"reqsign::aws::config","log.module_path":"reqsign::aws::config","log.file":"/opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/reqsign-0.6.9/src/aws/config.rs","log.line":176},"target":"reqsign::aws::config"}
{"timestamp":"2022-12-12T16:44:44.717063Z","level":"WARN","fields":{"message":"load credential via imds v2 failed: http://169.254.169.254/latest/api/token: status code 400","log.target":"reqsign::aws::credential","log.module_path":"reqsign::aws::credential","log.file":"/opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/reqsign-0.6.9/src/aws/credential.rs","log.line":141},"target":"reqsign::aws::credential"}
{"timestamp":"2022-12-12T16:44:44.717086Z","level":"INFO","fields":{"message":"load credential failed but we allowing anonymous access","log.target":"reqsign::aws::credential","log.module_path":"reqsign::aws::credential","log.file":"/opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/reqsign-0.6.9/src/aws/credential.rs","log.line":171},"target":"reqsign::aws::credential"}
{"timestamp":"2022-12-12T16:44:44.748011Z","level":"WARN","fields":{"message":"service=s3 operation=list path=/ -> errored: ObjectPermissionDenied (permanent) at ObjectPage::next_page => S3Error { code: \"AccessDenied\", message: \"Access Denied\", resource: \"\", request_id: \"xx\" }\n\nContext:\n    response: Parts { status: 403, version: HTTP/1.1, headers: {\"x-amz-bucket-region\": \"ap-northeast-1\", \"x-amz-request-id\": \"xx\", \"x-amz-id-2\": \"xx\", \"content-type\": \"application/xml\", \"transfer-encoding\": \"chunked\", \"date\": \"Mon, 12 Dec 2022 16:44:44 GMT\", \"server\": \"AmazonS3\"} }\n    service: s3\n    path: /\n","log.target":"opendal::services","log.module_path":"opendal::layers::logging","log.file":"/opt/rust/cargo/registry/src/github.com-1ecc6299db9ec823/opendal-0.22.1/src/layers/logging.rs","log.line":1221},"target":"opendal::services"}

I found two problems:

x-aws-ec2-metadata-token-ttl-seconds request header for http://169.254.169.254/latest/api/token is required.

curl -X PUT -v "http://169.254.169.254/latest/api/token"
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Length: 0
< Date: Mon, 12 Dec 2022 16:23:52 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection 0

curl -X PUT -v "http://169.254.169.254/latest/api/token" -H "x-aws-ec2-metadata-token-ttl-seconds: 60"
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> x-aws-ec2-metadata-token-ttl-seconds: 60
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 60
< Content-Length: 56
< Date: Mon, 12 Dec 2022 16:25:20 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
<
* Closing connection 0
xxxxxxxxx

The response format for http://169.254.169.254/latest/meta-data/iam/security-credentials is json.

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60"`
ROLE=`curl -H "x-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/`
curl -H "x-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/xxxxxxxx HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> x-aws-ec2-metadata-token: xxxxxxxx
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 60
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Mon, 12 Dec 2022 15:42:45 GMT
< Content-Length: 1462
< Date: Mon, 12 Dec 2022 16:33:07 GMT
< Server: EC2ws
< Connection: close
<
{
  "Code" : "Success",
  "LastUpdated" : "2022-12-12T15:42:33Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "xxxxxxxx",
  "SecretAccessKey" : "xxxxxxxx",
  "Token" : "xxxxxxxx",
  "Expiration" : "2022-12-12T22:12:07Z"
* Closing connection 0```

[yufan022]

cc @Xuanwo