CURL fails to make call to IMDS in AWS request signing filter if Envoy process uses over 1024 file descriptors

[PeterL328]

Title: CURL fails to make call to IMDS in AWS request signing filter if Envoy process uses over 1024 file/socket descriptors

Description:
CURL fails to make call to IMDSv1 in AWS request signing filter if Envoy process uses over 1024 file/socket descriptors.

Potential cause:
From my investigations, CURL failed to make the call to IMDSv1 because poll() could not be used and CURL used select() as fallback. Since select() is very limited due to it being only able to handle up to 1024 file descriptors(i.e opened files, socket connections, etc), poll() is generally preferred with a much higher limit for handing file descriptors. For reference on poll() and select() usage in CURL, see here.

The reason CURL is falling back to select() is due to CURL checking if CMAKE_TOOLCHAIN_FILE is defined. If it is defined, then CURL will not use poll() and use select(). In the Envoy repo, we build CURL with CMAKE_TOOLCHAIN_FILE defined. This causes Envoy's CURL to always use select() instead of poll().

Possible fixes:
This issue has recently been addressed in the CURL repo, PR link. It should be part of version 7.87 of CURL and Envoy should just get the fix for free.

This issue is created to raise awareness of the issue in case people run into some similar errors.

Logs:

[2022-11-04 20:26:45.905][136][debug][filter] [external/envoy/source/extensions/filters/http/aws_request_signing/aws_request_signing_filter.cc:48] aws request signing from decodeHeaders use_unsigned_payload: false
[2022-11-04 20:26:45.905][136][debug][aws] [external/envoy/source/extensions/common/aws/credentials_provider_impl.cc:43] Getting AWS credentials from the environment
[2022-11-04 20:26:45.905][136][debug][aws] [external/envoy/source/extensions/common/aws/credentials_provider_impl.cc:73] Getting AWS credentials from the instance metadata
[2022-11-04 20:26:45.905][136][debug][misc] [external/envoy/source/extensions/common/aws/utility.cc:283] Could not fetch AWS metadata: A libcurl function was given a bad argument
[2022-11-04 20:26:46.906][136][debug][misc] [external/envoy/source/extensions/common/aws/utility.cc:283] Could not fetch AWS metadata: A libcurl function was given a bad argument
[2022-11-04 20:26:47.906][136][debug][misc] [external/envoy/source/extensions/common/aws/utility.cc:283] Could not fetch AWS metadata: A libcurl function was given a bad argument
[2022-11-04 20:26:48.906][136][debug][misc] [external/envoy/source/extensions/common/aws/utility.cc:283] Could not fetch AWS metadata: A libcurl function was given a bad argument
[2022-11-04 20:26:49.906][136][error][aws] [external/envoy/source/extensions/common/aws/credentials_provider_impl.cc:82] Could not retrieve credentials listing from the instance metadata
[2022-11-04 20:26:49.906][136][debug][aws] [external/envoy/source/extensions/common/aws/credentials_provider_impl.cc:215] No AWS credentials found, using anonymous credentials

[sc0ttbeardsley]

great find @PeterL328 ! 👏

[wbpcode]

Thanks for this great find. And could you also create a RP for this fix (may be after the curl releases a new version)?

[wbpcode]

cc @derekargueta @mattklein123

[PeterL328]

Thanks for this great find. And could you also create a RP for this fix (may be after the curl releases a new version)?

Sure, no problem.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[PeterL328]

Looks like CURL ver 7.87 has been released and it contains the fix for the above. Will make a PR in Envoy to upgrade CURl to 7.87 and we also need to reopen this issue.

[PeterL328]

@wbpcode Are you able to reopen this issue?

[PeterL328]

Created the PR #24719 to bump CURL version