InstanceProfileProvider: bug with Guzzle 7.0.1, works with 6.4.5 #2057
Assignees
Labels
bug
This issue is a bug.
Comments
zzgab
added
bug
|
Thanks for reaching out to us about this @zzgab. Unfortunately I haven't been able to reproduce the described behavior using the specified versions of PHP, the AWS SDK for PHP, and the Guzzle HTTP client. The following information should help us better troubleshoot this behavior.
The following code snippets result in successful attempts to retrieve credentials from the instance metadata service on t2.micro and t3a.large instances on my end. SDK code<?php
require_once("./vendor/autoload.php");
use Aws\S3\S3Client;
use Aws\Credentials\InstanceProfileProvider;
use Aws\Exception\AwsException;
$i = 0;
do {
$i++;
$profile = new InstanceProfileProvider();
$client = new S3Client([
'region' => 'us-west-2',
'version' => 'latest',
'credentials' => $profile,
]);
try {
$resp = $client->listBuckets();
echo "Found " . sizeof($resp['Buckets']) . " buckets on attempt {$i}." . PHP_EOL;
} catch (AwsException $e) {
echo "S3 error: {$e->getMessage()}" . PHP_EOL;
}
} while (isset($e) !== true);Guzzle code<?php
require_once('./vendor/autoload.php');
$client = new GuzzleHttp\Client();
$credLocation = "http://169.254.169.254/latest/meta-data/iam/security-credentials/" . getenv('AWS_ROLENAME');
$i = 0;
do {
$i++;
try {
$req = $client->request('GET', $credLocation);
$code = $req->getStatusCode();
echo "Attempt {$i} statusCode: {$code}" . PHP_EOL;
} catch (Exception $e) {
echo "Request failed! {$e}" . PHP_EOL;
}
} while (isset($e) !== true); |
diehlaws
added
the
response-requested
|
Hi @diehlaws, thanks for your quick response. I can confirm that I do reproduce the behaviour, using your SDK snippet (slightly modified to avoid the infinite loop in case of success). I run the application (and your snippet) not natively on the EC2 instance, but inside a Docker container (image: The snippet works fine with (I read: "Found 5 buckets") but it fails with : NB: I executed the snippet PHP file directly (direct HTTP request in public html, not through routing framework or dependency injection). My complete (but I don't think it has an effect on my current problem) Please note that the Guzzle snippet works, including However, the environment does not contain an Thanks for your help. |
github-actions
bot
removed
the
response-requested
|
I had the same problem, running in a docker container with PHP 7.4. I also tried with 7.2 and 7.3 and got the same error. Thanks for the info @zzgab, with downgrading guzzle to version 6 it works again. :) I'm using the Symfony bundle and enabled the debug mode and ran the SQS pull. This was the output I hope it can be a bit of help. GUZZLE 7.1 |
|
@Stollie maybe you could add also debug from 6.5.4 ? |
|
Here it is on PHP 7.2. From another container. |
|
Just confirming this bug as well. In our case, we were running into We were fairly certain from the beginning of troubleshooting that this was something specific to the AWS PHP SDK since we have Node.js applications running on the same cluster under the same service account using SES/SNS/SQS without issue. But can confirm that rolling Guzzle back to 6.5.4 fixed the issue. |
|
Hi all, I believe we have the same issue (reading the ticket and the condition). debugging internally, in our case, we've narrowed it down to the way the PHP/Guzzle first try to access the AWS Metadata v2 Token API http://169.254.169.254/latest/api/token which:
I could reproduce the issue within the same docker container using curl: Strangely, the same curl command was successful on the host itself, so there is something fishy on the way AWS might deal with the docker network and the token API call. Also, access to any other EC2 metadata endpoint (like the direct Ec2 credentials endpoint) is still working from within the container normally with or without the token header (because we don't enforce metadata V2 just yet). I believe this might explain why the older version of guzzle is working as it might not yet have implemented the code to use metadata v2 access. I'll have to dig further on my side as to why AWS metadata v2 token API access is timeout within a docker container (this does not quite make sense yet from a TCP/HTTP point of view). Hopefully this is helpful. |
|
This is a generic docker bridge network issue accessing that token API: Per https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ (specifically the Protecting against open layer 3 firewalls and NATs) the return packet has a TTL of 1, and as a bridge container is 2 hops away, the return packet of the response of the HTTP GET call never gets back (the TCP SYN-ACK will have usual TTL which is why TCP established normally). In our case, I can work around the problem with a simple iptables rule: Disclaimer, this is a security change which would defeat the protection AWS implemented. Please use at your own risk. I'm not sure if that's the ideal approach, but that's one way of fixing it. |
|
@fischaz even if that's the issue.. how does this explain that it works with Guzzle6? Do you have any idea? |
|
That I cannot explain. if guzzle only takes care of the HTTP aspect of things, it should fail the same way when the Ec2Metadata class tries to query the endpoint with guzzle (v6 or v7)... Sorry... |
a better way might be to use
to set the instance metadata options, because this can be done for launch configurations and templates too. |
This is the recommended way of dealing with this use case. For more context, EC2 has a guide on configuring IMDSv2. |
|
I was not aware of this, this is indeed a better workaround for this issue. |
|
regarding this issue, could it be because Guzzle refactored in V7 their exceptions? eg in 7.0.0-beta2, there is the entry:
and looking (admittedly at master in github, not the specific tag) at the guzzle code (https://github.com/guzzle/guzzle/blob/master/src/Handler/CurlFactory.php#L205) my view is that guzzle is not throwing a RequestException (bad headers), but a now the InstanceCredentials (https://github.com/aws/aws-sdk-php/blob/master/src/Credentials/InstanceProfileProvider.php#L215) specifically look for a RequestException (I guess that's what it used to do in v6) to catch/re-throw for the _invoke (https://github.com/aws/aws-sdk-php/blob/master/src/Credentials/InstanceProfileProvider.php#L84) to catch and ignore.... but as v7 throws a different exception, this goes to the next case which is not caught and the exception is final. would that make sense? |
|
this problem occurred with me using aws sdk php version |
|
@howardlopez Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug
S3Clientused in the context ofInstanceProfilePovider.The description is the same as #2022 Aws\Exception\CredentialsException Timeout, but it fails only with
guzzlehttp/guzzlev7.0.1.The same SDK version on same instance (and same application) with
guzzlehttp/guzzlev 6.5.4 works fine. Therefore I don't think that the issue is the EC2 instance being unable to respond quick enough.Version of AWS SDK for PHP?
v3.138.9 (but I believe it's the same with 3.145)
Version of PHP (
php -v)?php 7.4
To Reproduce (observed behavior)
When running this code on an EC2 instance with an IAM role granting the appropriate S3 permissions:
with the Guzzlehttp version reported above (7.0.1): I get an exception:
** Expected behaviour**
Exact same code, same machine, works fine if I downgrade
guzzlehttp/guzzleto version 6.5.4.The text was updated successfully, but these errors were encountered: