Platform Discovery timing out on Minikube

[leosarra]

Bug Description

Hello everyone.
While running Istio 1.13 on Minikube I noticed that all proxies are subject to a flat 5-second delay when starting up.
This is due to the Platform Discovery that is performed before starting the pilot-agent.
Specifically, since Istio 1.13, the AWS check is taking more than 5 sec.
Previous releases did not show such behavior.

The issue can be easily reproduced as follows:

1. Setup a Minikube cluster
2. Install Istio via istioctl install
3. Check logs of the newly deployed ingressgateway to see if the following entry is present:
   2022-03-09T10:59:41.839438Z info timed out waiting for platform detection, treating it as Unknown

I did some troubleshooting of the issue myself and I noticed that the last release added some checks on permissions errors that, if triggered, warrant for a more complex check using AWS SDK Client. (PR #36380)
In the case of Minikube, reading product UUID will lead to a permission error, due to this, since Istio 1.13, the AWS SDK Client will be used as a fallback and that will lead to the timeout.

Would it be possible to change the checks such that Minikube would not be affected by this degradation?
Or can the check using AWS SDK be changed such that it will not end up in a timeout?

Version

client version: 1.13.1
control plane version: 1.13.1
data plane version: 1.13.1 (1 proxies)
Client Version: v1.20.1
Server Version: v1.21.7

Additional Information

No response

[howardjohn]

@ramaraochavali

[ramaraochavali]

Can you please try setting CLOUD_PLATFORM to none and see if it helps?

[leosarra]

Can you please try setting CLOUD_PLATFORM to none and see if it helps?

Hello @ramaraochavali
Yes, setting CLOUD_PLATFORM=none would help as Platform Discovery is getting skipped and platform is set to Unknown out-of-the-box. Even so, I feel that this is only a workaround.
The AWS check will delay startup by 5 sec when executed on Minikube with default Istio configuration. Do you think that, under these circumstances, this new delay has to be expected or not?
I also wonder if other environments may show similar behavior with the AWS check leading to a timing out.

[howardjohn]

It should not be required to set CLOUD_PLATFORM, I think that was just to verify the root cause. If we cannot fix this we should revert the AWS change

…

On Wed, Mar 9, 2022 at 8:35 AM Leonardo Sarra ***@***.***> wrote: Can you please try setting CLOUD_PLATFORM to none and see if it helps? Hello @ramaraochavali <https://github.com/ramaraochavali> Yes setting CLOUD_PLATFORM=none would help as Platform Discovery is getting skipped and platform is set to Unknown out-of-the-box. Even so, I feel that this is only a workaround. The AWS check will delay startup by 5 sec when executed on Minikube with default Istio configuration. Do you think that this additional delay has to be expected or not under these circumstances? I also wonder if other environments may show similar behavior with the AWS check leading to a timing out. — Reply to this email directly, view it on GitHub <#37832 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXICNOAZ4UYQYZ3Z52DU7DHN5ANCNFSM5QJHX54A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[ramaraochavali]

Yeah. I was just trying to root cause. I will try to fix it tomorrow.

[ramaraochavali]

@LithiumSR We discussed the same problem here #36380 (comment) and one of the reasons for adding "none" is this only. However it would be nice if we have a better solution. I am going through AWS docs and found a better way to do it.

I will submit a PR tomorrow.

[leosarra]

Ahh, I see the reasoning behind the changes.
Thank you @ramaraochavali for looking into it :)