Make istiod-less Remote Cluster the default for multicluster

[nmittler]

For 1.8 (Multicluster Beta), we want to switch the default remote cluster type to be istiod-less. We also want to deprecate the old remote cluster type.

All tests should use the istiod-less remote by default as well.

[ ] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[x] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Additional context

[stevenctl]

@irisdingbj @linsun can you point me to a doc or help me understand why we need to set ISTIOD_CUSTOM_HOST?

[linsun]

@stevenctl ISTIOD_CUSTOM_HOST is an env var that is used to generate the webhook certificate for istiod to include the host. This is useful when the host is not from a trust-able authority.

[linsun]

Update on this: #27921

[irisdingbj]

#28181

[ericvn]

@irisdingbj @nmittler Should #28181 have satisfied this issue and it can be closed?

[stevenctl]

No, we don't have a documented path for users to enable this in production. What we're doing in integration tests is a bit non-standard and actually is the cause of some of the flakes we see.

[brian-avery]

@stevenctl @nmittler Who's doing the work here? What work is needed? This is marked as a p0 for 1.10. Is that still accurate?

[stevenctl]

The work required:

• Tests
  • We have the capability to make ITs use this model, but that uses some hacks. If we can eliminate the CUSTOM_HOST thing by installing custom certs on our test containers, that may work.
  • We'll likely want to run 2 prow jobs for networking test, one that has this model and one with the old model of remotes.
• Docs for how to set this up as a user.

As far as priority level, who is going to work on this and when, I'm not sure.

[nmittler]

@linsun @GregHanson IBM had been leading this work previously. Is there any plan to help drive the feature to Beta?

[nmittler]

not stale

[ryantking]

@nmittler @linsun @irisdingbj @stevenctl Any opposition moving this into the 1.12 milestone?

[Kmoneal]

@linsun @irisdingbj @stevenctl Any updates if this will need to be moved to 1.13?

[Kmoneal]

I'll move this to 1.13 due to no recent activity

[stevenctl]

@frankbu at this point, it's mostly docs, right? do you think it will make 1.13?

[stevenctl]

Assuming this will miss 1.13? I haven't kept close attention.

[frankbu]

The istiodless remotes integration tests are working, so mostly just docs need to be updated, but the implementation is not ideal because currently there is no way for an istiod to know if a remote secret is for a remote cluster or another primary as @stevenctl rightfully pointed out here: #36121 (comment). We probably want a deterministic implementation before we promote?

[stevenctl]

Agreed.