Investigate using istio-ingressgateway by default

[tcnghia]

Expected Behavior

Knative Serving installation does not need to install something in istio-system namespace. Knative Serving does not have to keep up with changes in Envoy proxy images.

Actual Behavior

We are installing an ELB knative-ingressgateway into istio-system namespace. Also, we are using a copy of Istio ELB istio-ingressgateway which requires making sure it is up-to-date with Istio's.

This was so that our Gateway can use those pods without conflicting with users' own Gateways.

However, Istio 1.0.1 may in better merging of Gateways, so may be we can base on Gateway Hosts to avoid conflicting (this didn't work with 0.8). Also Istio now have validation so users Gateway conflict may be less confusing than before (silent unavailability of service)

We should investigate allowing the users to choose what ingressgateway ELB they can use, which defaults to the Istio default istio-ingressgateway. This way we not only avoid maintaining this service ourselves, we also avoid installing things into istio-system namespace and save the user 1 static IP.

[tcnghia]

However for backward compatibility we probably want to keep providing knative-ingressgateway Service until the next release and update docs to transition away.

[SergeC]

@tcnghia Since Knative is in alpha or beta stage it's common case if backward compatibility got broken. More important to keep fast development to get it to the production stage.

[lichuqiang]

/cc

[tcnghia]

/assign @lichuqiang

[lichuqiang]

Took a closer look at the issue, here are general thoughts:
keep the 'Gateway' resource as is, what changed is in fact underlying gateway service/deployment, so on gateway update, we'll need to:

1. feel the change in ingress reconciler, and update Loadbalancer in ingress status accordingly.
2. update the label selector and ports fields accordingly in knative-shared-gateway

and here are the tasks I think needed to achieve this:

1. add a config item for the gateway service url in a new configmap, and default its value to istio-ingressgateway.istio-system.svc.cluster.local, ClusterIngress reconciler is supposed to watch the configmap, and trigger global resync on a change.

2. update existing knative-ingressgateway usage in test codes to the default istio-ingressgateway. I don't see the requirement to change the config in tests, so hard code it should be enough. @tcnghia thoughts?

3. update the docs to tell users how to replace the default gateway with their own. Currently we are not managing configs with tools like helm, so I'm afraid users will have to update fields in Gateway template manually. Suggestions?

[tcnghia]

@lichuqiang #1,#2 sound good to me.
For #3, we could include current knative-ingressgateway template as an example, and may be link to the appropriate Istio discussion so our users know where to get help. In the end I think this area belongs to the Istio side and users would get better support there.

[tcnghia]

Per discussion with @jonjohnsonjr and @mattmoor we will extend feature this to allow exposing to multiple ingress gateway to allow users to easily migrate without disruption.

In 0.3 we will expose to both knative-ingressgateway and istio-ingressgateway. In 0.4 we will only expose to istio-ingressgateway.