-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Network configuration when using private clusters #4868
Comments
This is quite strange, since |
I am experiencing the exact same issue. I have installed knative (build/serving/eventing) on 1.11x, 1.12x, and 1.13x private GKE clusters. These clusters have the latest istio installed and have the master authorized networks disabled (have tried this with these networks enabled as well) and am unable to creates builds or ksvcs under any scenario. Have also tried installed knative v0.6x and v0.7x under all the above GKE settings and no luck either |
Can you share information about how to create a cluster like the one where you are seeing this? |
@mattmoor Below are the configurations that I'm using to create my gke cluster and to bootstrap it with knative.
After running |
cc @tcnghia |
Thanks for the detailed repro instructions. Early this week will be a bit chaotic shutting down 0.8, but this should be very helpful attempting to reproduce what you are seeing so that we can get your problem sorted out. |
@mattmoor Hate to pester you, but I'm curious if there has been any update on the knative + private GKE issue. |
I'll try to find someone to look into it. I pinged @tcnghia , but realized he is out today. Sorry for the delay. |
No worries at all. Thanks for putting this on the radar.
…On Wed, Aug 21, 2019, 11:15 AM Matt Moore ***@***.***> wrote:
I'll try to find someone to look into it. I pinged @tcnghia
<https://github.com/tcnghia> , but realized he is out today. Sorry for
the delay.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#4868?email_source=notifications&email_token=AFYTD6MKNTW3F3KOUGIJKRLQFVL2XA5CNFSM4IFWEL62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD42ANHI#issuecomment-523503261>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFYTD6P2OP35PKAM2WM425LQFVL2XANCNFSM4IFWEL6Q>
.
|
I think this is a firewall issue, similar that of elastic/cloud-on-k8s#1437 Can you please try the workaround there? thanks |
8443 is the port that you need to allow https://github.com/knative/serving/blob/master/config/400-webhook-service.yaml#L26 |
The short explanation is that GKE private cluster by default only allows the GKE master to access your Services at port 443 or 80. Our webhook uses 8443 here, so it needs to be white-listed. Instruction for that is here https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules There may be other webhooks like Istio's that may need a white list. |
My identical problem was resolved by @tcnghia's suggestion to add ingress 8443 to the firewall |
BTW, the reason why 443 wasn't used is to avoid a privileged port (knative/build#604). I just look at Istio's webhooks and it look like they use 443, so no need to have additional rule for Istio. 8443 should be enough. |
@sjmiller609 awesome! thanks a lot for confirmation. |
@bbhuston if you could confirm this works, then we should discuss if/what changes we need to close this out. |
@mattmoor Sorry for the delayed response. Was on an awesome vacation and was a little too lazy to check up on this. Anyway, I reran the terraform/gke/knative setup that I posted above and then manually opened up port 8443 for the clusters master and worker node firewall rules. And BOOM! It works. Thank you for the follow-up and please feel free to close this issue. |
Thanks for confirming. I think we'll need to update the doc with this information, since avoiding 443 is still a good path (avoiding privileged port) /close |
@tcnghia: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Now on the official GCP documentation! |
I got this issue on microk8s on Windows:
Any suggestion on what I should do to start diagnosing them cause and finding alternatives? |
In what area(s)?
/area networking
Ask your question here:
We encountered an issue when using
Knative
in a private cluster environment. Consider the following architecture:We have a cluster for our engineers running in GKE as a private cluster (master and nodes are inaccessible via the Internet). Unfortunately, when applying a
Knative
service it fails with:Everything works as expected when installing the service on a public cluster. Any help on this is highly appreciated 🙂
The text was updated successfully, but these errors were encountered: