Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-aggregator fails to proxy for HTTP/2 enabled EAS #81310

Open
tamalsaha opened this issue Aug 12, 2019 · 7 comments

Comments

@tamalsaha
Copy link
Member

commented Aug 12, 2019

I am running an EAS server here: https://api.crd.builders . This server is using trusted cert and has not authN and authZ (public). This is hosted on Google appengine and serves via HTTP/2 & HTTP/1.1 .

$ curl -vv https://api.crd.builders/apis/meta.appscode.com/v1alpha1
*   Trying 2001:4860:4802:36::15...
* TCP_NODELAY set
* Connected to api.crd.builders (2001:4860:4802:36::15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=api.crd.builders
*  start date: Aug 11 12:46:46 2019 GMT
*  expire date: Nov  9 12:46:46 2019 GMT
*  subjectAltName: host "api.crd.builders" matched cert's "api.crd.builders"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5606de8154b0)
> GET /apis/meta.appscode.com/v1alpha1 HTTP/2
> Host: api.crd.builders
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< content-type: application/json
< vary: Accept-Encoding
< x-cloud-trace-context: 4ddacc2896ce4b81c84ed9608ab9f632;o=1
< date: Mon, 12 Aug 2019 20:13:51 GMT
< server: Google Frontend
< content-length: 497
< 
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "meta.appscode.com/v1alpha1",
  "resources": [
    {
      "name": "resourcedescriptors",
      "singularName": "",
      "namespaced": false,
      "group": "meta.appscode.com",
      "version": "v1alpha1",
      "kind": "ResourceDescriptor",
      "verbs": [
        "create",
        "delete",
        "deletecollection",
        "get",
        "list",
        "patch",
        "update",
        "watch"
      ]
    }
  ]
* Connection #0 to host api.crd.builders left intact
}

Now, I am trying to expose this as an EAS inside a minikube 1.15.0 cluster using the following YAMLs.

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: default
spec:
  type: ExternalName
  externalName: api.crd.builders
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1alpha1.meta.appscode.com
spec:
  group: meta.appscode.com
  groupPriorityMinimum: 1000
  versionPriority: 15
  service:
    name: api
    namespace: default
  version: v1alpha1

But this is not working by default. I added some logging commands to kube-apiserver and this is what I see:

pharmer@182ed33

I0812 20:07:29.858299       1 handler.go:153] kube-aggregator: GET "/apis/meta.appscode.com/v1alpha1" satisfied by nonGoRestful
I0812 20:07:29.858373       1 pathrecorder.go:240] kube-aggregator: "/apis/meta.appscode.com/v1alpha1" satisfied by exact match
I0812 20:07:29.858447       1 upgradeaware.go:249] Request was not an upgrade
I0812 20:07:29.858823       1 round_trippers.go:419] curl -k -v -XGET  -H "X-Forwarded-Uri: /apis/meta.appscode.com/v1alpha1" -H "X-Forwarded-Host: api.crd.builders:443" -H "X-Forwarded-Proto: https" -H "Accept: application/vnd.kubernetes.protobuf, */*" -H "User-Agent: kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/controller-discovery" -H "Accept-Encoding: gzip" -H "X-Forwarded-For: 127.0.0.1" -H "X-Remote-User: system:kube-controller-manager" -H "X-Remote-Group: system:authenticated" 'https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s'
___ L 231: https://api.crd.builders:443 |__ https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s
req:
 GET /apis/meta.appscode.com/v1alpha1?timeout=32s HTTP/2.0
Host: api.crd.builders:443
Accept: application/vnd.kubernetes.protobuf, */*
Accept-Encoding: gzip
User-Agent: kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/controller-discovery


newReq:
 GET /apis/meta.appscode.com/v1alpha1?timeout=32s HTTP/2.0
Host: api.crd.builders:443
Accept: application/vnd.kubernetes.protobuf, */*
Accept-Encoding: gzip
User-Agent: kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/controller-discovery


I0812 20:07:29.911716       1 round_trippers.go:438] GET https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s  in 52 milliseconds
I0812 20:07:29.911892       1 round_trippers.go:444] Response Headers:
I0812 20:07:29.912306       1 wrap.go:47] GET /apis/meta.appscode.com/v1alpha1?timeout=32s: (54.421772ms) 503
goroutine 38331 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc0015038f0, 0x1f7)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:204 +0xc8
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc0015038f0, 0x1f7)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:183 +0x35
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc006b5f480, 0x1f7)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:209 +0xa6
net/http/httputil.(*ReverseProxy).ServeHTTP(0xc0032b29b0, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc00bfd6fd8)
	/usr/local/go/src/net/http/httputil/reverseproxy.go:297 +0x79b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy.(*UpgradeAwareHandler).ServeHTTP(0xc008d94480, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc500)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/proxy/upgradeaware.go:243 +0x830
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc001a0c930, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:171 +0x750
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc00693dcc0, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:241 +0x548
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc003a437a0, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x85
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x42f075c, 0xf, 0xc0027d1e60, 0xc003a437a0, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6c3
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x4fa
net/http.HandlerFunc.ServeHTTP(0xc003461e40, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/usr/local/go/src/net/http/server.go:1995 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x5c7
net/http.HandlerFunc.ServeHTTP(0xc0053b5ef0, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/usr/local/go/src/net/http/server.go:1995 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x1ec3
net/http.HandlerFunc.ServeHTTP(0xc003461e80, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc300)
	/usr/local/go/src/net/http/server.go:1995 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc200)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:81 +0x527
net/http.HandlerFunc.ServeHTTP(0xc0014f7180, 0x7f3d3ba5bc48, 0xc00e6ad480, 0xc0071cc200)
	/usr/local/go/src/net/http/server.go:1995 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc006640300, 0xc00306e880, 0x73b1480, 0xc00e6ad480, 0xc0071cc200)
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:111 +0xb3
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/home/tamal/go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:98 +0x1b1

logging error output: "Error: 'EOF'\nTrying to reach: 'https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s'"
 [kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/controller-discovery 127.0.0.1:56826]

Note 2 things here:

kube-aggregator is using HTTP/2 transport and logging error output: "Error: 'EOF'\nTrying to reach: 'https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s'".

Then I modified the kube-aggregator to use the http.DefaultTransport and everything was working.

pharmer@cf0f519

I0812 19:23:01.184659       1 handler.go:153] kube-aggregator: GET "/apis/meta.appscode.com/v1alpha1" satisfied by nonGoRestful
I0812 19:23:01.185354       1 pathrecorder.go:240] kube-aggregator: "/apis/meta.appscode.com/v1alpha1" satisfied by exact match
I0812 19:23:01.185637       1 upgradeaware.go:249] Request was not an upgrade
___ L 231: https://api.crd.builders:443 |__ https://api.crd.builders:443/apis/meta.appscode.com/v1alpha1?timeout=32s
req:
 GET /apis/meta.appscode.com/v1alpha1?timeout=32s HTTP/2.0
Host: api.crd.builders:443
Accept: application/vnd.kubernetes.protobuf, */*
Accept-Encoding: gzip
User-Agent: kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/system:serviceaccount:kube-system:namespace-controller


newReq:
 GET /apis/meta.appscode.com/v1alpha1?timeout=32s HTTP/2.0
Host: api.crd.builders:443
Accept: application/vnd.kubernetes.protobuf, */*
Accept-Encoding: gzip
User-Agent: kube-controller-manager/v1.15.2 (linux/amd64) kubernetes/f627830/system:serviceaccount:kube-system:namespace-controller

So, it seems that kube-aggregator proxy does not work correctly with HTTP/2 enabled EAS. My questions are:

  1. Is there a way to force kube-aggregator to use HTTP/1?
  2. How can this be fixed?
@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

/sig api-machinery

@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

@lavalamp

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

Why do you think it's HTTP/2 related?

@fedebongio

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2019

@liggitt

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

this doesn't seem http/2 related, but related to the aggregator configuration of your apiserver.

do you have your full apiserver invocation, including all flags?

I'm looking for:

  • whether you are using --ssh-user
  • whether you are using --enable-aggregator-routing
@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

Thanks for the quick response. I am using the default setting of minikube. I just added --v=10 to raise log level.

whether you are using --ssh-user

No.

whether you are using --enable-aggregator-routing

No.

Here the steps, if you want to reproduce it locally using minikube:

$ minikube version
minikube version: v1.3.0
commit: 43969594266d77b555a207b0f3e9b3fa1dc92b1f

$ minikube delete; minikube start
🔥  Deleting "minikube" in virtualbox ...
💔  The "minikube" cluster has been deleted.
😄  minikube v1.3.0 on Ubuntu 18.04
🔥  Creating virtualbox VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.15.2 on Docker 18.09.8 ...
🚜  Pulling images ...
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"

$ kubectl version --short
Client Version: v1.15.0
Server Version: v1.15.2

$ kubectl create -f https://github.com/kmodules/resource-metadata/raw/master/artifacts/appengine.yaml
apiservice.apiregistration.k8s.io/v1alpha1.meta.appscode.com created
service/api created

Here are kube-apiserver flags:

Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0812 22:06:47.258392       1 flags.go:33] FLAG: --address="127.0.0.1"
I0812 22:06:47.258443       1 flags.go:33] FLAG: --admission-control="[]"
I0812 22:06:47.258457       1 flags.go:33] FLAG: --admission-control-config-file=""
I0812 22:06:47.258466       1 flags.go:33] FLAG: --advertise-address="192.168.99.100"
I0812 22:06:47.258473       1 flags.go:33] FLAG: --allow-privileged="true"
I0812 22:06:47.258481       1 flags.go:33] FLAG: --alsologtostderr="false"
I0812 22:06:47.258494       1 flags.go:33] FLAG: --anonymous-auth="true"
I0812 22:06:47.258501       1 flags.go:33] FLAG: --api-audiences="[]"
I0812 22:06:47.258525       1 flags.go:33] FLAG: --apiserver-count="1"
I0812 22:06:47.258535       1 flags.go:33] FLAG: --audit-dynamic-configuration="false"
I0812 22:06:47.258541       1 flags.go:33] FLAG: --audit-log-batch-buffer-size="10000"
I0812 22:06:47.258548       1 flags.go:33] FLAG: --audit-log-batch-max-size="1"
I0812 22:06:47.258555       1 flags.go:33] FLAG: --audit-log-batch-max-wait="0s"
I0812 22:06:47.258562       1 flags.go:33] FLAG: --audit-log-batch-throttle-burst="0"
I0812 22:06:47.258569       1 flags.go:33] FLAG: --audit-log-batch-throttle-enable="false"
I0812 22:06:47.258576       1 flags.go:33] FLAG: --audit-log-batch-throttle-qps="0"
I0812 22:06:47.258584       1 flags.go:33] FLAG: --audit-log-format="json"
I0812 22:06:47.258591       1 flags.go:33] FLAG: --audit-log-maxage="0"
I0812 22:06:47.258597       1 flags.go:33] FLAG: --audit-log-maxbackup="0"
I0812 22:06:47.258604       1 flags.go:33] FLAG: --audit-log-maxsize="0"
I0812 22:06:47.258610       1 flags.go:33] FLAG: --audit-log-mode="blocking"
I0812 22:06:47.258616       1 flags.go:33] FLAG: --audit-log-path=""
I0812 22:06:47.258623       1 flags.go:33] FLAG: --audit-log-truncate-enabled="false"
I0812 22:06:47.258629       1 flags.go:33] FLAG: --audit-log-truncate-max-batch-size="10485760"
I0812 22:06:47.258638       1 flags.go:33] FLAG: --audit-log-truncate-max-event-size="102400"
I0812 22:06:47.258645       1 flags.go:33] FLAG: --audit-log-version="audit.k8s.io/v1"
I0812 22:06:47.258652       1 flags.go:33] FLAG: --audit-policy-file=""
I0812 22:06:47.258658       1 flags.go:33] FLAG: --audit-webhook-batch-buffer-size="10000"
I0812 22:06:47.258665       1 flags.go:33] FLAG: --audit-webhook-batch-initial-backoff="10s"
I0812 22:06:47.258675       1 flags.go:33] FLAG: --audit-webhook-batch-max-size="400"
I0812 22:06:47.258694       1 flags.go:33] FLAG: --audit-webhook-batch-max-wait="30s"
I0812 22:06:47.258701       1 flags.go:33] FLAG: --audit-webhook-batch-throttle-burst="15"
I0812 22:06:47.258708       1 flags.go:33] FLAG: --audit-webhook-batch-throttle-enable="true"
I0812 22:06:47.258714       1 flags.go:33] FLAG: --audit-webhook-batch-throttle-qps="10"
I0812 22:06:47.258722       1 flags.go:33] FLAG: --audit-webhook-config-file=""
I0812 22:06:47.258728       1 flags.go:33] FLAG: --audit-webhook-initial-backoff="10s"
I0812 22:06:47.258735       1 flags.go:33] FLAG: --audit-webhook-mode="batch"
I0812 22:06:47.258741       1 flags.go:33] FLAG: --audit-webhook-truncate-enabled="false"
I0812 22:06:47.258748       1 flags.go:33] FLAG: --audit-webhook-truncate-max-batch-size="10485760"
I0812 22:06:47.258755       1 flags.go:33] FLAG: --audit-webhook-truncate-max-event-size="102400"
I0812 22:06:47.258762       1 flags.go:33] FLAG: --audit-webhook-version="audit.k8s.io/v1"
I0812 22:06:47.258769       1 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
I0812 22:06:47.258776       1 flags.go:33] FLAG: --authentication-token-webhook-config-file=""
I0812 22:06:47.258782       1 flags.go:33] FLAG: --authorization-mode="[Node,RBAC]"
I0812 22:06:47.258793       1 flags.go:33] FLAG: --authorization-policy-file=""
I0812 22:06:47.258800       1 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
I0812 22:06:47.258807       1 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
I0812 22:06:47.258813       1 flags.go:33] FLAG: --authorization-webhook-config-file=""
I0812 22:06:47.258820       1 flags.go:33] FLAG: --basic-auth-file=""
I0812 22:06:47.258826       1 flags.go:33] FLAG: --bind-address="0.0.0.0"
I0812 22:06:47.258833       1 flags.go:33] FLAG: --cert-dir="/var/run/kubernetes"
I0812 22:06:47.258840       1 flags.go:33] FLAG: --client-ca-file="/var/lib/minikube/certs/ca.crt"
I0812 22:06:47.258850       1 flags.go:33] FLAG: --cloud-config=""
I0812 22:06:47.258857       1 flags.go:33] FLAG: --cloud-provider=""
I0812 22:06:47.258863       1 flags.go:33] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0812 22:06:47.258873       1 flags.go:33] FLAG: --contention-profiling="false"
I0812 22:06:47.258879       1 flags.go:33] FLAG: --cors-allowed-origins="[]"
I0812 22:06:47.258902       1 flags.go:33] FLAG: --default-not-ready-toleration-seconds="300"
I0812 22:06:47.258910       1 flags.go:33] FLAG: --default-unreachable-toleration-seconds="300"
I0812 22:06:47.258917       1 flags.go:33] FLAG: --default-watch-cache-size="100"
I0812 22:06:47.258923       1 flags.go:33] FLAG: --delete-collection-workers="1"
I0812 22:06:47.258930       1 flags.go:33] FLAG: --deserialization-cache-size="0"
I0812 22:06:47.258936       1 flags.go:33] FLAG: --disable-admission-plugins="[]"
I0812 22:06:47.258947       1 flags.go:33] FLAG: --enable-admission-plugins="[NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota]"
I0812 22:06:47.258970       1 flags.go:33] FLAG: --enable-aggregator-routing="false"
I0812 22:06:47.258977       1 flags.go:33] FLAG: --enable-bootstrap-token-auth="true"
I0812 22:06:47.258998       1 flags.go:33] FLAG: --enable-garbage-collector="true"
I0812 22:06:47.259006       1 flags.go:33] FLAG: --enable-inflight-quota-handler="false"
I0812 22:06:47.259013       1 flags.go:33] FLAG: --enable-logs-handler="true"
I0812 22:06:47.259019       1 flags.go:33] FLAG: --enable-swagger-ui="false"
I0812 22:06:47.259026       1 flags.go:33] FLAG: --encryption-provider-config=""
I0812 22:06:47.259032       1 flags.go:33] FLAG: --endpoint-reconciler-type="lease"
I0812 22:06:47.259038       1 flags.go:33] FLAG: --etcd-cafile="/var/lib/minikube/certs/etcd/ca.crt"
I0812 22:06:47.259045       1 flags.go:33] FLAG: --etcd-certfile="/var/lib/minikube/certs/apiserver-etcd-client.crt"
I0812 22:06:47.259052       1 flags.go:33] FLAG: --etcd-compaction-interval="5m0s"
I0812 22:06:47.259061       1 flags.go:33] FLAG: --etcd-count-metric-poll-period="1m0s"
I0812 22:06:47.259068       1 flags.go:33] FLAG: --etcd-keyfile="/var/lib/minikube/certs/apiserver-etcd-client.key"
I0812 22:06:47.259076       1 flags.go:33] FLAG: --etcd-prefix="/registry"
I0812 22:06:47.259082       1 flags.go:33] FLAG: --etcd-servers="[https://127.0.0.1:2379]"
I0812 22:06:47.259097       1 flags.go:33] FLAG: --etcd-servers-overrides="[]"
I0812 22:06:47.259110       1 flags.go:33] FLAG: --event-ttl="1h0m0s"
I0812 22:06:47.259117       1 flags.go:33] FLAG: --experimental-encryption-provider-config=""
I0812 22:06:47.259123       1 flags.go:33] FLAG: --external-hostname=""
I0812 22:06:47.259129       1 flags.go:33] FLAG: --feature-gates=""
I0812 22:06:47.259138       1 flags.go:33] FLAG: --help="false"
I0812 22:06:47.259182       1 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
I0812 22:06:47.259202       1 flags.go:33] FLAG: --insecure-bind-address="127.0.0.1"
I0812 22:06:47.259210       1 flags.go:33] FLAG: --insecure-port="0"
I0812 22:06:47.259217       1 flags.go:33] FLAG: --kubelet-certificate-authority=""
I0812 22:06:47.259223       1 flags.go:33] FLAG: --kubelet-client-certificate="/var/lib/minikube/certs/apiserver-kubelet-client.crt"
I0812 22:06:47.259231       1 flags.go:33] FLAG: --kubelet-client-key="/var/lib/minikube/certs/apiserver-kubelet-client.key"
I0812 22:06:47.259238       1 flags.go:33] FLAG: --kubelet-https="true"
I0812 22:06:47.259244       1 flags.go:33] FLAG: --kubelet-port="10250"
I0812 22:06:47.259253       1 flags.go:33] FLAG: --kubelet-preferred-address-types="[InternalIP,ExternalIP,Hostname]"
I0812 22:06:47.259265       1 flags.go:33] FLAG: --kubelet-read-only-port="10255"
I0812 22:06:47.259273       1 flags.go:33] FLAG: --kubelet-timeout="5s"
I0812 22:06:47.259279       1 flags.go:33] FLAG: --kubernetes-service-node-port="0"
I0812 22:06:47.259286       1 flags.go:33] FLAG: --log-backtrace-at=":0"
I0812 22:06:47.259298       1 flags.go:33] FLAG: --log-dir=""
I0812 22:06:47.259306       1 flags.go:33] FLAG: --log-file=""
I0812 22:06:47.259312       1 flags.go:33] FLAG: --log-file-max-size="1800"
I0812 22:06:47.259319       1 flags.go:33] FLAG: --log-flush-frequency="5s"
I0812 22:06:47.259326       1 flags.go:33] FLAG: --logtostderr="true"
I0812 22:06:47.259332       1 flags.go:33] FLAG: --master-service-namespace="default"
I0812 22:06:47.259339       1 flags.go:33] FLAG: --max-connection-bytes-per-sec="0"
I0812 22:06:47.259346       1 flags.go:33] FLAG: --max-mutating-requests-inflight="200"
I0812 22:06:47.259352       1 flags.go:33] FLAG: --max-requests-inflight="400"
I0812 22:06:47.259359       1 flags.go:33] FLAG: --min-request-timeout="1800"
I0812 22:06:47.259365       1 flags.go:33] FLAG: --oidc-ca-file=""
I0812 22:06:47.259372       1 flags.go:33] FLAG: --oidc-client-id=""
I0812 22:06:47.259378       1 flags.go:33] FLAG: --oidc-groups-claim=""
I0812 22:06:47.259384       1 flags.go:33] FLAG: --oidc-groups-prefix=""
I0812 22:06:47.259390       1 flags.go:33] FLAG: --oidc-issuer-url=""
I0812 22:06:47.259397       1 flags.go:33] FLAG: --oidc-required-claim=""
I0812 22:06:47.259405       1 flags.go:33] FLAG: --oidc-signing-algs="[RS256]"
I0812 22:06:47.259417       1 flags.go:33] FLAG: --oidc-username-claim="sub"
I0812 22:06:47.259424       1 flags.go:33] FLAG: --oidc-username-prefix=""
I0812 22:06:47.259431       1 flags.go:33] FLAG: --port="0"
I0812 22:06:47.259437       1 flags.go:33] FLAG: --profiling="true"
I0812 22:06:47.259444       1 flags.go:33] FLAG: --proxy-client-cert-file="/var/lib/minikube/certs/front-proxy-client.crt"
I0812 22:06:47.259451       1 flags.go:33] FLAG: --proxy-client-key-file="/var/lib/minikube/certs/front-proxy-client.key"
I0812 22:06:47.259460       1 flags.go:33] FLAG: --request-timeout="1m0s"
I0812 22:06:47.259467       1 flags.go:33] FLAG: --requestheader-allowed-names="[front-proxy-client]"
I0812 22:06:47.259490       1 flags.go:33] FLAG: --requestheader-client-ca-file="/var/lib/minikube/certs/front-proxy-ca.crt"
I0812 22:06:47.259498       1 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[X-Remote-Extra-]"
I0812 22:06:47.259510       1 flags.go:33] FLAG: --requestheader-group-headers="[X-Remote-Group]"
I0812 22:06:47.259521       1 flags.go:33] FLAG: --requestheader-username-headers="[X-Remote-User]"
I0812 22:06:47.259533       1 flags.go:33] FLAG: --runtime-config=""
I0812 22:06:47.259542       1 flags.go:33] FLAG: --secure-port="8443"
I0812 22:06:47.259549       1 flags.go:33] FLAG: --service-account-api-audiences="[]"
I0812 22:06:47.259559       1 flags.go:33] FLAG: --service-account-issuer=""
I0812 22:06:47.259566       1 flags.go:33] FLAG: --service-account-key-file="[/var/lib/minikube/certs/sa.pub]"
I0812 22:06:47.259581       1 flags.go:33] FLAG: --service-account-lookup="true"
I0812 22:06:47.259587       1 flags.go:33] FLAG: --service-account-max-token-expiration="0s"
I0812 22:06:47.259594       1 flags.go:33] FLAG: --service-account-signing-key-file=""
I0812 22:06:47.259600       1 flags.go:33] FLAG: --service-cluster-ip-range="10.96.0.0/12"
I0812 22:06:47.259609       1 flags.go:33] FLAG: --service-node-port-range="30000-32767"
I0812 22:06:47.259618       1 flags.go:33] FLAG: --skip-headers="false"
I0812 22:06:47.259625       1 flags.go:33] FLAG: --skip-log-headers="false"
I0812 22:06:47.259631       1 flags.go:33] FLAG: --ssh-keyfile=""
I0812 22:06:47.259637       1 flags.go:33] FLAG: --ssh-user=""
I0812 22:06:47.259644       1 flags.go:33] FLAG: --stderrthreshold="2"
I0812 22:06:47.259651       1 flags.go:33] FLAG: --storage-backend=""
I0812 22:06:47.259657       1 flags.go:33] FLAG: --storage-media-type="application/vnd.kubernetes.protobuf"
I0812 22:06:47.259667       1 flags.go:33] FLAG: --target-ram-mb="0"
I0812 22:06:47.259675       1 flags.go:33] FLAG: --tls-cert-file="/var/lib/minikube/certs/apiserver.crt"
I0812 22:06:47.259682       1 flags.go:33] FLAG: --tls-cipher-suites="[]"
I0812 22:06:47.259692       1 flags.go:33] FLAG: --tls-min-version=""
I0812 22:06:47.259699       1 flags.go:33] FLAG: --tls-private-key-file="/var/lib/minikube/certs/apiserver.key"
I0812 22:06:47.259706       1 flags.go:33] FLAG: --tls-sni-cert-key="[]"
I0812 22:06:47.259714       1 flags.go:33] FLAG: --token-auth-file=""
I0812 22:06:47.259720       1 flags.go:33] FLAG: --v="10"
I0812 22:06:47.259727       1 flags.go:33] FLAG: --version="false"
I0812 22:06:47.259735       1 flags.go:33] FLAG: --vmodule=""
I0812 22:06:47.259742       1 flags.go:33] FLAG: --watch-cache="true"
I0812 22:06:47.259749       1 flags.go:33] FLAG: --watch-cache-sizes="[]"
I0812 22:06:47.259767       1 services.go:45] Setting service IP to "10.96.0.1" (read-write).
@tamalsaha

This comment has been minimized.

Copy link
Member Author

commented Aug 15, 2019

@liggitt , I wonder if you got a chance to look into this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.