New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no endpoint created for service #11795

Closed
loky9000 opened this Issue Jul 24, 2015 · 18 comments

Comments

Projects
None yet
6 participants
@loky9000
Copy link

loky9000 commented Jul 24, 2015

HI
I've just a begin to play with kubernetes and tried to create couple rc+service from yours examples. https://github.com/GoogleCloudPlatform/kubernetes/tree/master/examples/celery-rabbitmq
but i faced that flower-service not expose any endpoint after create

kubectl  describe svc flower-service
Name:           flower-service
Namespace:      default
Labels:         name=flower
Selector:       app=taskQueue,component=flower
Type:           LoadBalancer
IP:         10.10.10.115
Port:           main    5555/TCP
NodePort:       main    30693/TCP
Endpoints:      <none>
Session Affinity:   None
No events. 

It's ok for rabbitmq-service but endpoint has port 31000 which not described in service definition

kubectl --server=10.180.228.105:8888 describe svc rabbitmq-service
Name:           rabbitmq-service
Namespace:      default
Labels:         name=rabbitmq
Selector:       app=taskQueue,component=rabbitmq
Type:           ClusterIP
IP:         10.10.10.159
Port:           <unnamed>   5672/TCP
Endpoints:      10.235.129.14:31000
Session Affinity:   None
No events.

How it's posible? This is expected behaviour ?

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 24, 2015

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 24, 2015

flower-controller

 apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: flower
  name: flower-controller
spec:
  replicas: 1
  selector:
    component: flower
  template:
    metadata:
      labels:
        app: taskQueue
        component: flower
    spec:
      containers:
      - image: endocode/flower
        name: flower
        resources:
          limits:
            cpu: 100m

flower-service

apiVersion: v1
kind: Service
metadata:
  labels:
    name: flower
  name: flower-service
spec:
  ports:
  - port: 5555
  selector:
    app: taskQueue
    component: flower
  type: LoadBalancer

as you can see service selector match to rc labels

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 24, 2015

I mis-parsed the original message, sorry.

Are these the full specs? The port should be assumed to be 5555 since not targetPort is specified.

What kubernetes version, what OS, what cloud, and what process did you use to make this happen?

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 27, 2015

Kubernetes v1.0.1
Ubuntu14.04
Mesos 0.22.0
service created by kubectl
nothing change with targetPort
kubectl expose create endpoint for pod but not rc

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 27, 2015

If I recall, you are using Mesos, so normal rules about endpoints and ports
do not apply. That said,
http://kubernetes.io/v1.0/docs/user-guide/debugging-services.html still
applies mostly - did you try following that?

On Mon, Jul 27, 2015 at 1:15 PM, loky9000 notifications@github.com wrote:

Kubernetes v1.0.1
Ubuntu14.04
Mesos 0.22.0
service created by kubectl

nothing change with targetPort

kubectl expose create endpoint for pod but not rc


Reply to this email directly or view it on GitHub
#11795 (comment)
.

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 28, 2015

endpoints creates for pod but target port wrong

$kubectl expose pod mysql --port=3306 --target-port=3306
NAME LABELS SELECTOR IP(S) PORT(S)
mysql name=mysql name=mysql 3306/TCP
$ kubectl describe svc mysql
Name: mysql
Namespace: default
Labels: name=mysql
Selector: name=mysql
Type: ClusterIP
IP: 10.10.10.206
Port: 3306/TCP
Endpoints: 10.183.211.227:31000
Session Affinity: None
No events

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 28, 2015

Again, Mesos does different things with the ports. What is the actual
problem you are experiencing?

On Tue, Jul 28, 2015 at 12:53 AM, loky9000 notifications@github.com wrote:

endpoints creates for pod but target port wrong

$kubectl expose pod mysql --port=3306 --target-port=3306
NAME LABELS SELECTOR IP(S) PORT(S)
mysql name=mysql name=mysql 3306/TCP
$ kubectl describe svc mysql
Name: mysql
Namespace: default
Labels: name=mysql
Selector: name=mysql
Type: ClusterIP
IP: 10.10.10.206
Port: 3306/TCP
Endpoints: 10.183.211.227:31000
Session Affinity: None
No events


Reply to this email directly or view it on GitHub
#11795 (comment)
.

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 28, 2015

i want to expose custom port for service not random. why --target-port=3306 ignored?

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 29, 2015

Mesos maps all pod ports to host ports. That's not something you can
control. Is something actually not working?

On Tue, Jul 28, 2015 at 2:43 PM, loky9000 notifications@github.com wrote:

i want to expose custom port for service not random. why
--target-port=3306 ignored?


Reply to this email directly or view it on GitHub
#11795 (comment)
.

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 29, 2015

endpoint creating it something mysterious.
We have a pod:

apiVersion: v1
kind: Pod
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  containers:
    - resources:
        limits :
          cpu: 0.5
      image: mysql:5.7
      name: mysql
      env:
        - name: MYSQL_ROOT_PASSWORD
          # change this
          value: 1qaz2wsx
      ports:
      - containerPort: 5555
        name: mysql

describe pod mysql

kubectl --server=10.234.1.41:8888 describe  pod mysql
Name:               mysql
Namespace:          default
Image(s):           mysql:5.7
Node:               ec2-54-211-177-168.compute-1.amazonaws.com/10.230.39.11
Labels:             name=mysql
Status:             Running
Reason:
Message:
IP:             172.17.0.1
Replication Controllers:    <none>
Containers:
  mysql:
    Image:  mysql:5.7
    Limits:
      cpu:      500m
    State:      Running
      Started:      Wed, 29 Jul 2015 11:56:17 +0000
    Ready:      True
    Restart Count:  0
Conditions:
  Type      Status
  Ready     True
Events:
  FirstSeen             LastSeen            Count   From                            SubobjectPath               Reason      Message
  Wed, 29 Jul 2015 11:54:27 +0000   Wed, 29 Jul 2015 11:54:27 +0000 1   {scheduler }                                            scheduled   Successfully assigned mysql to ec2-54-211-177-168.compute-1.amazonaws.com
  Wed, 29 Jul 2015 11:55:13 +0000   Wed, 29 Jul 2015 11:55:13 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    implicitly required container POD   pulled      Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
  Wed, 29 Jul 2015 11:55:13 +0000   Wed, 29 Jul 2015 11:55:13 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    implicitly required container POD   created     Created with docker id 5ec9ab587402
  Wed, 29 Jul 2015 11:55:14 +0000   Wed, 29 Jul 2015 11:55:14 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    implicitly required container POD   started     Started with docker id 5ec9ab587402
  Wed, 29 Jul 2015 11:56:16 +0000   Wed, 29 Jul 2015 11:56:16 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    spec.containers{mysql}          pulled      Successfully pulled image "mysql:5.7"
  Wed, 29 Jul 2015 11:56:17 +0000   Wed, 29 Jul 2015 11:56:17 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    spec.containers{mysql}          created     Created with docker id 63cc7a02016f
  Wed, 29 Jul 2015 11:56:17 +0000   Wed, 29 Jul 2015 11:56:17 +0000 1   {kubelet ec2-54-211-177-168.compute-1.amazonaws.com}    spec.containers{mysql}          started     Started with docker id 63cc7a02016f

Lets try to expose that pod

kubectl --server=10.234.1.41:8888 expose pod mysql --port=3306 --target-port=3306
NAME      LABELS       SELECTOR     IP(S)     PORT(S)
mysql     name=mysql   name=mysql             3306/TCP

looks good but...

kubectl --server=10.234.1.41:8888 describe svc  mysql
Name:           mysql
Namespace:      default
Labels:         name=mysql
Selector:       name=mysql
Type:           ClusterIP
IP:         10.10.10.241
Port:           <unnamed>   3306/TCP
Endpoints:      <none>
Session Affinity:   None
No events.

no endpoints created for this time and no ports listen on mesos slave where mysql docker container placed - so no access from external to mysql service on pod.

So please give me the way what i'm doing wrong?
Why endpoints has been created previously but not created at this time with the same mysql pod configuration?what does it depend?
Mysql must accept connections from external and from another pods in this particular cluster

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jul 30, 2015

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 30, 2015

Mesos maps all pod ports to host ports. That's not something you can
control. Is something actually not working?

Currently i have two pods mysql and tomcat which served on different mesos slaves
and i can't access from tomcat container to mysql 3306 port via localhost:3306 like described here: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/admin/networking.md

i need to connect tomcat app to db so how i can point mysql container host to tomcat container?
I'm tried to use skydns for acces to mysql host via internal kubernetes dns service but it;s not working for me
#11924

@loky9000

This comment has been minimized.

Copy link

loky9000 commented Jul 30, 2015

and another thing:
endpoints on all mesos slaves point on port 31000 maybe it's some mesos restriction

kubectl --server=10.234.1.41:8888 get endpoints --namespace=kube-system
NAME       ENDPOINTS
kube-ui    10.182.203.143:31000
ubuntu@ip-10-234-1-41:/opt/kubernetes/repo$ kubectl --server=10.234.1.41:8888 get endpoints
NAME             ENDPOINTS
k8sm-scheduler   10.234.1.41:10251
kubernetes       10.234.1.41:6443
mysql            10.91.176.3:31000
tomcat           10.180.202.211:31000
@karlkfi

This comment has been minimized.

Copy link
Contributor

karlkfi commented Aug 7, 2015

Not all of the examples have been tested on Kubernetes-Mesos.
It looks like you're missing a requires port name (a mesos-specific requirement). I don't know the exact reason for that, but it's documented on the mesos/contrib known issues page: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/contrib/mesos/docs/issues.md

@jdef may have more details on why.

Let us know if that solves the issue.

@jdef

This comment has been minimized.

Copy link
Contributor

jdef commented Aug 7, 2015

@loky9000 which endpoint-controller are you using?

k8s-mesos implements some workarounds for people that have default docker networking (non-cluster-routable IPs for containers). as part of these workarounds, for each pod instance we allocate a host port for each pod port and create endpoints such that they map to the hostIP:hostPort of the pod, instead of the k8s default (podIP:podPort). connections to the allocated hostPort are proxied to the podPort by docker.

31000 looks like a port that the scheduler is allocating from a mesos resource offer (that's the beginning of the default "ports" resource range for slaves). if you try to create a pod with a pod.spec.container.port that uses a hostPort value outside of the range of the resource offers in the mesos cluster (the "ports" resources configured on your slaves), the pod will never be scheduled because the scheduler will never receive an offer that satisfies the hostPort requirements of the pod spec.

service ports, as karl mentioned (and linked to via issues.md) currently require some special treatment by the end user (try defining a name for each service.spec.port).

@jdef

This comment has been minimized.

Copy link
Contributor

jdef commented Aug 18, 2015

could be related to mesosphere/kubernetes-mesos#322

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Sep 2, 2015

Is this still valid, or should it be closed?

@jdef

This comment has been minimized.

Copy link
Contributor

jdef commented Sep 2, 2015

Needs more information from @loky9000. It sounds like you are trying to run two pods, and with the expectation that they'll communicate through localhost:xyz-mapped addresses, which will never work because that's not how k8s networking works. Only containers within the same pod can use localhost:xyz. I have no idea how you're setting up kubedns.

This has been sitting a while. Closing out for now, please re-open if you have additional context/details to provide.

@jdef jdef closed this Sep 2, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment