This repository has been archived by the owner. It is now read-only.

The kubernetes on Mesos can't run successfully on the same machine #673

Closed
NanXiao opened this Issue Dec 8, 2015 · 19 comments

Comments

Projects
None yet
3 participants
@NanXiao

NanXiao commented Dec 8, 2015

Hi all,

I want to set up kubernetes on Mesos on the same local Ubuntu machine. And all processes use public IP address(16.187.250.141):

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 24:be:05:18:a7:46 brd ff:ff:ff:ff:ff:ff
    inet 16.187.250.141/22 brd 16.187.251.255 scope global em1
       valid_lft forever preferred_lft forever
    inet6 fe80::26be:5ff:fe18:a746/64 scope link
       valid_lft forever preferred_lft forever

The Mesos start up command:

nan@ubuntu:~/mesos-0.25.0/build$ ./bin/mesos-master.sh --ip=16.187.250.141 --work_dir=/var/lib/mesos

The kubernetes related environmental variables:

$ export
......
declare -x KUBERNETES_MASTER="http://16.187.250.141:8888"
declare -x KUBERNETES_MASTER_IP="16.187.250.141"
declare -x MESOS_MASTER="16.187.250.141:5050"
......

Executing following command:

$ km apiserver \
  --address=${KUBERNETES_MASTER_IP} \
  --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001 \
  --service-cluster-ip-range=10.10.10.0/24 \
  --port=8888 \
  --cloud-provider=mesos \
  --cloud-config=mesos-cloud.conf \
  --secure-port=0 \
  --v=1 >apiserver.log 2>&1 &

The apiserver.log is like this:

    $ cat apiserver.log
I1208 07:46:51.545451   25391 server.go:387] Will report 16.187.250.141 as public IP address.
I1208 07:46:51.545577   25391 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1208 07:46:51.545614   25391 standalone.go:69] spinning up asyc master detector poller
I1208 07:46:51.545631   25391 standalone.go:78] spawning asyc master detector listener
I1208 07:46:51.545706   25391 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1208 07:46:51.545824   25391 master.go:379] Node port range unspecified. Defaulting to 30000-32767.
I1208 07:46:51.549764   25391 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1208 07:46:51.550024   25391 client.go:149] cloud master changed to '16.187.250.141:5050'
[restful] 2015/12/08 07:46:51 log.go:30: [restful/swagger] listing is available at https://16.187.250.141:6443/swaggerapi/
[restful] 2015/12/08 07:46:51 log.go:30: [restful/swagger] https://16.187.250.141:6443/swaggerui/ is mapped to folder /swagger-ui/
I1208 07:46:51.637467   25391 server.go:682] Serving insecurely on 16.187.250.141:8888

It seems OK.

But executing this command:

$ km controller-manager \
     --master=${KUBERNETES_MASTER_IP}:8888 \
     --cloud-provider=mesos \
    --cloud-config=./mesos-cloud.conf  \
    --v=1 >controller.log 2>&1 &

The controller.log is like this:

$ cat controller.log
I1208 07:47:09.403613   25410 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1208 07:47:09.403741   25410 standalone.go:69] spinning up asyc master detector poller
I1208 07:47:09.403756   25410 standalone.go:78] spawning asyc master detector listener
I1208 07:47:09.403880   25410 nodecontroller.go:132] Sending events to api server.
E1208 07:47:09.404279   25410 controllermanager.go:151] Failed to start service controller: the cloud provider does not support TCP load balancers.
I1208 07:47:09.404536   25410 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1208 07:47:09.404568   25410 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1208 07:47:09.404762   25410 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1208 07:47:09.408151   25410 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1208 07:47:09.408258   25410 client.go:149] cloud master changed to '16.187.250.141:5050'
E1208 07:47:10.024485   25410 util.go:82] Recovered from panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:76
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/util/util.go:64
/usr/local/go/src/runtime/asm_amd64.s:437
/usr/local/go/src/runtime/panic.go:423
/usr/local/go/src/runtime/panic.go:42
/usr/local/go/src/runtime/sigpanic_unix.go:24
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/mesos/client.go:100
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/mesos/client.go:167
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/mesos/mesos.go:224
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/mesos/mesos.go:258
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/contrib/mesos/pkg/node/statusupdater.go:66
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/contrib/mesos/pkg/runtime/util.go:115
/home/nan/kubernetes/_output/local/go/src/k8s.io/kubernetes/contrib/mesos/pkg/runtime/util.go:116
/usr/local/go/src/runtime/asm_amd64.s:1696
E1208 07:47:10.026275   25410 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1208 07:47:10.027093   25410 resource_quota_controller.go:62] Synchronization error: the server does not allow access to the requested resource (get resourceQuotas) (&errors.StatusError{ErrStatus:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"the server does not allow access to the requested resource (get resourceQuotas)", Reason:"Forbidden", Details:(*unversioned.StatusDetails)(0xc820244280), Code:403}})

So why can't km controller-manager run successfully? Could anyone give some clues? Thanks in advance!

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 8, 2015

The panic is fixed in kubernetes/kubernetes#18017. This will be part of the v0.7.1 release. It's not critical for the DCOS package because there we have S6 supervising and restarting the controller manager.

sttts commented Dec 8, 2015

The panic is fixed in kubernetes/kubernetes#18017. This will be part of the v0.7.1 release. It's not critical for the DCOS package because there we have S6 supervising and restarting the controller manager.

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 8, 2015

I would close this ticket as duplicate as #656. If you think it's a different issue, feel free to re-open it.

sttts commented Dec 8, 2015

I would close this ticket as duplicate as #656. If you think it's a different issue, feel free to re-open it.

@sttts sttts closed this Dec 8, 2015

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 8, 2015

@sttts : But the following logs are the same reason?

E1208 07:47:10.026275   25410 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1208 07:47:10.027093   25410 resource_quota_controller.go:62] Synchronization error: the server does not allow access to the requested resource (get resourceQuotas) (&errors.StatusError{ErrStatus:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"the server does not allow access to the requested resource (get resourceQuotas)", Reason:"Forbidden", Details:(*unversioned.StatusDetails)(0xc820244280), Code:403}})

NanXiao commented Dec 8, 2015

@sttts : But the following logs are the same reason?

E1208 07:47:10.026275   25410 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1208 07:47:10.027093   25410 resource_quota_controller.go:62] Synchronization error: the server does not allow access to the requested resource (get resourceQuotas) (&errors.StatusError{ErrStatus:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"the server does not allow access to the requested resource (get resourceQuotas)", Reason:"Forbidden", Details:(*unversioned.StatusDetails)(0xc820244280), Code:403}})
@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 8, 2015

The apiserver is not ready it seems.

sttts commented Dec 8, 2015

The apiserver is not ready it seems.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 8, 2015

But from apiserver log, it seems OK.

NanXiao commented Dec 8, 2015

But from apiserver log, it seems OK.

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 8, 2015

Did you wait starting the ctrl manager until the apiserver is up an running? I think this is a startup race causing the panic. The referenced PR fixes that. The ListWatch construct accessing the apiserver will cope with errors and try again.

sttts commented Dec 8, 2015

Did you wait starting the ctrl manager until the apiserver is up an running? I think this is a startup race causing the panic. The referenced PR fixes that. The ListWatch construct accessing the apiserver will cope with errors and try again.

@jdef

This comment has been minimized.

Show comment
Hide comment
@jdef

jdef Dec 8, 2015

Member

according to the timestamps the controller manager was started after the
apiserver. there's nothing guaranteeing that the nodes != nil in the state
cache (nor is anything logging the error that refill() might return):

https://github.com/mesosphere/kubernetes/blob/release-v0.7.0-v1.1.1/pkg/cloudprovider/providers/mesos/client.go#L100

furthermore, it looks like we're still using state.json to poll the
master
https://github.com/mesosphere/kubernetes/blob/release-v0.7.0-v1.1.1/pkg/cloudprovider/providers/mesos/client.go#L176
... this endpoint is renamed /state in recent release of mesos

state.json should still be supported in mesos 0.25. however, it's
probably going away in 0.26:

https://issues.apache.org/jira/browse/MESOS-2719

On Tue, Dec 8, 2015 at 6:22 AM, Dr. Stefan Schimanski <
notifications@github.com> wrote:

Did you wait starting the ctrl manager until the apiserver is up an
running? I think this is a startup race causing the panic. The referenced
PR fixes that. The ListWatch construct accessing the apiserver will cope
with errors and try again.


Reply to this email directly or view it on GitHub
#673 (comment)
.

Member

jdef commented Dec 8, 2015

according to the timestamps the controller manager was started after the
apiserver. there's nothing guaranteeing that the nodes != nil in the state
cache (nor is anything logging the error that refill() might return):

https://github.com/mesosphere/kubernetes/blob/release-v0.7.0-v1.1.1/pkg/cloudprovider/providers/mesos/client.go#L100

furthermore, it looks like we're still using state.json to poll the
master
https://github.com/mesosphere/kubernetes/blob/release-v0.7.0-v1.1.1/pkg/cloudprovider/providers/mesos/client.go#L176
... this endpoint is renamed /state in recent release of mesos

state.json should still be supported in mesos 0.25. however, it's
probably going away in 0.26:

https://issues.apache.org/jira/browse/MESOS-2719

On Tue, Dec 8, 2015 at 6:22 AM, Dr. Stefan Schimanski <
notifications@github.com> wrote:

Did you wait starting the ctrl manager until the apiserver is up an
running? I think this is a startup race causing the panic. The referenced
PR fixes that. The ListWatch construct accessing the apiserver will cope
with errors and try again.


Reply to this email directly or view it on GitHub
#673 (comment)
.

@jdef

This comment has been minimized.

Show comment
Hide comment
@jdef

jdef Dec 8, 2015

Member

xref #618 state.json is going away in mesos v0.26

Member

jdef commented Dec 8, 2015

xref #618 state.json is going away in mesos v0.26

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 9, 2015

@sttts @jdef : So how can I resolve this issue? P.S. My k8s is master branch, and Mesos is also the new released v0.25.

NanXiao commented Dec 9, 2015

@sttts @jdef : So how can I resolve this issue? P.S. My k8s is master branch, and Mesos is also the new released v0.25.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 9, 2015

@sttts : BTW, I can't re-open this issue, maybe need your help.

NanXiao commented Dec 9, 2015

@sttts : BTW, I can't re-open this issue, maybe need your help.

@sttts sttts reopened this Dec 9, 2015

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts commented Dec 9, 2015

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 9, 2015

@sttts I download the k8s yesterday, and check it again, the k8s code has contained this patch. So I don't think it is the root cause. thx!

NanXiao commented Dec 9, 2015

@sttts I download the k8s yesterday, and check it again, the k8s code has contained this patch. So I don't think it is the root cause. thx!

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 9, 2015

@sttts @jdef : BTW, my Go version is the latest 1.5.1. Maybe Go version is too new?

NanXiao commented Dec 9, 2015

@sttts @jdef : BTW, my Go version is the latest 1.5.1. Maybe Go version is too new?

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 9, 2015

@NanXiao please check out kubernetes/kubernetes#18427. It fixes to panics. One of these you encountered.

sttts commented Dec 9, 2015

@NanXiao please check out kubernetes/kubernetes#18427. It fixes to panics. One of these you encountered.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 10, 2015

@sttts :Yes, the panic issue goes off now! Thanks!

But the km controller-manager command still fails:

$ km controller-manager \
     --master=${KUBERNETES_MASTER_IP}:8888 \
     --cloud-provider=mesos \
    --cloud-config=./mesos-cloud.conf  \
    --v=1 >controller.log 2>&1 &

And the controller.log is like this:

$ cat controller.log
I1209 23:47:41.782961   11750 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1209 23:47:41.783194   11750 standalone.go:69] spinning up asyc master detector poller
I1209 23:47:41.783222   11750 standalone.go:78] spawning asyc master detector listener
I1209 23:47:41.783291   11750 nodecontroller.go:132] Sending events to api server.
E1209 23:47:41.783683   11750 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load balancers.
I1209 23:47:41.783712   11750 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1209 23:47:41.783900   11750 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1209 23:47:41.783932   11750 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1209 23:47:41.787999   11750 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1209 23:47:41.788083   11750 client.go:155] cloud master changed to '16.187.250.141:5050'
E1209 23:47:42.405005   11750 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
E1209 23:47:42.405189   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:47.645666   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:52.852565   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:53.026991   11750 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
E1209 23:47:58.070350   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)

But from the log of Mesos master:

    ......
I1209 23:33:00.240255 11664 master.cpp:378] Flags at startup: --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --ip="16.187.250.141" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --port="5050" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/home/nan/mesos-0.25.0/build/../src/webui" --work_dir="/var/lib/mesos" --zk_session_timeout="10secs"
I1209 23:33:00.240664 11664 master.cpp:425] Master allowing unauthenticated frameworks to register
I1209 23:33:00.240689 11664 master.cpp:430] Master allowing unauthenticated slaves to register
I1209 23:33:00.240737 11664 master.cpp:467] Using default 'crammd5' authenticator
W1209 23:33:00.240785 11664 authenticator.cpp:505] No credentials provided, authentication requests will be refused
I1209 23:33:00.240808 11664 authenticator.cpp:512] Initializing server SASL
I1209 23:33:00.257146 11678 master.cpp:1603] The newly elected leader is master@16.187.250.141:5050 with id fcb6329b-3f65-4f12-a3b3-70ee5e64e9b7
    ......

It seems there should be OK for k8s to connect Mesos master. Could you give some suggestions?
Thanks in advance!

Best Regards
Nan Xiao

NanXiao commented Dec 10, 2015

@sttts :Yes, the panic issue goes off now! Thanks!

But the km controller-manager command still fails:

$ km controller-manager \
     --master=${KUBERNETES_MASTER_IP}:8888 \
     --cloud-provider=mesos \
    --cloud-config=./mesos-cloud.conf  \
    --v=1 >controller.log 2>&1 &

And the controller.log is like this:

$ cat controller.log
I1209 23:47:41.782961   11750 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1209 23:47:41.783194   11750 standalone.go:69] spinning up asyc master detector poller
I1209 23:47:41.783222   11750 standalone.go:78] spawning asyc master detector listener
I1209 23:47:41.783291   11750 nodecontroller.go:132] Sending events to api server.
E1209 23:47:41.783683   11750 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load balancers.
I1209 23:47:41.783712   11750 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1209 23:47:41.783900   11750 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1209 23:47:41.783932   11750 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1209 23:47:41.787999   11750 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1209 23:47:41.788083   11750 client.go:155] cloud master changed to '16.187.250.141:5050'
E1209 23:47:42.405005   11750 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
E1209 23:47:42.405189   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:47.645666   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:52.852565   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)
E1209 23:47:53.026991   11750 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
E1209 23:47:58.070350   11750 nodecontroller.go:202] Error monitoring node status: the server does not allow access to the requested resource (get nodes)

But from the log of Mesos master:

    ......
I1209 23:33:00.240255 11664 master.cpp:378] Flags at startup: --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --initialize_driver_logging="true" --ip="16.187.250.141" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --port="5050" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/home/nan/mesos-0.25.0/build/../src/webui" --work_dir="/var/lib/mesos" --zk_session_timeout="10secs"
I1209 23:33:00.240664 11664 master.cpp:425] Master allowing unauthenticated frameworks to register
I1209 23:33:00.240689 11664 master.cpp:430] Master allowing unauthenticated slaves to register
I1209 23:33:00.240737 11664 master.cpp:467] Using default 'crammd5' authenticator
W1209 23:33:00.240785 11664 authenticator.cpp:505] No credentials provided, authentication requests will be refused
I1209 23:33:00.240808 11664 authenticator.cpp:512] Initializing server SASL
I1209 23:33:00.257146 11678 master.cpp:1603] The newly elected leader is master@16.187.250.141:5050 with id fcb6329b-3f65-4f12-a3b3-70ee5e64e9b7
    ......

It seems there should be OK for k8s to connect Mesos master. Could you give some suggestions?
Thanks in advance!

Best Regards
Nan Xiao

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 10, 2015

Try with a higher log level. I think you can see the complete HTTP url then that controller manager uses. Just try that manually then.

sttts commented Dec 10, 2015

Try with a higher log level. I think you can see the complete HTTP url then that controller manager uses. Just try that manually then.

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 10, 2015

@sttts: the controller prints more log:

$ more controller.log
I1210 05:33:13.570789   12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253   12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294   12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469   12082 standalone.go:67] Detect()
I1210 05:33:13.571485   12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501   12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608   12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706   12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831   12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994   12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124   12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208   12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393   12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452   12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
 usage
I1210 05:33:13.572479   12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806   12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013   12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098   12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118   12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218   12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579   12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
E1210 05:33:14.192639   12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192692   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192703   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192712   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192722   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192823   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
I1210 05:33:14.192842   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192851   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192861   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192905   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935   12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034   12082 request.go:861] Response Body: <HTML><HEAD>

But how can I know the response matches which request? Thx!

NanXiao commented Dec 10, 2015

@sttts: the controller prints more log:

$ more controller.log
I1210 05:33:13.570789   12082 controllermanager.go:217] Creating hostIP:hostPort endpoint controller
I1210 05:33:13.571253   12082 mesos.go:75] new mesos cloud, master='16.187.250.141:5050'
I1210 05:33:13.571294   12082 standalone.go:46] creating new standalone detector for &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16
.187.250.141:5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.571469   12082 standalone.go:67] Detect()
I1210 05:33:13.571485   12082 standalone.go:69] spinning up asyc master detector poller
I1210 05:33:13.571501   12082 standalone.go:78] spawning asyc master detector listener
I1210 05:33:13.571608   12082 nodecontroller.go:132] Sending events to api server.
I1210 05:33:13.571706   12082 standalone.go:140] polling for master leadership at '16.187.250.141:5050'
I1210 05:33:13.571729   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/services
I1210 05:33:13.571757   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.571772   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.571831   12082 standalone.go:80] waiting for polled to send updates
I1210 05:33:13.571986   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/replicationcontrollers
I1210 05:33:13.572010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572026   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
E1210 05:33:13.571994   12082 controllermanager.go:152] Failed to start service controller: the cloud provider does not support TCP load ba
lancers.
I1210 05:33:13.572124   12082 round_trippers.go:222] GET http://16.187.250.141:8888/apis/extensions/v1beta1/daemonsets
I1210 05:33:13.572151   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572167   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572208   12082 persistentvolume_claim_binder_controller.go:346] Starting PersistentVolumeClaimBinder
I1210 05:33:13.572262   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572268   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572300   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572314   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572305   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572331   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/host-path"
I1210 05:33:13.572371   12082 plugins.go:265] Loaded volume plugin "kubernetes.io/nfs"
I1210 05:33:13.572393   12082 persistentvolume_recycler_controller.go:211] Starting PersistentVolumeRecycler
I1210 05:33:13.572452   12082 resource_quota_controller.go:133] Resource quota controller queued all resource quota for full calculation of
 usage
I1210 05:33:13.572479   12082 client.go:77] Reloading cached Mesos state
I1210 05:33:13.572283   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572499   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.572346   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572508   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572531   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572535   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572547   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572571   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572584   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572521   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572612   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572632   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.572662   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572676   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572684   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/resourcequotas
I1210 05:33:13.572702   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572714   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572756   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumeclaims
I1210 05:33:13.572551   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572791   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572815   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572329   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/nodes
I1210 05:33:13.572846   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572858   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572864   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/serviceaccounts?fieldSelector=metadata.name%3Dde
fault
I1210 05:33:13.572869   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/persistentvolumes
I1210 05:33:13.572885   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572892   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572777   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.572948   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572167   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573010   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573019   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572834   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573072   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572876   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573120   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.572641   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/namespaces
I1210 05:33:13.573193   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573202   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.573286   12082 round_trippers.go:222] GET http://16.187.250.141:8888/api/v1/pods
I1210 05:33:13.573304   12082 round_trippers.go:229] Request Headers:
I1210 05:33:13.573313   12082 round_trippers.go:232]     User-Agent: km/v1.2.0 (linux/amd64) kubernetes/c2cfcd5
I1210 05:33:13.576806   12082 standalone.go:214] Got mesos state, content length 1795
I1210 05:33:13.577013   12082 standalone.go:147] detected leadership change from '<nil>' to 'master@16.187.250.141:5050'
I1210 05:33:13.577098   12082 standalone.go:185] master leader poller sleeping for 29.994633799s
I1210 05:33:13.577118   12082 standalone.go:88] detected master change: &MasterInfo{Id:*master,Ip:*0,Port:*5050,Pid:*master@16.187.250.141:
5050,Hostname:*16.187.250.141,Version:nil,Address:nil,XXX_unrecognized:[],}
I1210 05:33:13.577218   12082 client.go:155] cloud master changed to '16.187.250.141:5050'
I1210 05:33:14.192579   12082 round_trippers.go:247] Response Status: 403 Forbidden in 619 milliseconds
I1210 05:33:14.192610   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192621   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
E1210 05:33:14.192639   12082 statusupdater.go:68] Error listing slaves without kubelet: HTTP request failed with code 403: 403 Forbidden
I1210 05:33:14.192667   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192692   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192703   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192712   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192722   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192765   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.192800   12082 round_trippers.go:250] Response Headers:
I1210 05:33:14.192812   12082 round_trippers.go:253]     Pragma: no-cache
I1210 05:33:14.192823   12082 round_trippers.go:253]     Content-Type: text/html; charset=utf-8
I1210 05:33:14.192833   12082 round_trippers.go:253]     Proxy-Connection: Keep-Alive
I1210 05:33:14.192842   12082 round_trippers.go:253]     Connection: Keep-Alive
I1210 05:33:14.192851   12082 round_trippers.go:253]     Content-Length: 606
I1210 05:33:14.192861   12082 round_trippers.go:253]     Cache-Control: no-cache
I1210 05:33:14.192905   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192935   12082 request.go:861] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.192785   12082 request.go:804] Response Body: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">

</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
I1210 05:33:14.193025   12082 round_trippers.go:247] Response Status: 403 Forbidden in 620 milliseconds
I1210 05:33:14.193034   12082 request.go:861] Response Body: <HTML><HEAD>

But how can I know the response matches which request? Thx!

@NanXiao

This comment has been minimized.

Show comment
Hide comment
@NanXiao

NanXiao Dec 11, 2015

@sttts The root cause has been found: because our server has set proxy. Thanks very much for all your help!

NanXiao commented Dec 11, 2015

@sttts The root cause has been found: because our server has set proxy. Thanks very much for all your help!

@NanXiao NanXiao closed this Dec 11, 2015

@sttts

This comment has been minimized.

Show comment
Hide comment
@sttts

sttts Dec 11, 2015

Great, that you found the issue!

sttts commented Dec 11, 2015

Great, that you found the issue!

@jdef jdef added this to the v0.7.1 milestone Dec 11, 2015

@jdef jdef removed the blocked label Dec 14, 2015

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.