Example with consul.io

[throrin19]

Hi,
I start to change all of my company architecture from monolyth API to microservices approache. After several weeks, we find Kong to (correct me if i am wrong) make the proxy and api gateway part and Consul to make the registrator part.

After several days, I find how to add into consul automatically all my services and now i search how to add this services into Kong automatically and how to set the loadBalancer if I have x occurences of one microservice.

Have you examples, sources, documentations, .. to do this ?

Thanks.

PS : Sorry for my english

[subnetmarco]

@throrin19 to use the DNS resolution provided by Consul (and leverage its load balancing), you need to specify the DNS resolver address in the Kong configuration, specifically you would need the following entry:

dns_resolver: server
dns_resolvers_available:
  server:
    address: "1.1.1.1:53"

Where 1.1.1.1:53 is the address to the Consul DNS resolver address and port. This will disable the default dnsmasq resolver.

At this point any upstream hostname, for example helloworld in upstream_url=http://helloworld/request, will be resolved by Consul.

[jimmytoan]

Can I use 2-3 DNS server? Will Kong load balancing between them? Can Kong detect failed dns server and won't query to that dns server?

[jimmytoan]

When I change configuration to:

dns_resolver: dnsmasq

dns_resolver: server
dns_resolvers_available:
server:
address: "192.168.9.31:8600"

dnsmasq:

port: 8053

I can't restart Kong:
sudo kong restart
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Leaving cluster..
[INFO] Kong 0.7.0
[INFO] Using configuration: /etc/kong/kong.yml
[ERR] Some services required by Kong are already running. Please execute "kong restart"!

[subnetmarco]

@jimmypk that doesn't look like a valid YAML format, try with proper indentation, like:

dns_resolver: server
dns_resolvers_available:
  server:
    address: "192.168.9.31:8600"

Kong will only accept one address. In order to load balancer the DNS queries across multiple servers, then you need to put a TCP load balancer in front of your DNS servers and point to the load balancer in the address YAML` property.

[jimmytoan]

I forgot the syntax when copy to here. The kong.yml is the same with your post. But it still doesn't run. I check log file at /usr/local/kong/logs/error.log

2016/03/29 09:53:03 [notice] 17679#0: signal process started 2016/03/29 09:53:03 [error] 17679#0: open() "/usr/local/kong/nginx.pid" failed (2: No such file or directory) 2016/03/29 09:53:44 [notice] 17760#0: signal process started 2016/03/29 09:53:44 [error] 17760#0: open() "/usr/local/kong/nginx.pid" failed (2: No such file or directory)

If I remove 4 lines of dns. Kong starts normally. I don't know why can't configure dns (that is consule dns server). Currently I use kong version 0.7.0

"you need to put a TCP load balancer in front of your DNS servers" => DNS is UDP protocol, we can't use TCP load balancer for DNS query.

[subnetmarco]

DNS is UDP protocol

Touchè, I wanted to say a UDP load balancer.

I check log file at /usr/local/kong/logs/error.log

Those errors should not affect the DNS resolutions. I will try locally with Consul and let you know the results of my test.

[jimmytoan]

Kong started ok with dns of consul server.
When I send HTTP request to test.domain.com (is pointed to Kong), Kong will query Consul DNS Server.
2016/03/29 10:25:30 [error] 2332#0: *10 test.service.consul could not be resolved (3: Host not found), client: 192.168.1.73, server: _, request: "POST /sessions HTTP/1.1", host: "test.domain.com"
What type of the dns query, it is normal query or SRV query?
I want to use Kong send services lookup (query SRV record) to Consul Server, is that possible with Kong?

[subnetmarco]

Kong will execute a normal DNS query, and it currently doesn't support SRV queries. Does Consul only support SRV queries or also normal queries?

[jimmytoan]

Consul only support SRV queries for service discovery.

[subnetmarco]

@jimmypk got it - then support for SRV should be introduced in 0.9.0 since we will adopt https://github.com/openresty/lua-resty-dns for DNS resolutions by then, and the lua-resty-dns module should support SRV.

[jimmytoan]

Thanks for your roadmap.
Currently I write plugin to do that with lua-resty-dns
Hope you think about this #1129 (comment)

[gomes]

@thefosk Hi, are u already working on this or is just planned?

[thibaultcha]

This is only planned, there are other things we must address before, including first releasing 0.8.

[gomes]

hey @thibaultcha I would like to help with this, maybe I can open an issue to discuss with you guys the better way to implement this, what do you think?

[thibaultcha]

The refactor/cli branch needs to be completed and merged first before tackling this, we can talk about it once that is done, but globally our plan is to switch to balancer_by_lua instead of the proxy_pass directive, and that would include having to resolve the upstream URLs ourselves, hence the need for lua-resty-dns, which, luckily, supports SRV records.

But the refactor/cli changes the way the nginx configuration is built and changes config variables, thus touching many parts of the codebase. Even basing our work on top of it would not be quite safe, since the refactor is not fully completed yet.

[gomes]

OK, I'll do some palliative method until we can discuss again. There is an issue tracking the refactor/cli change?

[thibaultcha]

It relates to many. For example:
#217
#656
#946
#974

[subnetmarco]

@jimmypk I was reading the Consul documentation, and it seems like it supports both A and SRV records for service discovery (as opposed to your previous comment which stated that it only supports SRV records):

For standard services queries, both A and SRV records are supported. SRV records provide the port that a service is registered on, enabling clients to avoid relying on well-known ports. SRV records are only served if the client specifically requests them.

https://www.consul.io/docs/agent/dns.html

[geovanisouza92]

I think that using the A records from consul and settings the ports mannually could be palliative, but considering my scenario, where all upstreams are docker containers, registered on consul with random ports (avoiding ports conflicts on docker), this still doesn't resolves the issue.

One thing that I'd used with vanilla nginx, is to generate nginx.conf from a template, rendering the upstreams IP and ports from information provided by consul, using consul-template and sending the HUP signal to reload nginx. Could this be the palliative action till 0.9.0 be released?

[hutchic]

I'm a little late to the discussion but was dropping in to point out that consul-template (as @geovanisouza92 noted) would work quite well for specifying nginx upstreams

[Kemicalish]

Any news about the release of Kong 0.9 and SRV change? 0.9.0 RC1 doesn't seems explicit about it.
SRV request would be great to avoid specifying static port in upstream url. BTW we are also switching from Monolithic app to Microservices architecture, and trio Kong / Consul / Nomad seems great but really need SRV records to handle dynamic ports config

[cmeury]

@Kemicalish It looks the answer to your question was given in the relevant pull request. Unfortunately, the SRV resolver won't make it into 0.9.

[thibaultcha]

Considering this question answered so far.

[MagicStarTrace]

Container internal port and external port is not the same, how to do it better

[hutchic]

@huyifanstar I'd suggest starting a new issue and explain the difficulty you're having.

[MagicStarTrace]

@throrin19

consul-server and kong-server:

kong Error

An unexpected error occurred.

2016-11-30 02:37:03 kongnginx_kong_1 migrating core for database kong
2016-11-30 02:37:03 kongnginx_kong_1 core migrated up to: 2015-01-12-175310_skeleton
2016-11-30 02:37:03 kongnginx_kong_1 core migrated up to: 2015-11-23-817313_nodes
2016-11-30 02:37:03 kongnginx_kong_1 core migrated up to: 2015-01-12-175310_init_schema
2016-11-30 02:37:03 kongnginx_kong_1 hmac-auth migrated up to: 2015-09-16-132400_init_hmacauth
2016-11-30 02:37:03 kongnginx_kong_1 migrating hmac-auth for database kong
2016-11-30 02:37:03 kongnginx_kong_1 core migrated up to: 2016-02-29-142793_ttls
2016-11-30 02:37:03 kongnginx_kong_1 migrating rate-limiting for database kong
2016-11-30 02:37:03 kongnginx_kong_1 rate-limiting migrated up to: 2015-08-03-132400_init_ratelimiting
2016-11-30 02:37:03 kongnginx_kong_1 rate-limiting migrated up to: 2016-07-25-471385_ratelimiting_policies
2016-11-30 02:37:03 kongnginx_kong_1 migrating acl for database kong
2016-11-30 02:37:03 kongnginx_kong_1 acl migrated up to: 2015-08-25-841841_init_acl
2016-11-30 02:37:03 kongnginx_kong_1 migrating ip-restriction for database kong
2016-11-30 02:37:03 kongnginx_kong_1 ip-restriction migrated up to: 2016-05-24-remove-cache
2016-11-30 02:37:03 kongnginx_kong_1 jwt migrated up to: 2015-06-09-jwt-auth
2016-11-30 02:37:03 kongnginx_kong_1 migrating jwt for database kong
2016-11-30 02:37:03 kongnginx_kong_1 migrating galileo for database kong
2016-11-30 02:37:03 kongnginx_kong_1 galileo migrated up to: 2016-04-15_galileo-import-mashape-analytics
2016-11-30 02:37:03 kongnginx_kong_1 jwt migrated up to: 2016-03-07-jwt-alg
2016-11-30 02:37:03 kongnginx_kong_1 migrating key-auth for database kong
2016-11-30 02:37:03 kongnginx_kong_1 migrating oauth2 for database kong
2016-11-30 02:37:03 kongnginx_kong_1 key-auth migrated up to: 2015-07-31-172400_init_keyauth
2016-11-30 02:37:03 kongnginx_kong_1 oauth2 migrated up to: 2016-07-15-oauth2_code_credential_id
2016-11-30 02:37:03 kongnginx_kong_1 oauth2 migrated up to: 2015-08-03-132400_init_oauth2
2016-11-30 02:37:03 kongnginx_kong_1 migrating response-ratelimiting for database kong
2016-11-30 02:37:03 kongnginx_kong_1 response-ratelimiting migrated up to: 2016-08-04-321512_response-rate-limiting_policies
2016-11-30 02:37:03 kongnginx_kong_1 response-ratelimiting migrated up to: 2015-08-03-132400_init_response_ratelimiting
2016-11-30 02:37:03 kongnginx_kong_1 migrating basic-auth for database kong
2016-11-30 02:37:03 kongnginx_kong_1 basic-auth migrated up to: 2015-08-03-132400_init_basicauth
2016-11-30 02:37:03 kongnginx_kong_1 2 migrations ran
2016-11-30 02:37:03 kongnginx_kong_1 Kong started

[MagicStarTrace]

@jimmypk consul-server and kong-server:

kong Error

An unexpected error occurred.

[siddo420]

with all the linked issues closed, is it safe to assume that Kong now supports querying SRV records as well?

this is important for multiple nodejs instances running on a single host

[subnetmarco]

It does in the next branch and upcoming 0.10 release.

[MagicStarTrace]

0.10.0 whether to include the keepalive function, it can inspect the health status of services ?

[magicliang]

Hi， @thefosk
I am new user to Kong, I am planning to use it and Consul to make up a service governance framework.
I have a same requirement as jimmypk asked:
"Can I use 2-3 DNS server? Will Kong load balancing between them? Can Kong detect failed dns server and won't query to that dns server?"
As you responded to him, Kong can not do this. Can you please introduce some details about how Kong is designed to communicate with a candidate DNS resolver? Does it have to query DNS resolver for every request? Why Kong can not support candidate list?
Sorry for my English.

[Tieske]

@magicliang from 0.10 onwards Kong does its own dns resolution. You can configure multiple dns servers, which will be queried randomly (load will be equally distributed). There are however no healthchecks for the dns servers to mark them down or bad.

Kong will not query on every request, it will cache results and only requery when the ttl expires.

[magicliang]

Thanks for the answers.
So I my understanding is: Now Kong can integrate with multiple Consuls?
Can you elaborate this? Do you have a example configuration format? Is something like this?

dns_resolver: server
dns_resolvers_available:
server:
address: "192.168.9.31:8600"

And I can understand there is no health check. But when a dns server is down, it will not respond to a SRV request, will Kong retry dns resolving in next dns server?

[Tieske]

see https://github.com/Mashape/kong/blob/master/kong.conf.default#L296

[magicliang]

I do some search about the SRV DNS record, my understanding is that every DNS server can configure a ttl for certain records, and it is the client that decides whether to purge the stale cache and get new records. I think that is what Kong will do, am I correct?

[link765]

if you change the dns_resolver, you must run the kong migrations!!!

[Tieske]

@huangnanyang no. you do not have to run migrations for changing the dns resolver. Migrations are only to update the structures in the DB when entities get new properties for example.

Setting another dns server in the config file will only require a reload.

@magicliang missed your question before, sorry about that. But for future reference: Yes, Kong will honor the TTL it received. Once it expires Kong will request the name again from the nameserver and use that new info going forward.

[qianliheng]

I use docker
consul container:
docker run -d -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 8600:53 -p 8600:53/udp -h dev28 --restart=always --name=consul progrium/consul -server -bootstrap -ui-dir=/ui -advertise 172.16.100.101 -client 0.0.0.0

registrator container:
docker run --restart=always --privileged --name=registrator --net=host -v /var/run/docker.sock:/tmp/docker.sock gliderlabs/registrator -ip 172.16.100.101 consul://172.16.100.101:8500

webservice container:
docker run -e "SERVICE_NAME=helloworld" -itd -p 8089:8080 -v /usr/local/ydwl:/usr/local/ydwl --name=helloworld jdk8

login: http://172.16.100.101:8500 I can see my webservice

dig @172.16.100.101 -p 8600 helloworld.service.consul
; <<>> DiG 9.10.3-P4-Ubuntu <<>> @172.16.100.101 -p 8600 helloworld.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47475
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;helloworld.service.consul. IN A

;; ANSWER SECTION:
helloworld.service.consul. 0 IN A 172.16.100.101

;; Query time: 0 msec
;; SERVER: 172.16.100.101#8600(172.16.100.101)
;; WHEN: Fri Jan 19 11:42:18 CST 2018
;; MSG SIZE rcvd: 84

kong container:
docker run -d --name kong
--link kong-database:kong-database
-e "KONG_DATABASE=cassandra"
-e "KONG_PG_HOST=kong-database"
-e "KONG_CASSANDRA_CONTACT_POINTS=kong-database"
-e "KONG_PROXY_ACCESS_LOG=/dev/stdout"
-e "KONG_ADMIN_ACCESS_LOG=/dev/stdout"
-e "KONG_PROXY_ERROR_LOG=/dev/stderr"
-e "KONG_ADMIN_ERROR_LOG=/dev/stderr"
-p 8000:8000
-p 8443:8443
-p 8001:8001
-p 8444:8444
kong:latest

I have add apis:
curl -i -X GET
--url http://localhost:8001/apis/
--data 'uris=/helloworld '
--data 'name=helloworld'
--data 'upstream_url=http://172.16.100.101:8089'

I also can visit : http://172.16.100.101:8000/helloworld
it return : "Hello World!"

but I want use consul DNS so I do this
etc/kong/kong.conf
dns_resolver: server
dns_resolvers_available:
server:
address: "172.16.100.101:8600"

then I add apis :
curl -i -X GET
--url http://localhost:8001/apis/
--data 'uris=/helloworld '
--data 'name=helloworld'
--data 'upstream_url=http://helloworld.service.sonsul:8089'

but I cann't visit http://172.16.100.101:8000/helloworld

kong logs write error:
*61645 [lua] responses.lua:107: after(): failed the initial dns/balancer resolve for 'helloworld.service.consul' with: dns lookup pool exceeded retries (1): timeout, client: 172.16.100.105, server: kong, request: "GET /helloworld HTTP/1.1", host: "172.16.100.101:8000"

sorry for my poor English!
Who can help me !!!

[qianliheng]

Kong:0.11.0