Kong performance with go-plugin

[Sachin-M-Desai]

We are performing performance tests with kong to see what type of plugin works best for us.

Our base performance with kong 3.2.2 (no plugins with CPU isolated local application server)

ubuntu@ip-10-101-1-31:~$ wrk -t 1 -d 180s -c 2 --latency -s post-sess-large.lua http://http-service:9500/setprofile
Running 3m test @ http://http-service:9500/setprofile
  1 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.21ms  763.40us  20.63ms   92.17%
    Req/Sec     1.75k   203.30     2.21k    73.67%
  Latency Distribution
     50%    1.02ms
     75%    1.30ms
     90%    1.79ms
     99%    4.35ms
  313088 requests in 3.00m, 72.26MB read
Requests/sec:   1738.47
Transfer/sec:    410.84KB

We wrote a simple hello-world type of go-plugin with Access/Response and Log handlers. These handlers are just empty returns. But we observe that kong's performance comes down quite a bit.

ubuntu:~$ wrk -t 1 -d 30s -c 2 --latency -s post-sess-large.lua http://http-service:9500/setprofile
Running 30s test @ http://http-service:9500/setprofile
  1 threads and 2 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.55ms  518.44us  20.21ms   87.70%
    Req/Sec     1.30k   133.78     1.49k    83.00%
  Latency Distribution
     50%    1.46ms
     75%    1.71ms
     90%    2.02ms
     99%    3.06ms
  38971 requests in 30.02s, 8.99MB read
Requests/sec:   1298.37
Transfer/sec:    306.83KB

This implies that writing/reading from the local socket itself brings down the performance by 25%. The CPU remains around 200% of the 4 CPUs given to kong docker.

1. What are the other alternatives for a better kong performance with plugins?
2. Do we have any benchmarks we can refer to for achieving performance with plugins?

Regards,
Sachin

[esatterwhite]

As I understand it, if performance matters and your plugin is a hot path, lua is really the only option.

The other language incur an RPC overhead for many operations that operate on the request or response

[esatterwhite]

or build it in C or rust and expose it to lua

[Sachin-M-Desai]

Thanks. The trouble for us is that neither lua nor RPC mechanism provides a satisfactory option when it comes to performance.

[Sachin-M-Desai]

Not sure if this is as per kong's design direction, but is loading openresty plugins another way to go about? We have much much better performance in nginx modules than in abstracted kong infra.