Socket.io - sticky sessions with nginx ingress controller for k8s [solved]

[DominicWatson]

Hi there, we have a java based application that implements a Socket.io server (i.e. not a nodejs server). Our Kubernetes cluster setup uses sticky sessions for our multi-replica deployment because as of now, our Engine.io/Socket.io server implementation does not support clustering for establishing a connection.

It appears, however, that Artillery is not storing and resending cookies during the Socket handshake. This means that requests are going to different servers which leads to failed handshakes. (hypothesis).

Is this correct and is it something that you might consider changing?

TIA

Updated with solution below

The below concept is a workaround for cookie issues using socket.io client with Artillery and using nginx ingress controller loadbalancing in k8s:

config:
  variables:
    route:
      - a
      - b
      - c
      - d
      - e
      - f
      - g
      - h
      - i
      - j
      # ... etc., enough values to spread across your replicas
scenarios:
  - name: "A SocketIo scenario"
    engine: "socketio"
    socketio:
      extraHeaders:
        Cookie: "route={{ route }}"
    flow:
      # ...

[hassy]

hi @DominicWatson 👋 Artillery uses the official Socket.io client under the hood, which should be taking care of storing & resending cookies when HTTP long polling is used. What does the output from Artillery look like when you run a test?

[DominicWatson]

Thanks @hassy - I'll dig a little deeper in that case. I just noticed that I received a large portion of 400 responses from my server as soon as I had multiple replicas and this led to the hypothesis (the server not recognising the request). Will update here if/when I get to the bottom of that.

[DominicWatson]

Testing a little further is still making me believe that the cookies are not being persisted between calls. I scaled my deployment to two replicas and tailed logs on both. When I ran my scenario with 1 virtual user, I see traffic to both k8s pods. This does not happen when using a browser where a single user correctly gets "stuck" on a single node/pod. Here is a stripped down table showing logs from the pods; you can see the requests spreading across the pods:

time	pod	method	url	resp.
09:48:09	1	GET	"/socket.io/?EIO=3&transport=polling&t=Ndbg6Mn&b64=1"	200
09:48:10	2	GET	"/socket.io/?EIO=3&transport=polling&t=Ndbg6k-&b64=1&sid=Ndbg6P1"	400
09:48:10	2	GET	"/socket.io/?EIO=3&transport=websocket&sid=Ndbg6P1"	101
09:48:10	1	POST	"/socket.io/?EIO=3&transport=polling&t=Ndbg6oJ&b64=1&sid=Ndbg6P1"	200
09:48:11	1	GET	"/socket.io/?EIO=3&transport=polling&t=Ndbg70K&b64=1"	200
09:48:11	2	GET	"/socket.io/?EIO=3&transport=polling&t=Ndbg71j&b64=1&sid=Ndbg6P2"	400
09:48:12	2	POST	"/socket.io/?EIO=3&transport=polling&t=Ndbg72-&b64=1&sid=Ndbg6P2"	400

I'm using the core Nginx ingress controller and using its cookie based sticky sessions. When I curl the first endpoint the socket.io connection uses for connection I get the following response (this is just showing that the route cookie is being set):

$> curl -v "https://{redacted}/socket.io/?EIO=3&transport=polling&t=Ndbg6Mn&b64=1"
...
< content-type: text/plain;charset=utf-8
< set-cookie: route=1623056917.458.13662.596377; Path=/; Secure; HttpOnly
...
83:0{"pingInterval":5000,"pingTimeout":25000,"upgrades":["websocket"],"sid":"Ndbg85E"}

Here is my yaml:

config:
  target: "https://mytargetdomain.test" # redacted
  phases:
    - duration: 1
      arrivalRate: 1
scenarios:
  - name: "Super simple socketio scenario"
    engine: "socketio"
    flow:
      - loop:
        - emit:
            channel: "msg"
            data: "hello world! {{ $randomString() }}"
            namespace: "/simpleloadtest"
        - think: 10
        count: 2

Output of the test isn't particularly helpful re debugging this as its just the stats. Is there anyway I can trace the detailed calls (I've searched through the docs and can't see anything related to debug tracing)?

Thanks again for your time.

[DominicWatson]

More info on this. Adding the following in the test yaml, allowed the connections to work on a single pod without any errors:

config:
  socketio:
    extraHeaders:
      Cookie: "route=1623081686.647.13662.138379"

However, I had to manually find out a valid route and use that rather than it being dynamic. Could be related to #1035 or #232 ?

[DominicWatson]

Ok, so the issue seems to be this one: socketio/socket.io-client#1159. Something to do with node not allowing the cookies here due to cross origin things.

The workaround suggested is to set extraHeaders.Cookie as we have above, but this of course needs to be dynamic per user session in Artillery. I tried something like below but failed to get it to work, might this be doable at all do you think just with a yaml config?

config:
  socketio:
    extraHeaders:
      Cookie: "{{ ck }}"
scenarios:
  - name: "Brief Test"
    weight: 1
    engine: "socketio"
    flow:
      - get:
          url: /
          capture:
            header: "set-cookie"
            as: "ck"
      - loop:
        - emit:
            channel: "msg"
            data: "hello brief world! {{ $randomString() }}"
            namespace: "/simpleloadtest"
        - think: 10
        count: 2

[DominicWatson]

Ok, so I have a solution that mods the core socketio engine in artillery by:

• enabling beforeScenario hooks, etc.
• delays socket creation until needed (not on compile)

Then I use a beforeScenario function to fetch cookies and set into a variable that is used in socketIo.extraHeaders.Cookie set in the scenario itself:

config:
  processor: "./functions.js"
  # ...
scenarios:
  - name: "my scenario"
    beforeScenario: "fetchCookies" # sets the env.socketCookies variable after getting cookies by visiting a page on the site
    socketio:
      extraHeaders:
        Cookie: "{{ socketCookies }}"
   # ...

Will make a fork and pull request with changes to the engine if you're interested @hassy

Edit

Here's the functions.js with setupCookies custom function:

var request = require('sync-request');
var Cookie = require('request-cookies').Cookie;

module.exports = {
    setupCookies: setupCookies
}

function setupCookies(context, ee, next) {
	var resp       = request( "GET", context.vars.target )
	  , rawcookies = resp.headers['set-cookie']
	  , formatted  = []
	  , cookies, i;

	for ( i in rawcookies ) {
		var cookie = new Cookie( rawcookies[i] );
		formatted.push( cookie.key + "=" + cookie.value );
	}
	
	context.vars.socketcookies = formatted.join( "; " );

	return next();
}

[hassy]

Wow, that's an amazingly thorough investigation, thanks so much for looking into it @DominicWatson! Yes, definitely interested in a PR fixing the issue! 🙏

[DominicWatson]

I need to fix the tests, but could you take a look in principal at these @hassy (v simple tweaks, but I presume there could be v good reasons for the things I'm changing)?

#1067
#1068

[maitrungduc1410]

this is so amazing @DominicWatson , thanks for your in-depth investigation on this

[maitrungduc1410]

Hi @DominicWatson can you let me know which version of socket.io you're using?

[DominicWatson]

Client is v2 - server is custom java server based on v2 protocol also.

[DominicWatson]

Ok, final thing on this.

1. It seems those tests are broken in Master, so leaving those PRs there.
2. While I believe those PRs may still be valid, I've found a workaround for my particular case as seen below:

config:
  variables:
    route:
      - a
      - b
      - c
      - d
      - e
      - f
      - g
      - h
      - i
      - j
scenarios:
  - name: "A SocketIo scenario"
    engine: "socketio"
    socketio:
      extraHeaders:
        Cookie: "route={{ route }}"
    flow:
      # ...

The workaround above works with my nginx ingress controllers load balancer implementation. It seems you can throw in your own route cookie value and it will always send it to the same pod. So seeding enough different route variables works!