acceptor-conns_sup pairs #198

juhlig · 2018-07-17T14:49:03Z

one conns_sup for each acceptor

essen · 2019-02-25T09:11:50Z

Last night on IRC:

22:05:22          petrohi | essen: I was running test to see the difference between ranch 1.7 and juhlig2's PR. The workload is 100K "devices" each cycling through opening connection for 1 second and then closing it. 1.7 got  
                          | maxed out with ~200ms latency to open connection. juhlig2's maxed out at 130ms.                                                                                                                       
22:06:18            essen | sounds promising                                                                                                                                                                                      
22:06:22          petrohi | There is clear breaking point at about 60-70K devices when latency jumps from single digit millisecond to 100s in both cases.                                                                         
22:07:25          petrohi | Further, the Golang webserver maxes out at 110ms latency with 70-80K breakpoint.                                                                                                                      
22:07:59          petrohi | Which to me looks like I hit some bottleneck in the kernel.                                                                                                                                           
22:08:51            essen | keep the test around to try it with NIF sockets once that's implemented in OTP, it'll probably make Erlang closer to Go                                                                               
22:09:24          petrohi | When this is coming out?                                                                                                                                                                              
22:09:59            essen | OTP 22 or 23, so this or next year, probably next year                                                                                                                                                
22:10:19          petrohi | OK                                                                                                                                                                                                    
22:10:32            essen | still the proposed Ranch 2 changes look like a clear improvement so we can go with that                                                                                                               
22:10:42            essen | did you try establishing 100K connections and closing them all at once?                                                                                                                               
22:11:58          petrohi | Yes, tried that with 1M connections. In 1.7 case got supervisor message queue to 100K messages for few seconds.                                                                                       
22:12:02            essen | oh and considering the PR is fairly naive still there's probably some more improvements we can achieve, ideally there's only one process instead of an acceptor/sup pair                              
22:12:39          petrohi | With 32 acceptors on 32 CPU machine juhlig2's had no backup in message queues.                                                                                                                        
22:12:51            essen | impressive                                                                                                                                                                                            
22:14:00          petrohi | But, I was unable to reproduce case when I had backup for 10s of seconds. Even 1.7 was ready to accept within seconds.                                                                                
22:14:33            essen | probably takes longer if it happens on a real system with other busy processes around                                                                                                                 
22:15:01          petrohi | True.                                                                                                                                                                                                 
22:15:38          petrohi | Anyway, I think simple approach with acceptor pairing is big improvement.                                                                                                                             
22:16:42          petrohi | Any ideas on how to work around kernel bottleneck? Without it stressing the ranch would be much more intense.                                                                                         
22:16:49            essen | yeah, probably going to start from there, get the API right and then try to improve                                                                                                                   
22:17:33            essen | hm don't know specifically, just the usual sysctl flags that need tweaking                                                                                                                            
22:18:25          petrohi | Yes. Tried somaxconn, tcp_max_syn_backlog and netdev_max_backlog to no avail.                                                                                                                         
22:18:58          petrohi | Basically no difference from defaults. Just removed SYN flood warning in syslog.                                                                                                                      
22:34:56            essen | probably need some linux support :-) i'm off, thanks for the heads up!                                                                                                                                
-- Mon, 25 Feb 2019 --                                                                                                                                                                                                            
07:33:13          petrohi | After digging some more it turned out that breaking point in connection time from sub 1ms to 100s is caused by SYN buffer overflow. This causes SYNs to be dropped and clients transparently resending
                          | SYNs, which results in such a big hike in latency. SYN buffer overflows because accept is not picking up connections fast enough. I played with the number of acceptors from 1x number of CPUs to 16x 
                          | without much                                                                                                                                                                                          
07:33:15          petrohi | difference. So my conclusion at this point is that accept has natural contention that gets saturated at about 70K accepts per second. Juhlig's PR has a meaningful effect only between 65K and 70K    
                          | connections per second. With PR latency stays below 1ms. With mainline 1.7 it climbs to about 10ms. Above 70K  difference in latency is 130ms vs 200ms correspondingly. But at that point, the server 
                          | is dropping SYNs                                                                                                                                                                                      
07:33:17          petrohi | in millions.                                                                                                                                                                                          
09:30:55            essen | hm i see                                                                                                                                                                                              
09:45:23            essen | petrohi: by accept has natural contention do you mean Erlang's? i can't imagine the syscall having this problem and nobody fixing it since

So I think we will start work on master starting from this branch (after review of course) and then see what more improvements we can add. The important part is getting the API right as soon as possible, the internals can be changed afterwards, even after releasing 2.0, if for example we find a single process to work better than separate acceptor/sup.

/cc @petrohi

juhlig · 2019-02-25T14:00:29Z

So basically you're saying to brush off the dust this PR acquired since I put it up (ie, resolve the merge conflicts), so it can be properly reviewed, yes? :)

essen · 2019-02-25T14:11:41Z

I'm saying I will review it. There's no hurry though, I probably won't have time before next week anyway.

juhlig · 2019-02-25T16:23:14Z

Yeah, same here. In theory, I have plenty of time since I'm on vacation this week, but family has already claimed it all, so there =^^=

petermm · 2019-03-05T18:00:08Z

fyi https://stressgrid.com/blog/100k_cps_with_elixir/ - they use the PR and subsequently SO_REUSEPORT to push past the SYN bottleneck..

essen · 2019-03-05T18:46:22Z

Yep mentioned it in #110.

essen · 2019-04-25T09:10:30Z

I will begin work on getting this merged next week.

essen · 2019-04-29T11:34:06Z

src/ranch.erl

+	[_, _, _, Protocol, _] = ranch_server:get_listener_start_args(Ref),
+	Protocol;
+info(Ref, protocol_options) ->
+	get_protocol_options(Ref).


This change seems completely unrelated and not necessary (I didn't see any change other than active/all_connections retrieval method having changed). Correct? So probably better to keep it all as it was and just replace the two relevant calls.

Huh, now how did that end up in there... You're right, it is unrelated.

I had been playing with this quite a long time ago, then got distracted, then forgot. IIRC, my reason for this was so one could query a very specific info for a listener instead of all of it, which takes some extra time to collect only to be ignored,

Feel free to update the PR to master and remove this, I'll get to it on Wednesday myself.

essen · 2019-05-03T13:38:51Z

src/ranch_conns_sup.erl

+		{'DOWN', MonitorRef, process, SupPid, Reason} ->
+			error(Reason)
+	end.
+
 -spec start_protocol(pid(), inet:socket()) -> ok.
 start_protocol(SupPid, Socket) ->


This function does not look necessary anymore, every calls are using the start_protocol/3 function.

essen · 2019-05-03T13:40:03Z

src/ranch.erl

+	ListenerSup = ranch_server:get_listener_sup(Ref),
+	{_, ConnsSupSup, _, _} = lists:keyfind(ranch_conns_sup_sup, 1,
+		supervisor:which_children(ListenerSup)),
+	_ = [ConnsSup ! {remove_connection, Ref, self()} || {_, ConnsSup, _, _} <- supervisor:which_children(ConnsSupSup)],


Many lines are too long, for example this one. Please try to limit to around 100 columns. (Basically if it's not fully visible on the Github UI it's probably too long.)

essen · 2019-05-05T08:29:01Z

You forgot to add a copyright header to the new file, can you write in a comment the copyright line for you and i'll add it in my copy.

juhlig · 2019-05-06T02:35:06Z

Ok, sure. Do you want your name in there or mine (both are ok with me), and 2011-2019 as the year or just 2019?

juhlig · 2019-05-06T06:44:25Z

Jan Uhlig <ju@mailingwork.de>

essen · 2019-05-06T08:17:45Z

Merged, thanks!

essen mentioned this pull request Oct 15, 2018

Allow using more than one connection supervisor #110

Closed

essen reviewed Apr 29, 2019

View reviewed changes

acceptor-conns_sup pairs

001bcb4

essen requested changes May 3, 2019

View reviewed changes

fix long lines

3519089

essen closed this May 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

acceptor-conns_sup pairs #198

acceptor-conns_sup pairs #198

juhlig commented Jul 17, 2018

essen commented Feb 25, 2019

juhlig commented Feb 25, 2019

essen commented Feb 25, 2019

juhlig commented Feb 25, 2019

petermm commented Mar 5, 2019

essen commented Mar 5, 2019

essen commented Apr 25, 2019

essen Apr 29, 2019

juhlig Apr 29, 2019 •

edited

essen Apr 29, 2019

essen May 3, 2019

essen May 3, 2019

essen commented May 5, 2019

juhlig commented May 6, 2019

juhlig commented May 6, 2019

essen commented May 6, 2019

acceptor-conns_sup pairs #198

acceptor-conns_sup pairs #198

Conversation

juhlig commented Jul 17, 2018

essen commented Feb 25, 2019

juhlig commented Feb 25, 2019

essen commented Feb 25, 2019

juhlig commented Feb 25, 2019

petermm commented Mar 5, 2019

essen commented Mar 5, 2019

essen commented Apr 25, 2019

essen Apr 29, 2019

Choose a reason for hiding this comment

juhlig Apr 29, 2019 • edited

Choose a reason for hiding this comment

essen Apr 29, 2019

Choose a reason for hiding this comment

essen May 3, 2019

Choose a reason for hiding this comment

essen May 3, 2019

Choose a reason for hiding this comment

essen commented May 5, 2019

juhlig commented May 6, 2019

juhlig commented May 6, 2019

essen commented May 6, 2019

juhlig Apr 29, 2019 •

edited