Pipelining behaviour question

[guilherme]

Hello,

I am trying to understand how does the pipelining behavior works. My expectation was that it would create one connection for every 5 requests (due to https://curl.haxx.se/libcurl/c/CURLMOPT_MAX_PIPELINE_LENGTH.html) when using this option.
But then I run a tcp dump to check what was going on and it was one connection for every request.

How is the pipelining behavior supposed to work ?

def request(pipelining = true)
  hydra = Typhoeus::Hydra.new(pipelining: pipelining)
  10.times do
    url = 'http://google.com'
    request = Typhoeus::Request.new(url, params: { q: "test" })
    hydra.queue(request)
  end
  hydra.run
end
request(true)

tcpdump

root@e09bf0b135b8:/app# tcpdump -i eth0 -w tcpdump-google.out -s 1520 port 80
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1520 bytes
^C70 packets captured
70 packets received by filter
0 packets dropped by kernel

root@e09bf0b135b8:/app# tcptrace  tcpdump-google.out 
1 arg remaining, starting with 'tcpdump-google.out'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov  4, 2004

70 packets seen, 70 TCP packets traced
elapsed wallclock time: 0:00:00.005269, 13285 pkts/sec analyzed
trace file elapsed time: 0:00:00.203851
TCP connection info:
  1: e09bf0b135b8:54578 - sin01s16-in-f14.1e100.net:80 (a2b)    4>    3<
  2: e09bf0b135b8:54580 - sin01s16-in-f14.1e100.net:80 (c2d)    4>    3<
  3: e09bf0b135b8:54582 - sin01s16-in-f14.1e100.net:80 (e2f)    4>    3<
  4: e09bf0b135b8:54584 - sin01s16-in-f14.1e100.net:80 (g2h)    4>    3<
  5: e09bf0b135b8:54586 - sin01s16-in-f14.1e100.net:80 (i2j)    4>    3<
  6: e09bf0b135b8:54588 - sin01s16-in-f14.1e100.net:80 (k2l)    4>    3<
  7: e09bf0b135b8:54590 - sin01s16-in-f14.1e100.net:80 (m2n)    4>    3<
  8: e09bf0b135b8:54592 - sin01s16-in-f14.1e100.net:80 (o2p)    4>    3<
  9: e09bf0b135b8:54594 - sin01s16-in-f14.1e100.net:80 (q2r)    4>    3<
 10: e09bf0b135b8:54596 - sin01s16-in-f14.1e100.net:80 (s2t)    4>    3<

When I use the CLI, it works:

root@e09bf0b135b8:/app# curl http://www.google.com http://www.google.com

root@e09bf0b135b8:/app# tcpdump -i eth0 -w tcpdump-google-curl.out -s 1520 port 80
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 1520 bytes 
^C14 packets captured
14 packets received by filter
0 packets dropped by kernel

root@e09bf0b135b8:/app# tcptrace tcpdump-google-curl.out 
1 arg remaining, starting with 'tcpdump-google-curl.out'
Ostermann's tcptrace -- version 6.6.7 -- Thu Nov  4, 2004

14 packets seen, 14 TCP packets traced
elapsed wallclock time: 0:00:00.004786, 2925 pkts/sec analyzed
trace file elapsed time: 0:00:00.099330
TCP connection info:
  1: e09bf0b135b8:34002 - syd15s02-in-f36.1e100.net:80 (a2b)    7>    7<  (complete)

[PericlesTheo]

I've never used that option before but I think you are passing the wrong key to Hydra. Ethon::Curls::Options contains all possible options and it's usually the curl options but lowercased but without the CURLMOPT I think.

So in that case you might need to pass https://github.com/typhoeus/ethon/blob/4c820e2116c7dba030e84d576a0592afd430eba1/lib/ethon/curls/options.rb#L219?

[PericlesTheo]

OK just saw the pipelining option https://github.com/typhoeus/ethon/blob/4c820e2116c7dba030e84d576a0592afd430eba1/lib/ethon/curls/options.rb#L214 so ignore what I said above

[drpump]

Be aware that pipelining is unreliable unless CURLOPT_PIPEWAIT is set to true (sometimes it works, sometimes you get multiple connections). See the Multiplexing section of https://curl.haxx.se/docs/http2.html. As far as I can tell, Ethon does not support this yet.