http_client can't get html page from caddy server

[somanybut]

I compile project in win7, then I use command "http_client -H www.example.com -s 169.254.0.3:8443 -p /1.txt -L debug" to visit my web server on another machine(ubuntu 16.04), but no reponse can be found.
but I use chrome to visit my web server, It works well.

The client log:

-1:-1:-1.677 [INFO] conn-hash: initialized
-1:-1:-1.678 [INFO] engine: instantiated engine
-1:-1:-1.679 [INFO] socket buffer size: 8192 bytes; max # packets is set to 10
-1:-1:-1.680 [DEBUG] local address: 0.0.0.0:58527
-1:-1:-1.680 [DEBUG] engine: grew heaps to 4 elements
-1:-1:-1.682 [DEBUG] [QUIC:13289104569042990576] cfcw: recv_off changed: read_off: 0; recv_off: 15728640
-1:-1:-1.683 [DEBUG] [QUIC:13289104569042990576] cubic: lsquic_cubic_init_ext(cubic, 13289104569042990576, 0x1)
-1:-1:-1.685 [INFO] [QUIC:13289104569042990576] cubic: initialized
-1:-1:-1.686 [DEBUG] [QUIC:13289104569042990576] rechist: instantiated received packet history
-1:-1:-1.686 [DEBUG] [QUIC:13289104569042990576-3] sfcw: recv_off changed: read_off: 0; recv_off: 6291456
-1:-1:-1.687 [DEBUG] [QUIC:13289104569042990576-3] stream: created stream 3 @00000000003DEE80
-1:-1:-1.688 [DEBUG] [QUIC:13289104569042990576-3] stream: calling on_new_stream
-1:-1:-1.689 [DEBUG] [QUIC:13289104569042990576] headers: stream created
-1:-1:-1.690 [DEBUG] [QUIC:13289104569042990576] frame-reader: reset state
-1:-1:-1.691 [DEBUG] [QUIC:13289104569042990576] conn: negotiating version Q039
-1:-1:-1.691 [DEBUG] [QUIC:13289104569042990576-1] sfcw: recv_off changed: read_off: 0; recv_off: 6291456
-1:-1:-1.692 [DEBUG] [QUIC:13289104569042990576-1] stream: created stream 1 @00000000003DEF90
-1:-1:-1.693 [DEBUG] [QUIC:13289104569042990576-1] stream: calling on_new_stream
-1:-1:-1.693 [DEBUG] [QUIC:13289104569042990576] hsk-adapter: stream created
-1:-1:-1.694 [DEBUG] [QUIC:13289104569042990576] sendctl: lsquic_send_ctl_can_send: n_out: 0 (unacked_retx: 0, out: 0); cwnd: 46720
-1:-1:-1.695 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.696 [DEBUG] engine: incref conn 13289104569042990576, '' -> 'T'
-1:-1:-1.697 [INFO] [QUIC:13289104569042990576] conn: Created new client connection
-1:-1:-1.699 [DEBUG] [QUIC:13289104569042990576] event: created full connection
-1:-1:-1.700 [DEBUG] engine: incref conn 13289104569042990576, 'H' -> 'HT'
-1:-1:-1.700 [DEBUG] [QUIC:13289104569042990576-5] stream: set priority to 16
-1:-1:-1.701 [DEBUG] [QUIC:13289104569042990576-5] sfcw: recv_off changed: read_off: 0; recv_off: 6291456
-1:-1:-1.702 [DEBUG] [QUIC:13289104569042990576-5] stream: created stream 5 @00000000003DF0A0
-1:-1:-1.703 [DEBUG] [QUIC:13289104569042990576-5] stream: calling on_new_stream
-1:-1:-1.703 [INFO] created new stream, path: /1.txt
-1:-1:-1.704 [DEBUG] engine: decref conn 13289104569042990576, 'HT' -> 'H'
-1:-1:-1.704 [DEBUG] [QUIC:13289104569042990576] conn: memory used: 2807 bytes
-1:-1:-1.705 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.706 [DEBUG] [QUIC:13289104569042990576] sendctl: lsquic_send_ctl_can_send: n_out: 0 (unacked_retx: 0, out: 0); cwnd: 46720
-1:-1:-1.707 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.707 [DEBUG] [QUIC:13289104569042990576] sendctl: lsquic_send_ctl_can_send: n_out: 0 (unacked_retx: 0, out: 0); cwnd: 46720
-1:-1:-1.708 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.709 [DEBUG] handshake: lsquic_enc_session_gen_chlo called, return 0, buf_len 1024.
-1:-1:-1.710 [DEBUG] [QUIC:13289104569042990576-1] stream: buffered 1024 bytes; 1024 bytes are now in buffer
-1:-1:-1.711 [INFO] [QUIC:13289104569042990576] hsk-adapter: wrote 1024 bytes of CHLO to stream
-1:-1:-1.711 [DEBUG] [QUIC:13289104569042990576-1] stream: will flush up to offset 1024
-1:-1:-1.712 [DEBUG] [QUIC:13289104569042990576] sendctl: created packet 1
-1:-1:-1.713 [DEBUG] [QUIC:13289104569042990576] event: created packet 1; flags: version=1, nonce=0, conn_id=1
-1:-1:-1.714 [DEBUG] [QUIC:13289104569042990576] event: generated STREAM frame: stream 1, offset: 0, size: 1024, fin: 0
-1:-1:-1.715 [DEBUG] [QUIC:13289104569042990576-1] stream: flushed to or past required offset 1024
-1:-1:-1.716 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_packet_scheduled: replenish tokens: 10

-1:-1:-1.717 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_packet_scheduled: tokens: 9
-1:-1:-1.718 [DEBUG] [QUIC:13289104569042990576] sendctl: lsquic_send_ctl_can_send: n_out: 1370 (unacked_retx: 0, out: 0); cwnd: 46720
-1:-1:-1.719 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.719 [DEBUG] engine: incref conn 13289104569042990576, 'H' -> 'HO'
-1:-1:-1.720 [DEBUG] engine: decref conn 13289104569042990576, 'HOT' -> 'HO'
-1:-1:-1.721 [DEBUG] [QUIC:13289104569042990576] pacer: intertick estimate: 0; real value: 0; error: 0
-1:-1:-1.722 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.722 [DEBUG] handshake: lsquic_enc_session_encrypt: hsk_state: 0
-1:-1:-1.723 [DEBUG] engine: encrypted packet 1; plaintext is 1358 bytes, ciphertext is 1370 bytes
-1:-1:-1.724 [DEBUG] engine: batched packet 1 for connection 13289104569042990576
-1:-1:-1.724 [DEBUG] engine: batched all outgoing packets for conn 13289104569042990576
-1:-1:-1.725 [DEBUG] engine: packets out returned 1 (out of 1)
-1:-1:-1.725 [DEBUG] [QUIC:13289104569042990576] event: sent packet 1, size 1370, frame types: STREAM PADDING
-1:-1:-1.727 [DEBUG] [QUIC:13289104569042990576] sendctl: packet 1 has been sent (frame types: STREAM PADDING)
-1:-1:-1.729 [DEBUG] [QUIC:13289104569042990576] sendctl: set retx alarm to 95063556005, which is 150000 usec from now, mode RETX_MODE_HANDSHAKE
-1:-1:-1.730 [DEBUG] engine: decref conn 13289104569042990576, 'HO' -> 'H'
-1:-1:-1.731 [DEBUG] engine: send_packets_out: sent 1 packet
-1:-1:-1.732 [DEBUG] [QUIC:13289104569042990576] sendctl: lsquic_send_ctl_can_send: n_out: 1370 (unacked_retx: 1370, out: 0); cwnd: 46720
-1:-1:-1.733 [DEBUG] [QUIC:13289104569042990576] pacer: pacer_can_schedule: 1
-1:-1:-1.734 [DEBUG] engine: incref conn 13289104569042990576, 'H' -> 'HT'
-1:-1:-1.735 [DEBUG] engine: incref conn 13289104569042990576, 'HT' -> 'HTA'

[Kait0]

As I said in another closed thread before I experience the same issue of not getting a response using windows 10 in both ipv4 and ipv6.

[dtikhonov]

@somanybut, thank you for the report. The log messages you include only describe the beginning of the session -- I don't know whether you get any packets back. Do you get any response packets?

Also, is your bug report Caddy-specific? That is, can you fetch an index from a different web server (e.g. www.google.com or www.litespeedtech.com?)

(The log message time display has been fixed in 560db45.)

[somanybut]

I try to visit www.litespeedtech.com, chrom is ok, but http_client can't get reponse from server, the same as to visit my Caddy server.

...
-1:-1:-1.522 [DEBUG] [QUIC:1113064085529127296] event: generated STREAM frame: stream 1, offset: 0,size: 1024, fin: 0
-1:-1:-1.527 [DEBUG] [QUIC:1113064085529127296-1] stream: flushed to or past required offset 1024
-1:-1:-1.529 [DEBUG] [QUIC:1113064085529127296] pacer: pacer_packet_scheduled: replenish tokens: 10
-1:-1:-1.532 [DEBUG] [QUIC:1113064085529127296] pacer: pacer_packet_scheduled: tokens: 9
-1:-1:-1.537 [DEBUG] [QUIC:1113064085529127296] sendctl: lsquic_send_ctl_can_send: n_out: 1370 (unacked_retx: 0, out: 0); cwnd: 46720
-1:-1:-1.542 [DEBUG] [QUIC:1113064085529127296] pacer: pacer_can_schedule: 1
-1:-1:-1.544 [DEBUG] engine: incref conn 1113064085529127296, 'H' -> 'HO'
-1:-1:-1.548 [DEBUG] engine: decref conn 1113064085529127296, 'HOT' -> 'HO'
-1:-1:-1.552 [DEBUG] [QUIC:1113064085529127296] pacer: intertick estimate: 0; real value: 0; error:0
-1:-1:-1.557 [DEBUG] [QUIC:1113064085529127296] pacer: pacer_can_schedule: 1
-1:-1:-1.560 [DEBUG] handshake: lsquic_enc_session_encrypt: hsk_state: 0
-1:-1:-1.562 [DEBUG] engine: encrypted packet 1; plaintext is 1358 bytes, ciphertext is 1370 bytes
-1:-1:-1.564 [DEBUG] engine: batched packet 1 for connection 1113064085529127296
-1:-1:-1.570 [DEBUG] engine: batched all outgoing packets for conn 1113064085529127296
-1:-1:-1.572 [DEBUG] engine: packets out returned 1 (out of 1)
-1:-1:-1.574 [DEBUG] [QUIC:1113064085529127296] event: sent packet 1, size 1370, frame types: STREAM PADDING
-1:-1:-1.578 [DEBUG] [QUIC:1113064085529127296] sendctl: packet 1 has been sent (frame types: STREAM PADDING)
-1:-1:-1.586 [DEBUG] [QUIC:1113064085529127296] sendctl: set retx alarm to 27226771314, which is 150000 usec from now, mode RETX_MODE_HANDSHAKE
-1:-1:-1.590 [DEBUG] engine: decref conn 1113064085529127296, 'HO' -> 'H'
-1:-1:-1.593 [DEBUG] engine: send_packets_out: sent 1 packet
-1:-1:-1.595 [DEBUG] [QUIC:1113064085529127296] sendctl: lsquic_send_ctl_can_send: n_out: 1370 (unacked_retx: 1370, out: 0); cwnd: 46720
-1:-1:-1.603 [DEBUG] [QUIC:1113064085529127296] pacer: pacer_can_schedule: 1
-1:-1:-1.605 [DEBUG] engine: incref conn 1113064085529127296, 'H' -> 'HT'
-1:-1:-1.608 [DEBUG] engine: incref conn 1113064085529127296, 'HT' -> 'HTA'

[somanybut]

command: http_client -H www.litespeedtech.com -p / -L debug

[dtikhonov]

@somanybut, you need to use -s option to specify IP address and port number.

[dtikhonov]

I think the way to figure this out is to see whether the QUIC packets make it from the client to the server. On your Windows machine (client), you can use Wireshark to see whether the packets are sent; on the Ubuntu machine (server), use Wireshark or tcpdump to see whether the QUIC packets are received.

[Kait0]

So let me give you some data on this:
I switch to a Ubuntu Dual boot. I also don't get a response from the google servers. I tested it with both my local lan connection and my mobile phone connection (via Wifi hotspot) so connection shouldn't be the problem.
I also tried connecting to www.litespeedtech.com where I indeed got an answer from the server although most of it was still encrypted in the console:

(Top 3 commands)

In and outcoming packets with wireshark for google ipv4:

Google ipv6:

and litespeed tech ipv4

Also I wanted to ask if Port 443 really is the correct one since google quic is not using tls right now.
Wiki: "443: HTTP Secure (HTTPS) HTTP over TLS/SSL"

[dtikhonov]

The response from www.litespeedtech.com is not encrypted: it is compressed (see the content-encoding: gzip header). The fact that you get a QUIC reply shows that the library works.

In the second screenshot, one can see that no QUIC responses come back from 74.125.22.106. I just tested with this IP address and it works for me -- I get a response. Try using the IP address of www.google.com where you are -- it may be different. For example, run nslookup www.google.com.

[Kait0]

Indeed the problem was the different ip adress in my region:

Thanks for your help!
You might want to add this as a small note in the examples.txt

[dtikhonov]

Excellent.

You might want to add this as a small note in the examples.txt

Feel like making a pull request? 😃

[Kait0]

Sure. I tried to make one. Let me know if it worked.

[litespeedtech]

Closing: no further comment by the original reporter on the reported issue for over a month.