Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

accepting quic stream: Application error 0x376 (remote): reconnect #6425

Closed
4 tasks done
04bondgoods opened this issue Nov 15, 2023 · 8 comments
Closed
4 tasks done
Labels
waiting for data Waiting for users to provide more data.

Comments

@04bondgoods
Copy link

Prerequisites

Platform (OS and CPU architecture)

Linux, AMD64 (aka x86_64)

Installation

GitHub releases or script from README

Setup

On one machine

AdGuard Home version

v0.107.41

Action

tail /$agh_logs/stderr.log

Expected result

normal output

Actual result

2023/11/15 14:48:10.252986 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 14:48:11.037386 [error] accepting quic stream: APPLICATION_ERROR (remote)
2023/11/15 14:48:11.123583 [error] accepting quic stream: Application error 0x1 (remote)
2023/11/15 14:48:11.259371 [error] accepting quic stream: Application error 0x1 (remote)
2023/11/15 14:48:21.806773 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 14:48:22.204339 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 14:48:52.267501 [info] Received signal "terminated"
2023/11/15 14:48:52.267538 [info] stopping AdGuard Home
2023/11/15 14:48:52.267543 [info] stopping http server...
2023/11/15 14:48:52.267655 [info] stopped http server
2023/11/15 14:48:52.267686 [info] dnsproxy: stopping dns proxy server
2023/11/15 14:48:52.268049 [info] udpListen.ReadFrom() returned because we're reading from a closed connection, exiting loop
2023/11/15 14:48:52.268080 [info] udpListen.ReadFrom() returned because we're reading from a closed connection, exiting loop
2023/11/15 14:49:22.490093 [info] AdGuard Home, version v0.107.41
2023/11/15 14:49:22.493384 [info] tls: using default ciphers
2023/11/15 14:49:22.495800 [info] safesearch default: disabled

Additional information and/or screenshots

as the log indicates, the agh core keeps terminating and restarting for every ~2 minutes.
during its termination 'lsof -i :53' shows that port 53 is not bound.
wondering what Application error 0x376 and 0x1 is, and what it has to do with accepting quic.
(BTW my clients do use quic:// and h3:// when querying agh)

@ainar-g
Copy link
Contributor

ainar-g commented Nov 15, 2023

This could be the same issue as #6422. Can you please try adding QUIC_GO_DISABLE_ECN=true to AdGuard Home's environment and see if it improves things?

Also, quic-go has been updated to a newer version in the Edge channel, so can you also check if the issue persist there without the env. variable?

@ainar-g ainar-g added the waiting for data Waiting for users to provide more data. label Nov 15, 2023
@04bondgoods
Copy link
Author

04bondgoods commented Nov 15, 2023

tested with the env var, seems that part of the problem persists.
but good thing is that it doesn't crash and restart anymore.
will test the edge (beta?) channel later

2023/11/15 19:17:33.292152 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:17:41.558701 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:18:02.272095 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:18:02.272540 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:18:02.272545 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:01.740319 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:01.803212 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:04.114388 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:17.145780 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:17.214380 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:25.518489 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:45.477965 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:47.343171 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:55.438559 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:58.048493 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:20:58.768197 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:05.953531 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:08.976222 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:11.674552 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:14.254595 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:17.126394 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:17.728155 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:17.797813 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:20.105444 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:23.547193 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:24.410051 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:32.747039 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:54.273863 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:56.587322 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:58.298068 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:58.298071 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:21:58.298082 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:22:27.763651 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:22:28.405391 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:22:29.070099 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:22:43.781378 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:23:11.863086 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:05.356290 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:05.997675 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:08.335585 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:25.838274 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:27.652473 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:32.520683 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:39.966019 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:39.966080 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:24:40.039730 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:25:30.548685 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:25:32.800599 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:25:32.863370 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 19:25:36.892519 [error] accepting quic stream: Application error 0x376 (remote): reconnect

@04bondgoods
Copy link
Author

tested in version v0.108.0-b.49, problem still there

2023/11/15 21:02:28.175873 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:28.245735 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:32.572795 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:37.484181 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:37.608833 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:37.680533 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:40.889815 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:02:57.449622 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:06.946153 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:26.331587 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:26.921451 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:26.993737 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:26.993778 [error] reading from quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:29.277874 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 21:03:41.765363 [error] accepting quic stream: Application error 0x376 (remote): reconnect

@ainar-g
Copy link
Contributor

ainar-g commented Nov 15, 2023

I'm not sure what caused AGH to terminate in the first place, since such application errors shouldn't cause it to panic.

We also cannot reproduce this on any of our machines, and there doesn't seem to be any mention of error code 0x376 in quic-go documentation, so this is likely some kind of remote error code that the upstream sends to terminate a connection.

Some additional pieces of data that should clarify things:

  1. Which Linux distribution are you using? What is the kernel there (e.g. uname -a)?

  2. Which QUIC and HTTP/3 upstreams are you using? Does the error still occur if you change them to some others?

@04bondgoods
Copy link
Author

uname -a
ubuntu 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

the quic/h3 upstreams used:
adguard unfiltered (quic), google (h3), cloudflare (h3)

removed these upstreams, and set serve-http3 to false, still:

2023/11/15 22:15:18.281123 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 22:15:50.165652 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 22:15:50.241464 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 22:15:51.087397 [error] accepting quic stream: Application error 0x376 (remote): reconnect
2023/11/15 22:16:21.824637 [error] accepting quic stream: Application error 0x376 (remote): reconnect

hope these info can help.
BTW I thought this is because of receiving quic stream, instead of sending.

@ainar-g
Copy link
Contributor

ainar-g commented Nov 15, 2023

Right, I was thinking that maybe it's related to another quic-go issue, but it probably isn't.

Are there many clients using DoQ? Have you noticed any correlations between a particular client and the occurrence of these errors? A misbehaving DNS client on one or a few of them perhaps?

@04bondgoods
Copy link
Author

around 2-3 clients using doq, not many queries per second. I did try making queries to see correlations, but didn't observe any. so I guess .. just wait for possible other cases then

@ainar-g ainar-g changed the title AGH service keeps crashing and restarting accepting quic stream: Application error 0x376 (remote): reconnect Nov 20, 2023
@04bondgoods
Copy link
Author

didn't pay much attention to DNSs for a while, and somehow this issue disappeared after several updates. so i closed it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting for data Waiting for users to provide more data.
Projects
None yet
Development

No branches or pull requests

2 participants