Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: TestServerReadAfterWriteHeader100Continue/h1 failures #67382

Closed
gopherbot opened this issue May 15, 2024 · 24 comments
Closed

net/http: TestServerReadAfterWriteHeader100Continue/h1 failures #67382

gopherbot opened this issue May 15, 2024 · 24 comments
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@gopherbot
Copy link
Contributor

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"

Issue created automatically to collect these failures.

Example (log):

=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 15, 2024
@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-15 13:27 gotip-linux-arm go@0222a028 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 13:52 gotip-darwin-arm64_11 go@6ccd8e4c net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 15:02 gotip-linux-loong64 go@c3c97ad1 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 20:03 gotip-linux-arm64-boringcrypto go@3d807615 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-15 21:33 gotip-linux-arm go@3c8f9256 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-16 01:32 gotip-linux-amd64-clang15 go@1ffc2967 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@dmitshur
Copy link
Contributor

CC @neild.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-15 14:39 gotip-linux-ppc64le_power10 go@80565407 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 15:02 gotip-linux-ppc64_power10 go@c3c97ad1 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 15:44 gotip-linux-ppc64_power10 go@18d0e6a1 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-15 21:44 gotip-linux-ppc64_power10 go@7730e5b7 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-16 15:19 gotip-linux-arm go@a22cb5ca net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 15:53 gotip-linux-arm64-boringcrypto go@2b3d98f2 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 15:53 gotip-linux-ppc64le_power9 go@2b3d98f2 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 15:56 gotip-linux-ppc64_power10 go@33d725e5 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 16:34 gotip-linux-ppc64_power8 go@6a050557 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 16:44 gotip-linux-ppc64le_power10 go@4c1cc1c9 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 17:32 gotip-linux-ppc64le_power8 go@18104621 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 18:37 gotip-linux-arm go@06b45781 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 18:42 gotip-linux-ppc64_power10 go@a523152e net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-16 18:56 gotip-linux-loong64 go@26ed0528 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-16 15:26 gotip-linux-arm go@643ad42d net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-17 01:23 gotip-linux-ppc64le_power10 go@2f642683 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-17 15:28 gotip-linux-arm go@d11e4172 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-17 15:58 gotip-linux-loong64 go@a61729b8 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-17 18:28 gotip-linux-ppc64le_power10 go@e95f6af0 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@dmitshur
Copy link
Contributor

Adding a few investigative notes.

As visible by comparing here and here, only the h1 subtest is flaky. h2 is consistently passing.

It seems to reproduce locally but requires a quite a few attempts:

src $ go test -count=10000 -run=^TestServerReadAfterWriteHeader100Continue$/^h1$ net/http
--- FAIL: TestServerReadAfterWriteHeader100Continue (25.92s)
    --- FAIL: TestServerReadAfterWriteHeader100Continue/h1 (25.91s)
        serve_test.go:7186: Get("http://127.0.0.1:59866") = Get "http://127.0.0.1:59866": dial tcp 127.0.0.1:59866: connect: operation timed out
panic: test timed out after 10m0s
	running tests:
		TestServerReadAfterWriteHeader100Continue (9m27s)
		TestServerReadAfterWriteHeader100Continue/h1 (9m27s)
[...]

@fraenkel
Copy link
Contributor

@neild Fixing the panic uncovered a lurking client side issue with 100 Continues.
The problem boils down to:

  1. The client sends an Excpect: 100-Continue
  2. The server responds with a 200
  3. persistConn.readResponse sees the 200 and closes the continueCh
  4. Request.write now closes the body and returns nil <--- this is the problem

The spec is not very explicit but it does seem that the body is still supposed to be sent.
https://datatracker.ietf.org/doc/html/rfc7231#section-5.1.1

      A server that responds with a final status code before reading the
      entire message body SHOULD indicate in that response whether it
      intends to close the connection or continue reading and discarding
      the request message (see [Section 6.6 of [RFC7230]](https://datatracker.ietf.org/doc/html/rfc7230#section-6.6)).

A question is whether there is a case when the request body should not be sent. One could look for a connection: close and perform the necessary cleanup early.

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-20 18:05 gotip-linux-ppc64_power8 go@f726c8d0 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-20 19:31 gotip-windows-386 go@d51fc260 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@neild
Copy link
Contributor

neild commented May 20, 2024

@dmitshur, @fraenkel, thanks for the analysis, it helped greatly in pointing me at the source of the problem.

TestServerReadAfterWriteHeader100Continue is flaky because it doesn't set Transport.ExpectContinueTimeout, so we actually send the request body immediately. There's a tiny window before the 0-timeout timer fires; if the server's response makes it to the client during that window, the test hangs. We can make it reliably hang by setting ExpectContinueTimeout to a long enough duration for the server to respond.

The test's handler sends a 200 response, reads the request body, and then writes the response body. (The 200 response gets an automatically-added Connection: close header.) The test hangs because there's a deadlock between the server reading the client's request body (which is never sent) and the client reading the response body (which is also never sent).

I'm not certain whether this handler is behaving reasonably or not. Sending response headers and Connection: close seems to indicate that the server doesn't want the request body. But the Connection: close was added automatically, not by the handler.

Maybe the client should just always send the body on receiving response headers. Or maybe we should close the server handler's request body after it sends headers, so it gets an error if it tries to read from it.

@fraenkel
Copy link
Contributor

@neild I too was trying to figure out which side is incorrect. Both sides seem to have issues. The behavior needs to work across clients and servers.
The client side does need to send the body since we already said a body is coming.
The server sending a final status before the entire request body is read is a tricky situation. The problem I envision is what to do and where to do it. I don't see any good outcome. You either hang waiting for data that may never come, read partial data because some of it is there or error because you sent before reading.

@neild
Copy link
Contributor

neild commented May 20, 2024

In the HTTP/1 case, receiving response headers disables the client's 100-continue timer, leading to this deadlock.

In the HTTP/2 case, receiving response headers does not disable the 100-continue timer. Setting a longer ExpectContinueTimeout causes the test to hang for the duration of that timeout while the server is waiting for the client's request body; after the timeout, the client sends the request body, the server reads it and sends the response body, and the test completes.

I'm not sure if that's all behaving as we want or not.

We probably want HTTP/1 and HTTP/2 to behave similarly, or at least for any divergence to be intentional.

@fraenkel
Copy link
Contributor

This might clarify some things:
https://www.rfc-editor.org/rfc/rfc9110#section-10.1.1

A server that responds with a final status code before reading the entire request content SHOULD indicate whether it intends to close the connection (e.g., see [Section 9.6](https://www.rfc-editor.org/rfc/rfc9112#section-9.6) of [[HTTP/1.1](https://www.rfc-editor.org/rfc/rfc9110#HTTP11)]) or continue reading the request content.

curl had this discussion https://curl.se/mail/lib-2004-08/0002.html

Mapping similar behavior to http/2 is most likely going to cause a GOAWAY.

@neild
Copy link
Contributor

neild commented May 21, 2024

TestServerReadAfterWriteHeader100Continue/h1 is flaky right now.

This test is not properly testing the behavior that it wants to exercise. It configures a client to send a request with a Expect: 100-continue header, which causes the client to wait for a server response before sending a request body. However, the test does not set the Transport.ExpectContinueTimeout configuration parameter, so the client immediately sends the body anyway.

The flaky test failures occur in when the server's response arrives before the client's (0-duration) timeout expires. Setting even a 1 millisecond ExpectContinueTimeout causes the test to consistently fail.

So: The test is wrong. But the code under test is also wrong. Fixing the test converts the flaky failure into a non-flaky failure.

The correct fix for the implementation of 100-continue is not immediately obvious.

I'm going to say that this issue (#67382) is about the flakiness. I'll make a CL that t.Skip's the test for now, which will stop the flakes (but we'll still have a bug).

I will file a new issue for the underlying bug in 100-continue handling.

@gopherbot
Copy link
Contributor Author

Change https://go.dev/cl/587155 mentions this issue: net/http: disable flaky 100-continue tests

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-20 20:23 gotip-linux-ppc64_power10 go@1028d973 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-20 20:23 gotip-darwin-arm64_13 go@1028d973 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-20 20:23 gotip-linux-arm go@1028d973 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 14:38 gotip-linux-386-softfloat go@1b9dc3e1 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 14:38 gotip-linux-loong64 go@66cc2b7c net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 15:09 gotip-linux-ppc64le_power10 go@dbe2e757 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 16:24 gotip-linux-arm go@d68d4854 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 17:17 gotip-linux-amd64-goamd64v3 go@d881ed63 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 17:17 gotip-linux-ppc64_power10 go@ba1c5b2c net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1
2024-05-21 17:17 gotip-linux-ppc64le_power10 go@ba1c5b2c net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-21 20:52 gotip-linux-arm go@64386585 net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@gopherbot
Copy link
Contributor Author

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "net/http" && test == "TestServerReadAfterWriteHeader100Continue/h1"
2024-05-21 17:17 gotip-linux-arm go@ba1c5b2c net/http.TestServerReadAfterWriteHeader100Continue/h1 [ABORT] (log)
=== RUN   TestServerReadAfterWriteHeader100Continue/h1
=== PAUSE TestServerReadAfterWriteHeader100Continue/h1
=== CONT  TestServerReadAfterWriteHeader100Continue/h1

watchflakes

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: Done
Development

No branches or pull requests

4 participants