upgrade to v5: curl sometimes reports: curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

[comphilip]

version: v5.0.4

func main() {
	os.Exit(cli.Run(new(options), func(ctx *cli.Context) error {
		argv := ctx.Argv().(*options)

		e := echo.New()
		e.Logger = slog.New(slog.NewTextHandler(os.Stdout, nil))
		e.Use(middleware.CORS("*"))
		e.Use(middleware.RequestLogger())
		e.Use(echoprometheus.NewMiddleware("echo"))
		e.Static("/package/", argv.PackageDirectory)
		log.Fatal(e.Start(fmt.Sprintf(":%d", argv.ListenPort)))
		return nil
	}))
}

this service behind traefik proxy v3.6.10.
in small ratio cases, getting files curl -LOJ https://internal.com/package/xxx reports: curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

Same issue happened on another web service with stream response

	c.Response().Header().Add(echo.HeaderContentDisposition, fileName)
	stream, err := getLogStream(c.Request().Context())
	if err != nil {
		return err
	}
	defer stream.Close()
	return c.Stream(http.StatusOK, echo.MIMETextPlainCharsetUTF8, stream)

Encountered exception during execution: IncompleteRead(56825 bytes read)
Traceback (most recent call last):
  File "/usr/lib/python3.12/http/client.py", line 584, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 551, in _read_next_chunk_size
    return int(line, 16)
           ^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.12/http/client.py", line 600, in _read_chunked
    while (chunk_left := self._get_chunk_left()) is not None:
                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 586, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 989, in run_context
    yield self
  File "/home/ubuntu/venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 1634, in run_flow_sync
    engine.call_flow_fn()
  File "/home/ubuntu/venv/lib/python3.12/site-packages/prefect/flow_engine.py", line 1009, in call_flow_fn
    result = call_with_parameters(self.flow.fn, self.parameters)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/venv/lib/python3.12/site-packages/prefect/utilities/callables.py", line 210, in call_with_parameters
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvnd0577jprefect/prefect-scripts-master/flows/data/sync_stock_config_and_log.py", line 80, in main
    _get_config_and_log(client, exchange, strategies, day)
  File "/tmp/tmpvnd0577jprefect/prefect-scripts-master/flows/data/sync_stock_config_and_log.py", line 43, in _get_config_and_log
    shutil.copyfileobj(stream, f)
  File "/usr/lib/python3.12/shutil.py", line 203, in copyfileobj
    while buf := fsrc_read(length):
                 ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 478, in read
    return self._read_chunked(amt)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 612, in _read_chunked
    raise IncompleteRead(b''.join(value)) from exc
http.client.IncompleteRead: IncompleteRead(56825 bytes read)

never happened in echo v4.

it seems echo close connection too early

[aldas]

if you add recovery middleware - does it change?

	e.Use(middleware.RequestLogger())
	e.Use(middleware.Recover())

[comphilip]

with middleware.RequestLogger() but no middleware.Recover(), got error log

time=2026-03-19T09:01:06.549Z level=ERROR msg=REQUEST_ERROR method=GET uri=/v1/streams/2630/log/20260319 status=200 latency=30.001921489s host=internal.unitloop.com bytes_in="" bytes_out=917704696 user_agent=curl/8.5.0 remote_ip=10.249.20.7 request_id="" error="write tcp 10.42.0.222:80->10.42.2.0:44176: i/o timeout"

[aldas]

but write tcp 10.42.0.222:80->10.42.2.0:44176: i/o timeout seems more like the Go HTTP server that Echo uses can not send the response to the connected client.
And you say that at the other end (Client side) you see incorrectly closed connections.
This seems to indicate that some proxy between the client and HTTP server is closing the connection or there are network problems temporarly.

[comphilip]

10.42.0.222:80->10.42.2.0:44176, 10.42.0.222:80 is echo service k8s pod address, 10.42.2.0:44176 is traefik proxy pod address

client <---> traefik proxy <---> echo service, all of them in same local network (same data rack), network speed is 10G, wonder why write tcp timeout

[aldas]

maybe there are some reconfigurations done in k8 network or traefik proxy is being redeployed/reloaded? Or another theory - it could happen when the client has bad connection - something like poor wifi connection.

edit: I forgot that everything is in the same rack.

[aldas]

are you using only CORS, RequestLogger, Prometheus middlewares? If you look inside debugger at c.Response() - how many http.ResponseWriter it is wrapping?

[comphilip]

for a simple static file service, only

		e.Use(middleware.CORS("*"))
		e.Use(middleware.RequestLogger())
		e.Use(echoprometheus.NewMiddleware("echo"))

echo v5 on 2026-03-17 on business idle time window, no other system component changed: traefik, kubernetes and so on.

then issues happened since 2026-03-18

[aldas]

Well. v4 has hack for listener KeepAlive. This should not be the for a long time, as Go standard library did some changes related to that in ~1.10 but to be use you can try to deploy custom http server that creates these same listener as v4 does.

package main

import (
	"context"
	"errors"
	"log/slog"
	"net"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/labstack/echo/v5"
	"github.com/labstack/echo/v5/middleware"
)

func main() {
	e := echo.New()
	e.Logger = slog.New(slog.NewTextHandler(os.Stdout, nil))

	e.Use(middleware.CORS("*"))
	e.Use(middleware.RequestLogger())
	e.Use(middleware.Recover())
	//e.Use(echoprometheus.NewMiddleware("echo"))

	//e.Static("/package/", argv.PackageDirectory)

	//port := fmt.Sprintf(":%d", argv.ListenPort)
	port := ":8080"
	runServer(e, port)
}

func runServer(e *echo.Echo, port string) {
	listner, err := newListener(port, "tcp")
	if err != nil {
		slog.Error("failed to create listener", "error", err)
		return
	}
	defer listner.Close()

	s := http.Server{
		Handler:           e,
		ReadHeaderTimeout: 30 * time.Second,
		WriteTimeout:      30 * time.Second,
	}

	serverErr := make(chan error, 1)
	go func() {
		if err := s.Serve(listner); err != nil && !errors.Is(err, http.ErrServerClosed) {
			serverErr <- nil
			return
		}
		serverErr <- nil
	}()

	ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer cancel()
	select {
	case err := <-serverErr:
		if err != nil {
			e.Logger.Error("failed to run server", "error", err)
		}
		return
	case <-ctx.Done():
	}

	if err := s.Shutdown(ctx); err != nil {
		e.Logger.Error("failed to shutdown server", "error", err)
		if err := s.Close(); err != nil {
			e.Logger.Error("failed to close server", "error", err)
		}
	}
}

// tcpKeepAliveListener sets TCP keep-alive timeouts on accepted
// connections. It's used by ListenAndServe and ListenAndServeTLS so
// dead TCP connections (e.g. closing laptop mid-download) eventually
// go away.
type tcpKeepAliveListener struct {
	*net.TCPListener
}

func (ln tcpKeepAliveListener) Accept() (c net.Conn, err error) {
	if c, err = ln.AcceptTCP(); err != nil {
		return
	} else if err = c.(*net.TCPConn).SetKeepAlive(true); err != nil {
		return
	}
	// Ignore error from setting the KeepAlivePeriod as some systems, such as
	// OpenBSD, do not support setting TCP_USER_TIMEOUT on IPPROTO_TCP
	_ = c.(*net.TCPConn).SetKeepAlivePeriod(3 * time.Minute)
	return
}

func newListener(address, network string) (*tcpKeepAliveListener, error) {
	l, err := net.Listen(network, address)
	if err != nil {
		return nil, err
	}
	return &tcpKeepAliveListener{l.(*net.TCPListener)}, nil
}

[aldas]

note to self: echo.StartConfig should have way to use custom Listeners.
relates to #1942

[comphilip]

I can reproduce the error in localhost, just echo web server and curl directly.

1. static host a directory contains a big file, in localhost
2. download at 1MBps via curl --limit-rate 1M -o /dev/null http://localhost:8000/package/big.log
3. echo closes connection about 40 seconds

environment: Ubuntu 24.04.3 LTS"

func main() {
	os.Exit(cli.Run(new(options), func(ctx *cli.Context) error {
		argv := ctx.Argv().(*options)

		e := echo.New()
		e.Logger = slog.New(slog.NewTextHandler(os.Stdout, nil))
		e.Use(middleware.RequestLogger())
		e.Static("/", argv.PackageDirectory)
		log.Fatal(e.Start(fmt.Sprintf(":%d", argv.ListenPort)))
		return nil
	}))
}

echo log

time=2026-03-19T21:57:43.423+08:00 level=INFO msg="Echo (v5.0.4). High performance, minimalist Go web framework https://echo.labstack.com" version=5.0.4
time=2026-03-19T21:57:43.423+08:00 level=INFO msg="http(s) server started" address=[::]:8000
time=2026-03-19T21:58:42.898+08:00 level=INFO msg=REQUEST method=GET uri=/big.log status=200 latency=30.000434905s host=localhost:8000 bytes_in="" bytes_out=41771288 user_agent=curl/8.5.0 remote_ip=::1 request_id=""

curl

curl --limit-rate 1M  -o /dev/null http://localhost:8000/big.log
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  4  992M    4 39.8M    0     0  1024k      0  0:16:31  0:00:39  0:15:52 1007k
curl: (18) transfer closed with 998498709 bytes remaining to read

[comphilip]

with github copilt's help, I need to override WriteTimeout to zero

		sc := echo.StartConfig{
			Address: fmt.Sprintf(":%d", argv.ListenPort),
			BeforeServeFunc: func(s *http.Server) error {
				s.WriteTimeout = 0
				return nil
			},
		}
		if err := sc.Start(context.Background(), e); !errors.Is(err, http.ErrServerClosed) {
			log.Panic(err)
		}

Thanks #2883 , Also #1953 , there is no error log even though the connection is closed by timeout, hard to troubleshooting

[aldas]

@comphilip if this problems reappears or did not go away, let me know. I am not convinced that WriteTimeout=0 is here the reason why v4 worked and v5 not.

Docs say

	// WriteTimeout is the maximum duration before timing out
	// writes of the response. It is reset whenever a new
	// request's header is read. Like ReadTimeout, it does not
	// let Handlers make decisions on a per-request basis.
	// A zero or negative value means there will be no timeout.

It could help maybe with streaming then there are no messages to the client for some time.

[comphilip]

@aldas problems gone away after WriteTimeout=0