Skip to content

concurrency.NewSession hang after etcd server is killed with SIGSTOP(19) #14631

@haojinming

Description

@haojinming

What happened?

concurrency.NewSession hang after etcd server is kill by SIGSTOP(19)

What did you expect to happen?

NewSession can return error after server is killed.

How can we reproduce it (as minimally and precisely as possible)?

  1. start three or more etcd server nodes.
  2. run main with following codes.
  3. kill -19 pidof etcd leader
package main

import (
	"fmt"
	"time"

	"github.com/pingcap/log"
	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
	"go.uber.org/zap"
)

func initEtcdClient() *clientv3.Client {
	var client *clientv3.Client
	var err error
	endpoints := []string{"172.16.5.32:2379", "172.16.5.32:2382", "172.16.5.32:2384"}
	client, err = clientv3.New(clientv3.Config{
		Endpoints:            endpoints,
		DialTimeout:          5 * time.Second,
		DialKeepAliveTimeout: 5 * time.Second,
	})
	if err != nil {
		fmt.Printf("create client fail:%v\\n", err)
		log.Panic("create client fail", zap.Error(err))
	}
	return client
}

func main() {

	number := 0
	client := initEtcdClient()
	for {
		log.Info("create session begin.", zap.Int("time", number))
		s, err := concurrency.NewSession(client)
		if err != nil {
			log.Panic("create client fail", zap.Error(err))
		}
		log.Info("create session finish.", zap.Int("time", number))
		s.Close()
		number++
		time.Sleep(time.Second)
	}
}

Anything else we need to know?

If re-create etcd client after kill -19, it can return error. However, in our application, the client is created at the beginning and stored to use in the while lifecycle of the application.

Etcd version (please run commands below)

go.etcd.io/etcd/client/v3 v3.5.5 ```console $ etcd --version # paste output here

$ etcdctl version

paste output here


</details>


### Etcd configuration (command line flags or environment variables)

<details>

# paste your configuration here

</details>


### Etcd debug information (please run commands blow, feel free to obfuscate the IP address or FQDN in the output)

<details>

```console
$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions