Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wayne worker run into Infinite loop if rabbitmq is down #245

Closed
silenceshell opened this Issue Jan 25, 2019 · 0 comments

Comments

Projects
None yet
3 participants
@silenceshell
Copy link
Contributor

silenceshell commented Jan 25, 2019

Describe the bug

If we shutdown the rabbitmq, the wayne worker will print quite a lot of logs such as:

2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized

To Reproduce
Steps to reproduce the behavior:

  1. deploy wayne on k8s
  2. scale replicas of rabbitmq to zeor
  3. display logs of wayne worker, will get a lot of logs as above.

Expected behavior

As rabbitmq is down, wayne worker should retry to reconnect, and should break this loop:

func processMessage(d *amqp.Delivery, w *BaseMessageWorker) {
	defer func() {
		if r := recover(); r != nil {
			logs.Critical(r)
			d.Reject(false)
		}
	}()

	var m message.Message
	err := json.Unmarshal(d.Body, &m)
	if err != nil {
		ackOrDie(d, false)
		return
	}
	if err = w.Process(&m); err != nil {
		logs.Error(err)
		ackOrDie(d, false) // 认为再次重试也无法处理
	} else {
		ackOrDie(d, false)
	}
}

Screenshots

NA

Desktop (please complete the following information):

NA

Server Version

- Kubernetes version: 1.11.1
- Wayne version: wayne v1.3.1-36-gc5157ca0c 

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.