Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Redis Cluster issue during roll outs of new nodes with same addr #1914

Merged
merged 3 commits into from Oct 4, 2021

Conversation

pfreixes
Copy link
Contributor

@pfreixes pfreixes commented Oct 2, 2021

This issue was surfaced in a Cloud Provider solution that used for
rolling out new nodes using the same address (hostname) of the nodes
that will be replaced in a Redis Cluster, while the former ones once
depromoted as Slaves would continue in service during some mintues
for redirecting traffic.

The solution basically identifies when the connection could be stale
since a MOVED response will be returned using the same address (hostname)
that is being used by the connection. At that moment we consider the
connection as no longer usable forcing to recycle the connection.

This issue was surfaced in a Cloud Provider solution that used for
rolling out new nodes using the same address (hostname) of the nodes
that will be replaced in a Redis Cluster, while the former ones once
depromoted as Slaves would continue in service during some mintues
for redirecting traffic.

The solution basically identifies when the connection could be stale
since a MOVED response will be returned using the same address (hostname)
that is being used by the connection. At that moment we consider the
connection as no longer usable forcing to recycle the connection.
@pfreixes
Copy link
Contributor Author

pfreixes commented Oct 2, 2021

Ill investigate why some tests are failing ...

// of the connection. Force a DNS resolution when all connections
// of the pool are recycled
return true
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace else if with else and add return false as a final clause. It should probably fix the tests...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to a switch, hope that this also fixes the broken tests.

error.go Show resolved Hide resolved
@vmihailenco vmihailenco merged commit 98bb99d into redis:master Oct 4, 2021
4 checks passed
@vmihailenco
Copy link
Member

Thanks 👍

@pfreixes
Copy link
Contributor Author

pfreixes commented Oct 4, 2021

Any chance to have a bugfix release version?

@vmihailenco
Copy link
Member

Sure, I will push 8.11.4

monkey92t added a commit that referenced this pull request Mar 19, 2022
* Upgrade redis-server version (#1833)

* Upgrade redis-server version

Signed-off-by: monkey <golang@88.com>

* XAutoClaim changed the return value

Signed-off-by: monkey <golang@88.com>

* add cmd: geosearch, geosearchstore (#1836)

* add cmd: geosearch, geosearchstore

Signed-off-by: monkey92t <golang@88.com>

* GeoSearchQuery and GeoSearchLocationQuery changed to pointer passing

Signed-off-by: monkey92t <golang@88.com>

* Added missing method XInfoStreamFull to Cmdable interface

* Run go mod tidy in redisotel

Signed-off-by: Bogdan Drutu <bogdandrutu@gmail.com>

* Revert "ConnPool check  fd for bad conns (#1824)" (#1849)

This reverts commit 346bfaf.

* Automate release process (#1852)

* Bump github.com/onsi/gomega from 1.10.5 to 1.14.0 (#1832)

* Bump github.com/onsi/gomega from 1.10.5 to 1.14.0

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.10.5 to 1.14.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](onsi/gomega@v1.10.5...v1.14.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Upgrade gomega to v1.15.0

Signed-off-by: monkey92t <golang@88.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: monkey92t <golang@88.com>

* Add version.go

* Fix otel example

* Fix package name in release script

* More fixes for otel example

* And more

* Fix release.sh

* Release v8.11.3 (release.sh)

* Create an annotated tag to give release.yml chance to run

* Tweak tag.sh

* Add Cmd.Slice helper to cast to []interface{} (#1859)

* after the connection pool is closed, no new connections should be added (#1863)

* after the connection pool is closed, no new connections should be added

Signed-off-by: monkey92t <golang@88.com>

* remove runGoroutine

Signed-off-by: monkey92t <golang@88.com>

* pool.popIdle add p.closed check

Signed-off-by: monkey92t <golang@88.com>

* upgrade golangci-lint v1.42.0

Signed-off-by: monkey92t <golang@88.com>

* Bump github.com/onsi/gomega from 1.15.0 to 1.16.0 (#1865)

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.15.0 to 1.16.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](onsi/gomega@v1.15.0...v1.16.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add go 1.17 to the build matrix

* Remove go 1.15 from build matrix

* Add scan struct example (#1870)

* Replace release job

* Bump github.com/cespare/xxhash/v2 from 2.1.1 to 2.1.2 (#1872)

Bumps [github.com/cespare/xxhash/v2](https://github.com/cespare/xxhash) from 2.1.1 to 2.1.2.
- [Release notes](https://github.com/cespare/xxhash/releases)
- [Commits](cespare/xxhash@v2.1.1...v2.1.2)

---
updated-dependencies:
- dependency-name: github.com/cespare/xxhash/v2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix tag script to push tag by tag

* Fix releasing.md

* Fix/pubsub ping mutex (#1878)

* Fix PubSub.Ping to hold the lock

* Fix PubSub.Ping to hold the lock

* add write cmd data-race test

Signed-off-by: monkey92t <golang@88.com>

Co-authored-by: monkey92t <golang@88.com>

* chore: cleanup OpenTelemetry example

* chore: gofmt all code

* Refactor TestParseURL

This is in preparation for supporting query parameters
in ParseURL:

- use an expected *Options instance to execute assertions on
- extract assertions into helper function
- enable parallel testing
- condense test table

* Add query parameter parsing to ParseURL()

Before this change, ParseURL would only accept a very restricted
set of URLs (it returned an error, if it encountered any parameter).

This commit introduces the ability to process URLs like

	redis://localhost/1?dial_timeout=10s

and similar.

Go programs which were providing a configuration tunable (e.g.
CLI flag, config entry or environment variable) to configure
the Redis connection now don't need to perform this task
themselves.

* chore: add links to readme

* chore: fix discussions link

* empty hooks.withContext removed

* chore: gofmt

* chore: use conventional commits and auto-generate changelog

* feat: add acl auth support for sentinels

* chore: swap to acl auth at the test-level

* Add support for BLMove command

* chore: update dependencies

* chore: update link

* feat: add SetVal method for each command

* feat: add Cmd.{String,Int,Float,Bool}Slice helpers and an example

* chore: tweak GH actions to run all jobs

* chore: add Lua scripting example

* Fix Redis Cluster issue during roll outs of new nodes with same addr (#1914)

* fix: recycle connections in some Redis Cluster scenarios

This issue was surfaced in a Cloud Provider solution that used for
rolling out new nodes using the same address (hostname) of the nodes
that will be replaced in a Redis Cluster, while the former ones once
depromoted as Slaves would continue in service during some mintues
for redirecting traffic.

The solution basically identifies when the connection could be stale
since a MOVED response will be returned using the same address (hostname)
that is being used by the connection. At that moment we consider the
connection as no longer usable forcing to recycle the connection.

* chore: lazy reload when moved or ask

* chore: use conv commit message

* chore: release v8.11.4 (release.sh)

* fix: add whitespace for avoid unlikely colisions

* fix: format

* chore: fix links

* chore: use ctx parameter in cmdInfo

* Bump github.com/onsi/ginkgo from 1.16.4 to 1.16.5 (#1925)

Bumps [github.com/onsi/ginkgo](https://github.com/onsi/ginkgo) from 1.16.4 to 1.16.5.
- [Release notes](https://github.com/onsi/ginkgo/releases)
- [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md)
- [Commits](onsi/ginkgo@v1.16.4...v1.16.5)

---
updated-dependencies:
- dependency-name: github.com/onsi/ginkgo
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add support for time.Duration write and scan

* test: add test case for setting and scanning durations

* chore: fix linter

* fix(extra/redisotel): set span.kind attribute to client

According to the opentelemetry specification this should always be set to client for database client
libraries.

I've also removed the SetAttributes call and instead set the attributes during creation of the span.
This is what the library SHOULD be doing according to the opentelemetry api specification.

* chore: update otel example

* fix: update some argument counts in pre-allocs

In some cases number of pre-allocated places in
argument array is missing 1 or 2 elements,
which results in re-allocation of twice as large array

* chore: add example how to delete keys without a ttl

* chore: don't enable all lints

* chore(deps): bump github.com/onsi/gomega from 1.16.0 to 1.17.0

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](onsi/gomega@v1.16.0...v1.17.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* feat: Add redis v7's NX, XX, GT, LT expire variants

* chore: add missing readme

* chore: tweak feature links

* chore: remove Discord

* fix: set timeout for WAIT command. Fixes #1963

* build: update `go` directive in `go.mod` to 1.17

This commit enables support for module graph pruning and lazy module
loading for projects that are at Go 1.17 or higher.

Reference: https://go.dev/ref/mod#go-mod-file-go
Reference: https://go.dev/ref/mod#graph-pruning
Reference: https://go.dev/ref/mod#lazy-loading
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* chore: update link

* chore: export cmder.SetFirstKeyPos to support build module commands

* feat(redisotel): ability to override TracerProvider (#1998)

* fix: add missing Expire methods to Cmdable

This is a followup to #1928

* chore(deps): bump github.com/onsi/gomega from 1.17.0 to 1.18.1

Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.17.0 to 1.18.1.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](onsi/gomega@v1.17.0...v1.18.1)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update README.md (#2011)

chore: add fmt library in example code

* chore: instrumentation name and version (#2012)

* fix: invalid type assert in stringArg

* chore: cleanup

* fix: example/otel compile error (#2028)

* fix: rename Golang to Go (#2030)

https://go.dev/doc/faq#go_or_golang

* feat: add support for passing extra attributes added to spans

* feat: set net.peer.name and net.peer.port in otel example

* chore: tweak Uptrace copy

* feat: add support for COPY command (#2016)

* feat: add support for acl sentinel auth in universal client

* chore(deps): bump actions/checkout from 2 to 3

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore: add hll example

* chore: tweak release script

* chore: release v8.11.5 (release.sh)

* chore: add discord back

Co-authored-by: Eugene Ponizovsky <ponizovsky@gmail.com>
Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kishan B <kishancs46@gmail.com>
Co-authored-by: Dominik Menke <dom@digineo.de>
Co-authored-by: Gökhan Özeloğlu <gozeloglu@gmail.com>
Co-authored-by: Justin Sievenpiper <justin@sievenpiper.co>
Co-authored-by: Алексей Романовский <aromanovsky@epiphan.com>
Co-authored-by: Stavros Panakakakis <stavrospanakakis@gmail.com>
Co-authored-by: Pau Freixes <pfreixes@gmail.com>
Co-authored-by: Ethan Hur <ethan0311@gmail.com>
Co-authored-by: Jackie <18378976+Pyrodash@users.noreply.github.com>
Co-authored-by: Kristinn Björgvin Árdal <kristinnardalsecondary@gmail.com>
Co-authored-by: ffenix113 <razerer@bigmir.net>
Co-authored-by: Bastien Penavayre <bastienPenava@gmail.com>
Co-authored-by: James3 Li(李麒傑) <james3_li@asus.com>
Co-authored-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: gzjiangtao2014 <gzjiangtao2014@corp.netease.com>
Co-authored-by: Nelz <nelz9999@users.noreply.github.com>
Co-authored-by: Daniel Richter <Nexyz9@gmail.com>
Co-authored-by: Seyed Ali Ghaffari <ali.ghaffari@outlook.com>
Co-authored-by: lintanghui <lintanghui@bilibili.com>
Co-authored-by: hidu <duv123+github@gmail.com>
Co-authored-by: Jonas Lergell <jonas.lergell@volvocars.com>
Co-authored-by: Alex Kahn <alexanderkahn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants