Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue 2540: Synchronise concurrent command calls to single-client to single-client mode #2568

Merged
merged 1 commit into from Jan 29, 2023

Conversation

Vivanov98
Copy link
Contributor

@Vivanov98 Vivanov98 commented Jan 28, 2023

Pull Request check-list

  • Does $ tox pass with this change (including linting)?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

Description of change

Fixing issue #2540 (example script in issue tested and bug no longer occurs)

The underlying cause is that the connection in single-client mode isn't protected by a lock, resulting in multiple coroutines reading/writing to a connection stream when multiple commands are called in parallel (forexample with asyncio.gather)

Additionally, redundant connections can be made if the first time the client is used, it is used in parallel (forexample with asyncio.gather).

A simple lock mechanism has been added to the execute_command function to ensure that single-client mode is always executing commands in sequence. This lock is also used each time the initialize method is called, ensuring that multiple connections aren't spun up if the first time the client is called happens to be with a parallel mechanism.

Worth mentioning that the use cases for using parallel calling mechanisms in single-client mode are probably limited, though this still seems useful to have in terms of consistency.

@codecov-commenter
Copy link

codecov-commenter commented Jan 28, 2023

Codecov Report

Base: 92.26% // Head: 92.27% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (6660ed5) compared to base (e39c7ba).
Patch coverage: 97.29% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2568   +/-   ##
=======================================
  Coverage   92.26%   92.27%           
=======================================
  Files         115      115           
  Lines       29610    29645   +35     
=======================================
+ Hits        27320    27354   +34     
- Misses       2290     2291    +1     
Impacted Files Coverage Δ
tests/test_asyncio/test_connection.py 97.39% <96.42%> (-0.31%) ⬇️
redis/asyncio/client.py 93.27% <100.00%> (+0.76%) ⬆️
tests/test_asyncio/test_cluster.py 97.48% <0.00%> (-0.13%) ⬇️
tests/test_cluster.py 96.84% <0.00%> (-0.11%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

init_call_count += 1
return mock_conn

with mock.patch.object(r.connection_pool, "get_connection", get_conn):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only deterministic test to validate the locking does what we want that I could think of required patching the connection and call_with_retry objects. This test is consistent, and would fail without the client changes.

@dvora-h dvora-h merged commit 428d609 into redis:master Jan 29, 2023
chayim pushed a commit that referenced this pull request Aug 31, 2023
Co-authored-by: Viktor Ivanov <viktor@infogrid.io>
Co-authored-by: Sergey Prokazov <sergey.prokazov@redis.com>
Co-authored-by: Anuragkillswitch <70265851+Anuragkillswitch@users.noreply.github.com>
Co-authored-by: dvora-h <67596500+dvora-h@users.noreply.github.com>
Co-authored-by: Alex Schmitz <aschmitz@box.com>
Co-authored-by: Alex Schmitz <alex.schmitz@gmail.com>
Co-authored-by: Chayim <chayim@users.noreply.github.com>
Co-authored-by: Bar Shaul <88437685+barshaul@users.noreply.github.com>
Co-authored-by: CrimsonGlory <CrimsonGlory@users.noreply.github.com>
Co-authored-by: Raymond Yin <raymond@tryevergreen.com>
Co-authored-by: zach.lee <zach.lee@sendbird.com>
Co-authored-by: James R T <jamestiotio@gmail.com>
Co-authored-by: dvora-h <dvora.heller@redis.com>
Co-authored-by: Marc Schöchlin <marc.schoechlin@flipapp.de>
Co-authored-by: Nick Gerow <nick.gerow@enlightedinc.com>
Co-authored-by: Igor Malinovskiy <u.glide@gmail.com>
Co-authored-by: Chayim I. Kirshen <c@kirshen.com>
Co-authored-by: Leibale Eidelman <me@leibale.com>
Co-authored-by: Thiago Bellini Ribeiro <hackedbellini@gmail.com>
Co-authored-by: woutdenolf <woutdenolf@users.sf.net>
Co-authored-by: shacharPash <93581407+shacharPash@users.noreply.github.com>
Co-authored-by: Mirek Długosz <miniopl+github@gmail.com>
Co-authored-by: Oran Avraham <252748+oranav@users.noreply.github.com>
Co-authored-by: mzdehbashi-github <85902780+mzdehbashi-github@users.noreply.github.com>
Co-authored-by: Tyler Hutcherson <tyler.hutcherson@redis.com>
Co-authored-by: Felipe Machado <462154+felipou@users.noreply.github.com>
Co-authored-by: AYMEN Mohammed <53928879+AYMENJD@users.noreply.github.com>
Co-authored-by: Marc Schöchlin <ms-github@256bit.org>
Co-authored-by: Avasam <samuel.06@hotmail.com>
Co-authored-by: Markus Gerstel <2102431+Anthchirp@users.noreply.github.com>
Co-authored-by: Kristján Valur Jónsson <sweskman@gmail.com>
Co-authored-by: Nick Gerow <Nick.G.123@hotmail.com>
Co-authored-by: Cristian Matache <cristianmatache@hotmail.com>
Co-authored-by: Anurag Bandyopadhyay <angbpy@gmail.com>
Co-authored-by: Seongchuel Ahn <aciddust20@gmail.com>
Co-authored-by: Alibi <aliby.bbb@gmail.com>
Co-authored-by: Smit Parmar <smitraj333@gmail.com>
Co-authored-by: Brad MacPhee <macphee@gmail.com>
Co-authored-by: Shahar Lev <shahar_lev@hotmail.com>
Co-authored-by: Vladimir Mihailenco <vladimir.webdev@gmail.com>
Co-authored-by: Kevin James <KevinJames@thekev.in>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Pacsuta <34983281+ant1fact@users.noreply.github.com>
Co-authored-by: Rich Bowen <rbowen@rcbowen.com>
Co-authored-by: gmbnomis <gmbnomis@users.noreply.github.com>
Co-authored-by: Vivanov98 <66319645+Vivanov98@users.noreply.github.com>
Co-authored-by: Kosuke <kosuke.zhang@gmail.com>
Co-authored-by: Sergey Prokazov <prokazov@users.noreply.github.com>
Co-authored-by: jmcbailey <jmcbailey@gmail.com>
Co-authored-by: Galtozzy <14139502+Galtozzy@users.noreply.github.com>
Co-authored-by: Abhishek Kumar Sinha <aksinha334@gmail.com>
Co-authored-by: Eom Taegyung "Iggy <iggy.eom@sendbird.com>
Co-authored-by: Mehdi ABAAKOUK <sileht@sileht.net>
Co-authored-by: Dongkeun Lee <3315213+zakaf@users.noreply.github.com>
Co-authored-by: woutdenolf <wout.de_nolf@esrf.eu>
Co-authored-by: Kurt McKee <contactme@kurtmckee.org>
Co-authored-by: Juraj Páll <palljuraj1@gmail.com>
Co-authored-by: Joan Fontanals <jfontanalsmartinez@gmail.com>
Co-authored-by: Stanislav Zmiev <zmievsa@gmail.com>
fix (#2566)
Fix unlink in cluster pipeline (#2562)
Fix issue 2540: Synchronise concurrent command calls to single-client mode. (#2568)
Fix: tuple function cannot be passed more than one argument (#2573)
Fix issue 2567: NoneType check before raising exception (#2569)
Fix issue 2349: Let async HiredisParser finish parsing after a Connection.disconnect() (#2557)
Fix issue with `pack_commands` returning an empty byte sequence (#2416)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' (#2583)
Fix #2581 UnixDomainSocketConnection' object has no attribute '_command_packer' .
Fix for `lpop` and `rpop` return typing (#2590)
Fixed CredentialsProvider examples (#2587)
Fixed issue #2598 - make Document class subscriptable
fix: replace async_timeout by asyncio.timeout (#2602)
Fix behaviour of async PythonParser to match RedisParser as for issue #2349 (#2582)
Fix (#2641)
fix: do not use asyncio's timeout lib before 3.11.2 (#2659)
Fix issue 2660: PytestUnraisableExceptionWarning from asycio client (#2669)
Fixing cancelled async futures (#2666)
Fix async (#2673)
Fix memory leak caused by hiredis (#2693) (#2694)
Fix incorrect usage of once flag in async Sentinel (#2718)
Fix topk list example. (#2724)
Fix `ClusterCommandProtocol` not itself being marked as a protocol (#2729)
Fix potential race condition during disconnection (#2719)
fix CI (#2748)
fix parse_slowlog_get (#2732)
fixes for issue #1128
fix create single_connection_client from url (#2752)
Fix `xadd` allow non negative maxlen (#2739)
Fix JSON.MERGE Summary (#2786)
Fixed key error in parse_xinfo_stream (#2788)
Fix dead weakref in sentinel connection causing ReferenceError (#2767) (#2771)
Fix dead weakref in sentinel conn (#2767)
fix redirects and some small cleanups (#2801)
Fix type hint for retry_on_error in async cluster (#2804)
Fix CI (#2809)
Fix async client with resp3 (#2657)
Fix `COMMAND` response in resp3 (redis 7+) (#2740)
Fix protocol version checking (#2737)
Fix parse resp3 dict response: don't use dict comprehension (#2757)
Fixing asyncio import (#2759)
fix (#2799)
fix async tests (#2806)
Fix socket garbage collection (#2859)
Fixing doc builds (#2869)
Fix a duplicate word in `CONTRIBUTING.md` (#2848)
Fix timeout retrying on Redis pipeline execution (#2812)
Fix type hints in SearchCommands (#2817)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants