feat: run warmup on Runtimes and Executor #5579

girishc13 · 2023-01-06T14:21:41Z

Goals:

resolves feat: run warmup requests on runtime startup to ensure that service is ready to accept connections #5467
Gateway and Head runtime now starts a warmup tasks on the run_forever runtime method.
The warmup task is created by the GatewayStreamer or the Head RequestHandler that delegates the warmup per deployments to the connection pool.
- A single task per deployment is created and await until completion.
The GrpcConnectionPool fetches the available shards and or replica (target) channels and creates a JinaInfoRPC request task.
- One task is created for each channel.
- The warmup cycle runs until:
  - all targets return a successful response or
  - a max timeout of 5 minutes is reached or
  - a stop event is set by the runtime due to early tear down of the runtime.
- In the warmup cycle, only unsuccessful targets are attempted.
CompositeGateway doesn't support the warmup due to the creation of separate GatewayStreamer objects for each type of Gateway. The issue will be addressed in feat: use single GatewayStreamer object for all gateways in the CompositeGateway #5592.
- the CompositeGateway methods shutdown, run_server, setup_server now handle each gateway operation as a task per created gateway.

codecov · 2023-01-06T15:29:51Z

Codecov Report

Merging #5579 (6232b42) into master (d70ec53) will increase coverage by 1.87%.
The diff coverage is 89.56%.

@@            Coverage Diff             @@
##           master    #5579      +/-   ##
==========================================
+ Coverage   86.11%   87.98%   +1.87%     
==========================================
  Files         124      124              
  Lines        9939    10047     +108     
==========================================
+ Hits         8559     8840     +281     
+ Misses       1380     1207     -173

Flag	Coverage Δ
jina	`87.98% <89.56%> (+1.87%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jina/serve/runtimes/worker/request_handling.py	`96.17% <ø> (+5.53%)`	⬆️
jina/serve/runtimes/head/request_handling.py	`90.78% <66.66%> (-3.33%)`	⬇️
jina/serve/runtimes/asyncio.py	`85.82% <78.57%> (+2.64%)`	⬆️
jina/serve/streamer.py	`93.33% <82.35%> (-3.28%)`	⬇️
jina/serve/networking.py	`90.45% <94.33%> (+1.93%)`	⬆️
jina/serve/runtimes/gateway/__init__.py	`98.36% <100.00%> (+0.08%)`	⬆️
jina/serve/runtimes/gateway/composite/gateway.py	`100.00% <100.00%> (ø)`
jina/serve/runtimes/head/__init__.py	`97.85% <100.00%> (+0.11%)`	⬆️
jina/serve/runtimes/worker/__init__.py	`95.74% <100.00%> (+7.88%)`	⬆️
jina/orchestrate/flow/base.py	`90.08% <0.00%> (+0.12%)`	⬆️
... and 27 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

triggering dry_run and discover endpoint requests before executor startup

…tion

…time-warmup

…into feat-serve-5467-runtime-warmup

girishc13 · 2023-01-11T08:57:20Z

jina/serve/networking.py

+
+    def __extract_target_to_channel(self, deployment):
+        replica_set = set()
+        replica_set.update(self._connections.get_replicas_all_shards(deployment))


I can extract out the warmup logic into a Warmer class but the self._connections is the inner class _ConnectionPoolMap of the GrpcConnectionPool. Does the inner class still make sense?

This reverts commit 34d7d4b.

This reverts commit 87ee55f.

jina/serve/runtimes/gateway/request_handling.py

This reverts commit 367a115.

…time-warmup

This reverts commit 479c810.

This reverts commit ab6119d.

This reverts commit a82c128.

…time-warmup

JoanFM · 2023-01-13T13:46:49Z

jina/serve/networking.py

@@ -1,6 +1,8 @@
 import asyncio
 import ipaddress
 import os
+import threading
+import time


please remove unneeded imports

JoanFM · 2023-01-13T13:48:26Z

jina/serve/runtimes/asyncio.py

@@ -1,6 +1,7 @@
 import argparse
 import asyncio
 import signal
+import threading


remove unneeded import

jina/serve/runtimes/head/request_handling.py

JoanFM · 2023-01-13T13:50:08Z

jina/serve/networking.py

+            # create a new set of stubs and channels for warmup to avoid
+            # loosing channel during remove_connection or reset_connection
+            stubs, _ = self._create_connection(address, deployment_name)
+            self._warmup_stubs.add(stubs)


I do not see where they are used

Check the usage of the property method self.warmup_stubs. What's the preference when exposing read only properties?

github-actions bot added size/M area/core This issue/PR affects the core codebase area/testing This issue/PR affects testing labels Jan 6, 2023

girishc13 mentioned this pull request Jan 6, 2023

feat: run dry_run requests as warmup with 5 min max timeout #5573

Closed

feat: add warmup coroutine to Gateway and Worker runtimes

5817aac

girishc13 force-pushed the feat-serve-5467-runtime-warmup branch from 7622872 to 5817aac Compare January 6, 2023 15:18

style: fix overload and cli autocomplete

330dd7a

girishc13 closed this Jan 6, 2023

girishc13 reopened this Jan 6, 2023

girishc13 closed this Jan 9, 2023

girishc13 reopened this Jan 9, 2023

github-actions bot added the area/cli This issue/PR affects the command line interface label Jan 9, 2023

Girish Chandrashekar and others added 9 commits January 9, 2023 17:35

test: reduce traced operations to account for warmup

10947b6

triggering dry_run and discover endpoint requests before executor startup

refactor: user threding.Event to signal graceful warmup task cancella…

967111a

…tion

style: fix overload and cli autocomplete

02faa77

chore: reduce debug logging

23e1777

feat: implement warmup using discovery requests

aa9189d

refactor: use asyncio.sleep

6824fbf

feat: create warmup task per deployment

b70008c

feat: implement warmup for HeadRuntime

c5155e8

feat: remove executor warmup task

280d481

girishc13 force-pushed the feat-serve-5467-runtime-warmup branch from 22acbb4 to 280d481 Compare January 10, 2023 16:47

Girish Chandrashekar added 2 commits January 11, 2023 09:14

feat: don't warmup deprecated head uses before and after

2885b4e

Merge remote-tracking branch 'origin/master' into feat-serve-5467-run…

0f2bf67

…time-warmup

github-actions bot removed the area/cli This issue/PR affects the command line interface label Jan 11, 2023

jina-bot and others added 3 commits January 11, 2023 08:48

style: fix overload and cli autocomplete

2be40b1

fix: revert changes to unrelated tests

4651f3b

Merge remote-tracking branch 'origin/feat-serve-5467-runtime-warmup' …

d1c7e21

…into feat-serve-5467-runtime-warmup

girishc13 commented Jan 11, 2023

View reviewed changes

Girish Chandrashekar added 4 commits January 12, 2023 11:05

Revert "feat: wait until grpc channel is ready before warmup"

4c86668

This reverts commit 34d7d4b.

feat: create JinaInfoRPC stub in the ConnectionStubs for reuse

a82c128

feat: enable grpc SO_REUSEPORT for multi process/threading

87ee55f

Revert "feat: enable grpc SO_REUSEPORT for multi process/threading"

017dfea

This reverts commit 87ee55f.

JoanFM requested changes Jan 12, 2023

View reviewed changes

jina/serve/runtimes/gateway/request_handling.py Outdated Show resolved Hide resolved

Girish Chandrashekar and others added 11 commits January 12, 2023 16:41

Revert "fix: await gather_endpoints coroutine"

99e0823

This reverts commit 367a115.

ci: debug warmup task

e11a41f

fix: pop removed connection channel from dict

ab6119d

Merge remote-tracking branch 'origin/master' into feat-serve-5467-run…

e9b6561

…time-warmup

fix: pop removed connection channel from dict

479c810

Revert "fix: pop removed connection channel from dict"

95a5767

This reverts commit 479c810.

Revert "fix: pop removed connection channel from dict"

9deda11

This reverts commit ab6119d.

Revert "feat: create JinaInfoRPC stub in the ConnectionStubs for reuse"

dd3516b

This reverts commit a82c128.

feat: create duplicate stubs for warmup requests

ada4e5e

Merge remote-tracking branch 'origin/master' into feat-serve-5467-run…

a23caf1

…time-warmup

style: fix overload and cli autocomplete

51c7e60

girishc13 closed this Jan 13, 2023

girishc13 reopened this Jan 13, 2023

github-actions bot added the area/cli This issue/PR affects the command line interface label Jan 13, 2023

Merge remote-tracking branch 'origin/master' into feat-serve-5467-run…

855bd7c

…time-warmup

github-actions bot removed the area/cli This issue/PR affects the command line interface label Jan 13, 2023

chore: remove debug logging

7211337

girishc13 marked this pull request as ready for review January 13, 2023 13:45

JoanFM requested changes Jan 13, 2023

View reviewed changes

Girish Chandrashekar added 2 commits January 13, 2023 14:58

chore: clean up imports

463a998

feat: close channels created for warmup stubs

6232b42

JoanFM approved these changes Jan 14, 2023

View reviewed changes

JoanFM merged commit ed9b448 into master Jan 14, 2023

JoanFM deleted the feat-serve-5467-runtime-warmup branch January 14, 2023 10:14

alaeddine-13 mentioned this pull request Feb 9, 2023

chore: draft release note 3.14.0 #5681

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: run warmup on Runtimes and Executor #5579

feat: run warmup on Runtimes and Executor #5579

girishc13 commented Jan 6, 2023 •

edited

codecov bot commented Jan 6, 2023 •

edited

girishc13 Jan 11, 2023

JoanFM Jan 13, 2023

JoanFM Jan 13, 2023

JoanFM Jan 13, 2023

girishc13 Jan 13, 2023

JoanFM Jan 14, 2023

feat: run warmup on Runtimes and Executor #5579

feat: run warmup on Runtimes and Executor #5579

Conversation

girishc13 commented Jan 6, 2023 • edited

codecov bot commented Jan 6, 2023 • edited

Codecov Report

girishc13 Jan 11, 2023

Choose a reason for hiding this comment

JoanFM Jan 13, 2023

Choose a reason for hiding this comment

JoanFM Jan 13, 2023

Choose a reason for hiding this comment

JoanFM Jan 13, 2023

Choose a reason for hiding this comment

girishc13 Jan 13, 2023

Choose a reason for hiding this comment

JoanFM Jan 14, 2023

Choose a reason for hiding this comment

girishc13 commented Jan 6, 2023 •

edited

codecov bot commented Jan 6, 2023 •

edited