Adds documentation on concurrent workers

docs/projects/manage-projects.md

@@ -3,58 +3,66 @@ title: Managing Projects
 description: Manage your RunPod project.
 sidebar_position: 2
 ---
- Projects enable you to develop and deploy endpoints entirely on RunPod's infrastructure.
 
- ## Create a project
- A RunPod project is a folder with everything you need to run a development session on a Pod.
+Projects enable you to develop and deploy endpoints entirely on RunPod's infrastructure.
+
+## Create a project
+
+A RunPod project is a folder with everything you need to run a development session on a Pod.
+
+1. To create a new project, run the following command.
 
-1. To create a new project, run the following command. 
 ```
 runpod project create
 ```
+
 2. Select a starter project. Starter projects include preliminary settings for different kinds of project environments, such as LLM or image diffusion development.
-3. Check the base image for included dependencies. 
-4. (Optional) If you need dependencies that are not included or added by your starter project, add them to the generated `requirements.txt` file. 
-5. Save your changes. 
+3. Check the base image for included dependencies.
+4. (Optional) If you need dependencies that are not included or added by your starter project, add them to the generated `requirements.txt` file.
+5. Save your changes.
 
 You've customized your project, and now you're ready to run a development session.
 
 ## Run a development session
 
 A development session is the active connection between your local environment and the project environment on your Pod. During a development session, local changes to your project propagate to the project environment in real time.
 
-1. To start a development session, run the following command. 
+1. To start a development session, run the following command.
+
 ```
 runpod project dev
 ```
-2. When you're done developing, press `ctrl` + `c` to end the session. Your Pod will terminate automatically when the session ends. 
+
+2. When you're done developing, press `ctrl` + `c` to end the session. Your Pod will terminate automatically when the session ends.
 
 :::tip
 
 You can resume developing at any time by running `runpod project dev` again.
 
 :::
 
-Now that you've developed your project, you can deploy an endpoint directly to RunPod or build a Dockerfile to create a portable image. 
+Now that you've developed your project, you can deploy an endpoint directly to RunPod or build a Dockerfile to create a portable image.
 
- ## Deploy a project
+## Deploy a project
 
- When you deploy a project, RunPod creates a serverless endpoint with access to saved project data on your network volume.
+When you deploy a project, RunPod creates a serverless endpoint with access to saved project data on your network volume.
 
- To deploy a project, run the following command. 
- ```
- runpod project deploy
- ```
+To deploy a project, run the following command.
 
- You now have a serverless endpoint. To learn more, refer to [Endpoints](/docs/serverless/endpoints/).
+```
+runpod project deploy
+```
+
+You now have a serverless endpoint. To learn more, refer to [Endpoints](/docs/serverless/endpoints/).
 
- ## Build a project
+## Build a project
 
- When you build a project, RunPod emits a Dockerfile. 
+When you build a project, RunPod emits a Dockerfile.
 
- To build a project, run the following command.
- ```
- runpod project build
-``` 
+To build a project, run the following command.
+
+```
+runpod project build
+```
 
- You can use the generated Dockerfile to build an image, then deploy the image to any API server.
+You can use the generated Dockerfile to build an image, then deploy the image to any API server.

docs/serverless/workers/handlers/handler-concurrency.md

@@ -0,0 +1,112 @@
+---
+title: Concurrent Handlers
+---
+
+RunPod supports asynchronous functions for request handling, enabling a single worker to manage multiple tasks concurrently through non-blocking operations. This capability allows for efficient task switching and resource utilization.
+
+Serverless architectures allow each worker to process multiple requests simultaneously, with the level of concurrency being contingent upon the runtime's capacity and the resources at its disposal.
+
+## Configure concurrency modifier
+
+The `concurrency_modifier` is a configuration option within `runpod.serverless.start` that dynamically adjusts a worker's concurrency level. This adjustment enables the optimization of resource consumption and performance by regulating the number of tasks a worker can handle concurrently.
+
+### Step 1: Define an asynchronous Handler function
+
+Create an asynchronous function dedicated to processing incoming requests.
+This function should efficiently yield results, ideally in batches, to enhance throughput.
+
+```python
+async def process_request(job):
+    # Simulates processing delay
+    await asyncio.sleep(1)
+    return f"Processed: {job['input']}"
+```
+
+### Step 2: Set up the `concurrency_modifier` function
+
+Implement a function to adjust the worker's concurrency level based on the current request load.
+This function should consider the maximum and minimum concurrency levels, adjusting as needed to respond to fluctuations in request volume.
+
+```python
+def adjust_concurrency(current_concurrency):
+    """
+    Dynamically adjusts the concurrency level based on the observed request rate.
+    """
+    global request_rate
+    update_request_rate()  # Placeholder for request rate updates
+
+    max_concurrency = 10  # Maximum allowable concurrency
+    min_concurrency = 1   # Minimum concurrency to maintain
+    high_request_rate_threshold = 50  # Threshold for high request volume
+
+    # Increase concurrency if under max limit and request rate is high
+    if request_rate > high_request_rate_threshold and current_concurrency < max_concurrency:
+        return current_concurrency + 1
+    # Decrease concurrency if above min limit and request rate is low
+    elif request_rate <= high_request_rate_threshold and current_concurrency > min_concurrency:
+        return current_concurrency - 1
+
+    return current_concurrency
+```
+
+### Step 3: Initialize the serverless function
+
+Start the serverless function with the defined handler and `concurrency_modifier` to enable dynamic concurrency adjustment.
+
+```python
+runpod.serverless.start({
+    "handler": process_request,
+    "concurrency_modifier": adjust_concurrency,
+})
+```
+
+---
+
+## Example code
+
+Here is an example demonstrating the setup for a RunPod serverless function capable of handling multiple concurrent requests.
+
+```python
+import runpod
+import asyncio
+import random
+
+# Simulated Metrics
+request_rate = 0
+
+async def process_request(job):
+    await asyncio.sleep(1)  # Simulate processing time
+    return f"Processed: { job['input'] }"
+
+def adjust_concurrency(current_concurrency):
+    """
+    Adjusts the concurrency level based on the current request rate.
+    """
+    global request_rate
+    update_request_rate()  # Simulate changes in request rate
+
+    max_concurrency = 10
+    min_concurrency = 1
+    high_request_rate_threshold = 50
+
+    if request_rate > high_request_rate_threshold and current_concurrency < max_concurrency:
+        return current_concurrency + 1
+    elif request_rate <= high_request_rate_threshold and current_concurrency > min_concurrency:
+        return current_concurrency - 1
+    return current_concurrency
+
+def update_request_rate():
+    """
+    Simulates changes in the request rate to mimic real-world scenarios.
+    """
+    global request_rate
+    request_rate = random.randint(20, 100)
+
+# Start the serverless function with the handler and concurrency modifier
+runpod.serverless.start({
+    "handler": process_request,
+    "concurrency_modifier": adjust_concurrency
+})
+```
+
+Using the `concurrency_modifier` in RunPod, serverless functions can efficiently handle multiple requests concurrently, optimizing resource usage and improving performance. This approach allows for scalable and responsive serverless applications.

docs/tutorials/introduction/containers/docker-commands.md

@@ -4,7 +4,7 @@ title: Docker commands
 
 RunPod enables bring-your-own-container (BYOC) development. If you choose this workflow, you will be using Docker commands to build, run, and manage your containers.
 
-:::note 
+:::note
 
 For a Dockerless workflow, see [RunPod projects](/docs/projects/overview.md).
 

docs/tutorials/introduction/containers/overview.md

@@ -31,7 +31,8 @@ To create Docker images, you use a process known as "Docker build." This process
 Using Docker images helps in various stages of software development, including testing, development, and deployment. Images ensure a seamless workflow across diverse computing environments.
 
 ### Why not use images?
-You must rebuild and push the container image, then edit your endpoint to use the new image each time you iterate on your code. Since development requires changing your code every time you need to troubleshoot a problem or add a feature, this workflow can be inconvenient. 
+
+You must rebuild and push the container image, then edit your endpoint to use the new image each time you iterate on your code. Since development requires changing your code every time you need to troubleshoot a problem or add a feature, this workflow can be inconvenient.
 
 For a streamlined development workflow, check out [RunPod projects](/docs/projects/overview.md). When you're done with development, you can create a Dockerfile from your project to reduce initialization overhead in production.
 

docs/tutorials/migrations/banana/images.md

@@ -26,18 +26,16 @@ import base64
 # create a new Potassium app
 app = Potassium("my_app")
 
+
 # @app.init runs at startup, and loads models into the app's context
 @app.init
 def init():
-    repo_id="Meina/MeinaUnreal_V3"
+    repo_id = "Meina/MeinaUnreal_V3"
 
     ddpm = DDPMScheduler.from_pretrained(repo_id, subfolder="scheduler")
-    
+
     model = DiffusionPipeline.from_pretrained(
-        repo_id, 
-        use_safetensors=True,
-        torch_dtype=torch.float16,
-        scheduler=ddpm
+        repo_id, use_safetensors=True, torch_dtype=torch.float16, scheduler=ddpm
     ).to("cuda")
 
     context = {
@@ -46,6 +44,7 @@ def init():
 
     return context
 
+
 # @app.handler runs for every call
 @app.handler()
 def handler(context: dict, request: Request) -> Response:
@@ -59,7 +58,9 @@ def handler(context: dict, request: Request) -> Response:
         negative_prompt=negative_prompt,
         guidance_scale=7,
         num_inference_steps=request.json.get("steps", 30),
-        generator=torch.Generator(device="cuda").manual_seed(request.json.get("seed")) if request.json.get("seed") else None,
+        generator=torch.Generator(device="cuda").manual_seed(request.json.get("seed"))
+        if request.json.get("seed")
+        else None,
         width=512,
         height=512,
     ).images[0]
@@ -68,10 +69,8 @@ def handler(context: dict, request: Request) -> Response:
     image.save(buffered, format="JPEG", quality=80)
     img_str = base64.b64encode(buffered.getvalue())
 
-    return Response(
-        json = {"output": str(img_str, "utf-8")}, 
-        status=200
-    )
+    return Response(json={"output": str(img_str, "utf-8")}, status=200)
+
 
 if __name__ == "__main__":
     app.serve()
@@ -101,10 +100,11 @@ try:
 except RuntimeError:
     quit()
 
+
 def handler(job):
-    """ Handler function that will be used to process jobs. """
-    job_input = job['input']
-    prompt = job_input['prompt']
+    """Handler function that will be used to process jobs."""
+    job_input = job["input"]
+    prompt = job_input["prompt"]
 
     time_start = time.time()
     image = pipe(prompt=prompt, num_inference_steps=1, guidance_scale=0.0).images[0]
@@ -114,7 +114,7 @@ def handler(job):
     image.save(buffer, format="PNG")
     image_bytes = buffer.getvalue()
 
-    return base64.b64encode(image_bytes).decode('utf-8')
+    return base64.b64encode(image_bytes).decode("utf-8")
 
 
 runpod.serverless.start({"handler": handler})

docusaurus.config.js

@@ -4,7 +4,9 @@
 // There are various equivalent ways to declare your Docusaurus config.
 // See: https://docusaurus.io/docs/api/docusaurus-config
 import { injectSpeedInsights } from "@vercel/speed-insights";
-
+const {
+  remarkCodeHike,
+} = require("@code-hike/mdx");
 import path from "path";
 import { themes as prismThemes } from "prism-react-renderer";
 
@@ -42,6 +44,9 @@ const config = {
           // Remove this to remove the "edit this page" links.
           routeBasePath: "",
           editUrl: "https://github.com/runpod/docs/blob/main",
+          beforeDefaultRemarkPlugins: [
+            [remarkCodeHike, { theme: "nord" }],
+          ],
         },
         /*
         blog: {
@@ -55,6 +60,7 @@ const config = {
         theme: {
           customCss: [
             "./src/css/custom.css",
+            require.resolve("@code-hike/mdx/styles.css"),
           ],
         },
       }),