feat: Add local model worker parameters

[shaohuzhang1]

fix: Ollama maximum output token field
feat: Add local model worker parameters

[f2c-ci-robot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[f2c-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[shaohuzhang1]

The provided Gunicorn command line is mostly correct, but there are a few minor improvements and clarifications that could be made:

1. Worker Configuration: The worker setting should be more robust. Instead of hardcoding it to 1, you can use a variable that checks if the configuration has been set; otherwise, default it to 1. This ensures that the application works even if the configuration file does not specify this option.

2. Security Considerations: Ensure that paths in the environment variables like LOCAL_MODEL_HOST, LOCAL_MODEL_PORT, etc., are properly sanitized to avoid security vulnerabilities such as injection attacks if they come from user input.

3. Logging Format: Double-check the format string for logging (--access-logformat) to ensure it correctly reflects what information needs to be logged by your application. You might want to adjust it based on your specific requirements.

Here’s an improved version of the command with these considerations:

@@ -24,12 +24,14 @@ def cmd(self):
     os.environ.setdefault('SERVER_NAME', 'local_model')
     log_format = '%(h)s %(t)s %(L)ss "%(r)s" %(s)s %(b)s '
     bind = f'{CONFIG.get("LOCAL_MODEL_HOST")}:{CONFIG.get("LOCAL_MODEL_PORT")}'
+    # Set worker using dynamic value or default
+    worker_setting = CONFIG.get("LOCAL_MODEL_HOST_WORKER")
+    if worker_setting is None:
+        worker_setting = 1
    
     cmd = [
         'gunicorn', 'smartdoc.wsgi:application',
         '-b', bind,
         '-k', 'gthread',
         '--threads', '200',
-        '-w', "1",
+        '-w', str(worker_setting),
         '--max-requests', '10240',
         '--max-requests-jitter', '2048',
         '--access-logformat', log_format,

Make sure to replace 'smartdoc.wsgi' with the actual path to your WSGI module if different, depending on how your Flask app is structured.

[shaohuzhang1]

The provided code snippet appears to be a section of a configuration dictionary class defined in Python. Here is a brief review for any potential issues:

1. Typo Correction: There is a typo in 'LOCAL_MODEL_HOST_WORKER'. The value should likely be an integer representing the number of worker processes, but it is currently missing its closing quotes.

2. Code Formatting: While not strictly necessary, indentation can improve readability for those unfamiliar with Python's syntax.

Suggested corrections:

@@ -93,7 +93,8 @@
     'SANDBOX': False,
     'LOCAL_MODEL_HOST': '127.0.0.1',
     'LOCAL_MODEL_PORT': '11636',
-    'LOCAL_MODEL_PROTOCOL': "http"
+    'LOCAL_MODEL_PROTOCOL': "http",
+    'LOCAL_MODEL_HOST_WORKER': 4  # Assuming you want four workers

These changes address the identified issue and make the code more readable and correct. If there are further optimizations needed, additional context about what these configurations represent (e.g., database settings, API endpoints) would be helpful.

[shaohuzhang1]

The code snippet you provided has a small typo in max_tokens that should be corrected to num_predict. Here's the updated line:

Updated Code Snippet:

num_predict = forms.SliderField(

No other significant issues or optimizations are present in this particular part of the code.