chore: add Groq proxy and update demo script #21
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| import requests | ||
| from flask import Flask, request, jsonify, Response | ||
|
Comment on lines
+1
to
+2
There was a problem hiding this comment. 2. Proxy dependencies are uninstalled The new proxy imports Flask and Requests, but the repository's only Python requirements file contains neither package and the setup scripts perform no pip installation. A clean checkout following the new python groq_proxy.py instruction therefore fails with ModuleNotFoundError. Agent Prompt
|
||
| import os | ||
|
|
||
| app = Flask(__name__) | ||
|
|
||
| # The actual Groq API endpoint | ||
| GROQ_API_URL = "https://api.groq.com/openai/v1" | ||
|
|
||
| @app.route('/v1/chat/completions', methods=['POST']) | ||
| def proxy_chat_completions(): | ||
| try: | ||
| data = request.get_json() | ||
|
|
||
| # Strip unsupported 'reasoning_content' from messages | ||
| if 'messages' in data: | ||
| for msg in data['messages']: | ||
| if 'reasoning_content' in msg: | ||
| del msg['reasoning_content'] | ||
|
|
||
| # Forward the cleaned payload to Groq | ||
| headers = { | ||
| 'Authorization': request.headers.get('Authorization'), | ||
| 'Content-Type': 'application/json' | ||
| } | ||
|
|
||
| # Make the request to Groq, passing stream=True to support streaming | ||
| stream = data.get('stream', False) | ||
| groq_resp = requests.post( | ||
| f"{GROQ_API_URL}/chat/completions", | ||
| json=data, | ||
| headers=headers, | ||
|
Comment on lines
+29
to
+32
There was a problem hiding this comment. 5. Upstream requests never time out Both outbound request paths omit a timeout, so a stalled Groq connection can leave the corresponding proxy request blocked indefinitely. Repeated stalls can accumulate blocked Flask workers and make the local model provider unresponsive. Agent Prompt
|
||
| stream=stream | ||
| ) | ||
|
Comment on lines
+29
to
+34
There was a problem hiding this comment. Suggestion: The outbound request has no timeout, so a stalled Groq connection can block a proxy worker indefinitely and accumulate stuck requests. [possible bug] Assessment: 🟠
Prompt for AI Agent 🤖This is a comment left during a code review.
**Path:** groq_proxy.py
**Line:** 29:34
**Comment:**
*Possible Bug: The outbound request has no timeout, so a stalled Groq connection can block a proxy worker indefinitely and accumulate stuck requests.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix |
||
|
|
||
| if stream: | ||
| def generate(): | ||
| for line in groq_resp.iter_lines(): | ||
| if line: | ||
| yield line + b'\n\n' | ||
|
|
||
| return Response( | ||
| generate(), | ||
| content_type=groq_resp.headers.get('Content-Type', 'text/event-stream') | ||
| ) | ||
|
Comment on lines
+42
to
+45
There was a problem hiding this comment. Suggestion: Streaming responses use Flask's default 200 status, so Groq 401, 429, and 5xx errors reach clients as successful responses. [api mismatch] Assessment: 🟠
Prompt for AI Agent 🤖This is a comment left during a code review.
**Path:** groq_proxy.py
**Line:** 42:45
**Comment:**
*Api Mismatch: Streaming responses use Flask's default 200 status, so Groq 401, 429, and 5xx errors reach clients as successful responses.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
Comment on lines
+42
to
+45
There was a problem hiding this comment. 4. Streaming errors become successes The streaming branch omits groq_resp.status_code, causing Flask to return HTTP 200 even when Groq responds with a 4xx or 5xx. Clients can consequently treat authentication, rate-limit, and provider errors as successful streamed completions. Agent Prompt
|
||
| else: | ||
| return jsonify(groq_resp.json()), groq_resp.status_code | ||
|
|
||
| except Exception as e: | ||
| print(f"Proxy Error: {e}") | ||
| return jsonify({"error": str(e)}), 500 | ||
|
|
||
| @app.route('/', defaults={'path': ''}) | ||
| @app.route('/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS']) | ||
| def catch_all(path): | ||
| # Pass through any other endpoints without modification | ||
| headers = { | ||
| 'Authorization': request.headers.get('Authorization'), | ||
| 'Content-Type': request.headers.get('Content-Type') | ||
| } | ||
| groq_resp = requests.request( | ||
| method=request.method, | ||
| url=f"{GROQ_API_URL}/{path}", | ||
| headers=headers, | ||
|
Comment on lines
+61
to
+64
There was a problem hiding this comment. 3. Pass-through duplicates api version The catch-all appends the complete incoming path to a base URL that already ends in /v1, so a request to the configured /v1/models route is sent to Groq as /openai/v1/v1/models. Model discovery and every non-chat /v1/* endpoint therefore fail upstream. Agent Prompt
|
||
| data=request.get_data(), | ||
| params=request.args | ||
| ) | ||
| return Response(groq_resp.content, status=groq_resp.status_code, headers=dict(groq_resp.headers)) | ||
|
|
||
| if __name__ == '__main__': | ||
| print("Starting Groq API Proxy on http://0.0.0.0:8080") | ||
| print("This proxy intercepts requests and removes 'reasoning_content' to prevent 400 errors.") | ||
| app.run(host='0.0.0.0', port=8080) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
7. Narration invents mcp tool
🐞 Bug≡ CorrectnessAgent Prompt
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools