Streaming responses using urllib3

[psymbio]

Context

What are you trying to do?
I'm trying to accommodate streaming responses for Openai's python library.

How do you expect to do it?
Using a custom transport layer with another class to support streaming responses. I think Tom's answer on encode/httpx#3078 is correct on the thinking - but I've tried a couple of different ways to solve the problem and they've not worked out for me.

Is it something you currently cannot do?
Wrap a streaming output properly.

Is this related to an existing issue/problem?
Yes: encode/httpx#3078
Some more context: pyodide/pyodide#4292

Alternatives

Can you achieve the same result doing it in an alternative way? Not really.
Is the alternative considerable?

Duplicate

Has the feature been requested before? Haven't come across it.
If so, please provide a link to the issue.

Contribution

Would you be willing to submit a PR? Not sure if this will be required.
(Help can be provided if you need assistance submitting a PR)

[sethmlarson]

@psymbio Have you checked out the existing methods for streaming response data in our documentation?

[sethmlarson]

Closing due to lack of activity, I think this question has been answered.

[psymbio]

Sorry for the delay in response @sethmlarson

Yes, I've looked at the documentation and written the following code:

import micropip
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/multidict/multidict-4.7.6-py3-none-any.whl', keep_going=True)
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/frozenlist/frozenlist-1.4.0-py3-none-any.whl', keep_going=True)
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/aiohttp/aiohttp-3.9.1-py3-none-any.whl', keep_going=True)
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/openai/openai-1.3.7-py3-none-any.whl', keep_going=True)
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/urllib3/urllib3-2.1.0-py3-none-any.whl', keep_going=True)
 
await micropip.install("ssl")
import ssl
await micropip.install("httpx", keep_going=True)
import httpx
await micropip.install('https://raw.githubusercontent.com/psymbio/pyodide_wheels/main/urllib3/urllib3-2.1.0-py3-none-any.whl', keep_going=True)
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
import json
 
class URLLib3Stream(httpx.SyncByteStream):
    def __init__(self, stream):
        self._stream = stream
 
    def __iter__(self):
        # print(self._stream)
        # print(dir(self._stream))
        # print(self._stream.__self__)
        # print(dir(self._stream.__self__))
        print(self._stream.__self__.data)
        with httpx._transports.default.map_httpcore_exceptions():
            for part in self._stream:
                yield part
 
    def close(self):
        if hasattr(self._stream, "close"):
            self._stream.close()
 
 
class URLLib3Transport(httpx.BaseTransport):
    def __init__(self):
        # self._pool = urllib3.PoolManager()
        self._pool = urllib3.connectionpool.ConnectionPool(host="localhost")
 
    def handle_request(self, request: httpx.Request):
        payload = json.loads(request.content.decode("utf-8").replace("'",'"'))
        urllib3_response = urllib3.request(
            method=request.method,
            url=str(request.url),
            headers=request.headers,
            json=payload,
            # stream=True,
            preload_content=False
        )
        headers = [(b"content-type", b"application/json")]
        return httpx.Response(
            200,
            headers=headers,
            stream=URLLib3Stream(urllib3_response.stream)
        )
 
client = httpx.Client(transport=URLLib3Transport())
 
from openai import OpenAI
from openai import AzureOpenAI
import openai
import os
 
os.environ['AZURE_OPENAI_API_KEY'] = "xxx"
openai_client = AzureOpenAI(
    api_version="2023-07-01-preview",
    azure_endpoint="https://xxx.openai.azure.com/",
    http_client=client
)
 
response = openai_client.chat.completions.with_raw_response.create(
    messages=[{
        "role": "user",
        "content": "sing me a song",
    }],
    model="gpt-35-turbo",
    max_tokens=30,
    temperature=0.7,
    # stream=True
)
 
completion = response.parse()
print(completion)

but seeing the output from this part of the code:

class URLLib3Stream(httpx.SyncByteStream):
    def __init__(self, stream):
        self._stream = stream

    def __iter__(self):
        print(self._stream.__self__.data)
        with httpx._transports.default.map_httpcore_exceptions():
            for part in self._stream:
                yield part

    def close(self):
        if hasattr(self._stream, "close"):
            self._stream.close()

Gives me three different responses for a single request:

b'{"id":"chatcmpl-8ucZ1UsLPxKy6SMYpE1JkJ4SsaSwT","object":"chat.completion","created":1708505299,"model":"gpt-35-turbo","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"I can\'t sing, but I can certainly help you find a song to listen to! What kind of music are you in the mood for?"},"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"usage":{"prompt_tokens":11,"completion_tokens":29,"total_tokens":40},"system_fingerprint":"fp_68a7d165bf"}\n'
b'{"id":"chatcmpl-8ucZ2tRwvhcyTgINUlrsZScHuWIXu","object":"chat.completion","created":1708505300,"model":"gpt-35-turbo","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"I\'m just a digital assistant, so I can\'t sing, but I can certainly help you find the lyrics to a song or recommend some good tunes"},"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"usage":{"prompt_tokens":11,"completion_tokens":30,"total_tokens":41},"system_fingerprint":"fp_68a7d165bf"}\n'
b'{"id":"chatcmpl-8ucZ5to0cm2ZVFGtIDK9RcvuN3iRL","object":"chat.completion","created":1708505303,"model":"gpt-35-turbo","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"choices":[{"finish_reason":"length","index":0,"message":{"role":"assistant","content":"I\'m just a virtual assistant, so I can\'t sing, but I can help you find the lyrics to a song or recommend some music for you"},"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}],"usage":{"prompt_tokens":11,"completion_tokens":30,"total_tokens":41},"system_fingerprint":"fp_68a7d165bf"}\n'

Not completely sure how to resolve this - because it's not working as intended.

[MRYingLEE]

@psymbio Have you found out the way? I am searching for the same solution.

[psymbio]

No, not really - I'm not sure why streaming isn't working here.

@tomchristie can you have a quick glance at this