Bugs When using OpenAI api with Celery/redit

[jfrancoisponcet16]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

Hello,
There seem to be a bug when using the Openai API with Celery/redit shared task. When running the function bellow.
I’m not sure exactly why. When In use langchain with Mistral api, Ollama or with any my own custom models it works really well, but it crashes when I use anything touching the openai api. I tried bypassing langchain to work directly with the openai api and I still do get this error Does someone have any idea why ?

To Reproduce

1. pip install Django
2. pip install celery
3. brew install redit // for macs
4. create a random view function that will call the task at task.py

Code snippets

in views.py

def some_view_fucntion(request):
    for x in range(2):
         some_ai_function.delay()

in task.py

@shared_task
def some_ai_function():
    message = "Hello, write a message about testing "
    print(message)
    completion = openai.chat.completions.create(
        model=“gpt-3.5-turbo”,
        messages=[
            {
                 “role”: “user”,
                 “content”: message,
            },
        ],
    )
    print(completion.choices[0].message.content)

RETURNS :

ERROR/MainProcess] Process ‘ForkPoolWorker-2’ pid:86900 exited with ‘signal 5 (SIGTRAP)’
ERROR/MainProcess] Task handler raised error: WorkerLostError(‘Worker exited prematurely: signal 5 (SIGTRAP) Job: 0.’)

Traceback (most recent call last):
File" …
/python3.11/site-packages/billiard/pool.py", line 1264, in mark_as_worker_lost
raise WorkerLostError(
billiard.exceptions.WorkerLostError: Worker exited prematurely: signal 5 (SIGTRAP) Job: 0.

OS

macOS (m2)

Python version

Python 3.11.5

Library version

openai 1.8.0

### Tasks
- [ ] Create a work around get_architecture()  for celery tasks on MacOS

[jfrancoisponcet16]

I got where the bug is. When you are using using a macOS with redis (without docker your application), Mac blocks Celery from accessing platform.architecture() which causes the function to fail. For few good reasons I did not yet dockerised my application and I encountered the problem. Found a workaround but it's not ideal. Is there a way that in the new version of the openai api _based_client.py throuwould be workaround of get_architecture() ?
Best,

[rattrayalex]

Ah, yes, I understand. We'll ensure those aspects can fail safely. Thank you for the report.

[zyz954489346]

I have same BUG on my Mac M1X.
This has a great impact on my local testing, including chating with GPT through celery.
Is in urgent need of repair.

Ah, yes, I understand. We'll ensure those aspects can fail safely. Thank you for the report.啊，是的，我明白了。我们将确保这些方面能够安全地失败。感谢您的报告。

[jfrancoisponcet16]

Hi @zyz954489346
Yes I can see they have change the code but it cannot fail safely with (try/except) due to the fact that get_achitecture() does not lead only to a python error.
Anyway a quick workaroud you can do is go on your site-packages>openai>_base_client.py>get_achritecture() and just set it to the following :

def get_architecture() -> Arch:
return "arm64"

comment the reste of the function:
# try:
# python_bitness, _ = platform.architecture()
# machine = platform.machine().lower()
# except : (etc...)

Best,
Jean FP