Using PyRIT on OpenAI API

[reinbugnot]

Hello, I am from the Cybersecurity Laboratory of National University of Singapore. I am interested in using PyRIT for checking the robustness of LLM models.

I have the following LLM from my openai subscription:

target_llm = OpenAI(api_key=os.environ.get("OPENAI_API_KEY_NUS"))

May I ask how do I implement PyRIT on an LLM that does not come from AzureOpenAI?

I do not have an azureopenai subscription so I could not utilize the code available in doc/code/demo/.

import os

from pyrit.common import default_values
from pyrit.prompt_target import AzureOpenAIChatTarget
from pyrit.models import ChatMessage

default_values.load_default_env()

target_llm = AzureOpenAIChatTarget(
    deployment_name=os.environ.get("AZURE_OPENAI_CHAT_DEPLOYMENT"),
    endpoint=os.environ.get("AZURE_OPENAI_CHAT_ENDPOINT"),
    api_key=os.environ.get("AZURE_OPENAI_CHAT_KEY"),
)

prompt = "test"
target_llm.complete_chat(messages=[ChatMessage(content=prompt, role="user")])

Thank you for your assistance!

[romanlutz]

Are you using a release version of the latest main branch?

In the latest main branch there's OpenAITarget which should fit your use case (https://github.com/Azure/PyRIT/blob/main/pyrit/prompt_target/openai_chat_target.py)

[rdheekonda]

It seems you're utilizing AzureOpenAIChatTarget which is meant to interact with Azure OpenAI endpoints. If your intention is to use the OpenAI chat target, I recommend the following code. Please execute this code, and don't hesitate to contact us if you face any problems.

Before executing the code, make sure to configure the necessary environment variables.

import os

from pyrit.common import default_values
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.models import ChatMessage

default_values.load_default_env()

target_llm = OpenAIChatTarget(
    deployment_name=os.environ.get("OPENAI_CHAT_KEY"),
    endpoint=os.environ.get("OPENAI_CHAT_ENDPOINT"),
    api_key=os.environ.get("OPENAI_CHAT_DEPLOYMENT"),
)

prompt = "test"
target_llm.complete_chat(messages=[ChatMessage(content=prompt, role="user")])

[reinbugnot]

Thank you for your prompt responses!

I tried running the code provided, and it gave me the following error:

It seems as though my PyRIT cannot find the OpenAIChatTarget function from pyrit.prompt_target.

I installed PyRIT using pip install pyrit as instructed in doc/setup/install_pyrit.md. My current python version is Python 3.10.13

Is there an alternative installation/update method that I need to do to fix this?

Thank you for your help!

[romanlutz]

The pyrit version is the one that matters. If you follow the instructions in the "bug report" issue template you'll find this:
Please provide the following information:

• OS: [e.g. Windows]
• Browser (if you're reporting a bug in jupyter): [e.g. Edge, Firefox, Chrome, Safari]
• Python version: [e.g. 3.10.11]
• PyRIT version: [e.g. 0.1.0 or installed from main branch in editable mode]
• version of Python packages: please run the following snippet and paste the output:

  import pyrit
  pyrit.show_versions()

I suspect you're using v0.1.1 rather than 0.1.2. This is newly added there.

[reinbugnot]

Thank you for your response!

It does seem like my PyRIT version was outdated. I upgraded to PyRIT 0.1.2 and i think it's able to accept my OpenAI key.

However, I seem to be having trouble with specifying the correct API endpoint and deployment name. I did as follows:

import os
from pyrit.common import default_values
from pyrit.prompt_target import OpenAIChatTarget
from pyrit.models import ChatMessage

default_values.load_default_env()

target_llm = OpenAIChatTarget(
    api_key=os.environ.get("OPENAI_API_KEY_NUS"),
    deployment_name="test"
    endpoint="https://api.openai.com/v1/chat/completions"
)

prompt = "test"
target_llm.complete_chat(messages=[ChatMessage(content=prompt, role="user")])

Which resulted in the following error:

I suspect that I'm not constructing my API endpoint in the expected format. There does not seem to be any available documentation that I could follow regarding this.

May I kindly ask for your help once again about this matter?

Thank you

[reinbugnot]

I think I was able to fix the endpoint part, but my deployment_name input is still incorrect. Where do I get this information?

Thank you for your help and sorry for the noob questions!

[rdheekonda]

Based on the problem you're experiencing, the issue might be due to not having the "test" model listed in your OpenAI models, or it could be related to your OpenAI account's quota limit for accessing these models.

I recently set up a new OpenAI account with a free trial and encountered the same error when trying to access models that were not available to me.

Code calling "gpt-4" model from OpenAI

No "gpt-4" model available.

I suggest verifying that the "test" model is part of your OpenAI models collection. If it's not available, then I'd suggest taking the sample model available, such as gpt-3.5-turbo value (if it's available) from the "Model" dropdown menu for deployment_name.

If the above suggested solution doesn't resolve the issue, ensure that your account is correctly configured with enough credit to access these models. Access to these models is not included in the free tier.

[reinbugnot]

Thank you so much for your help! I was able to make it work on the basic gpt 3.5. It does seem that I had some issues setting up our model. Thank you.