Prompt engineering on ChatGSE

[slobentanzer]

This is a discussion thread to share experiences with prompts and share successful (or ineffective) prompt set JSON files. Join the conversation!

[Wang-Cankun]

I really like your bioCypher and ChatGSE. I am also building a chat assistant to help do some single-cell analysis and interpretation. I think the current prompt can be improved in some ways. For example, adding the more specific role of both user and the assistant. Such as "You are an expert in some field". "GPT wants to imitate, not succussed so If we want to succeed, we need to ask it." This is an interesting psychological quirk in LLMs, mentioned at State of GPT by OpenAI.

I am a biomedical professor/biomedical student and I would like you to play the role of an expert who is good at contextualizing the user's findings with biomedical background knowledge.

Adding more examples in the prompt also known as few-shot:

Prompt 1:
Q: "We just ran a series of PCR tests on samples from our latest experiment on CRISPR gene-editing, and I'm seeing a lot of variation in the results. Could this be due to our primer design, or do you think it's more likely a problem with the PCR conditions themselves?"
A: "variations in PCR test results can be due to several factors. Primer design can indeed play a critical role, particularly if the primers are not specific enough or if they have a tendency to form secondary structures. PCR conditions are also vital; the annealing temperature, the quality of your template DNA, or the fidelity of your DNA polymerase could be influencing factors."
Prompt 2:
Q: "I have a list of gene expressions from our latest microarray experiment. Could you help me understand the significance of each of these genes in our current context?"
A: "the significance of these genes would greatly depend on the context of your experiment. For example, are these genes related to a specific pathway that you're studying, or are they linked to a particular disease or condition? Could you provide me more details about the specific genes and the context of your study? This would help me provide a more thorough and relevant analysis."

However, few shot prompting has limitations on reasoning. I think the chain of reasoning may be better at it, mentioned by OpenAI's GPT best practice and to Instruct the model to work out its own solution before rushing to a conclusion. However, this requires more $ to process.

[slobentanzer]

Hi @Wang-Cankun,
thanks a lot for your input. What's your tool's name, is there code available? I would be happy to see if we can find synergies in our projects. You may have seen that we are already working on a generic backend of LLM interaction that is independent of the UI solution, https://github.com/biocypher/biochatter.

Could you clarify something regarding the first comment you made about the roles: Does your (OpenAI's) advice take into account the different roles built into the chat API, i.e. System and User roles? AFAIK, the system role (which we use to say "you are an assistant to a biomedical researcher") should communicate with the model in second person, not first (e.g. "I am a biomedical professor"). We use system messages for the flattery component of the prompt (i.e., the primary model messages are all system messages). Only the questions from the human are encoded as user messages.

[Wang-Cankun]

My original comment was used in ChatGPT's use case. Regarding the system/user role used in OpenAI API, the example system role set the behavior of the assistant would be like "You are an expert in assistant biomedical research who is good at contextualizing the user's findings with biomedical background knowledge". The user role would be "I am a biomedical professor/biomedical student", from my experience setting to student context would guide GPT to explain more on the basis of terms, as different types users expect different levels of responses.

[slobentanzer]

I see now, good point and makes sense to guide GPT into its role. We don't currently have user role flattery components, but that could easily be extended.

In terms of empirically validating the performance of these prompt sets, we are also working on building benchmark datasets into the biochatter library to be able to compare the prompt sets and maybe even have a leaderboard at some point. I'll make this visible in the repo and app as soon as we have it.