Prompt engineering

Today's agenda

Here is my suggested agenda for today. Thiago and I are happy to modify it after your feedback at the start of our one hour session today:

(1) Allow everyone to make brief introductions, identifying learning goals and specific use cases we want to work on over the next eight weeks

(2) Make sure everyone is registered with the MarineLives Anthropic organisation and now has access to the ai-and-history-collaboratory workspace.

(3) Show everyone how to use the Anthropic console to write and improve prompts

(4) Give a very brief overview of the collaboratory GitHub repository and wiki

(5) OUR FOCUS TODAY: To work through the material on the prompt engineering page in a very interactive way, soliciting comment and additional narrative and analytical prompts from collaboratory members, and encouraging everyone to explore a wider range of prompt types, which we can subsequently document in the wiki

---

Typology of prompts

• Information Retrieval Prompts: To extract specific information such as dates, names, places, core concepts

• Text Generation Prompts: To create abstracts, analyses, narratives, summaries, reports

• Code Generation Prompts: To generate code for tasks like data analysis, data linkage, or data visualization.

• Reasoning Prompts: To interrogate and reason about connectivity, causality, sequencing

Can your help us improve this typology of prompts?

---

Diction, syntax, style, and rhetoric

• Prompts are essentially instructions: Just like any form of communication, the way you construct a prompt influences how it's received and interpreted. Clear, concise, and well-structured prompts lead to better results from the AI.

• Diction matters: The specific words you choose in a prompt can significantly impact the AI's response. For example, asking the AI to "describe" an event will yield a different result than asking it to "analyze" or "evaluate" it. Historians are trained to be sensitive to the nuances of language, and this skill is crucial in prompt engineering.

• Syntax shapes the response: The grammatical structure of your prompt guides the AI's understanding. Using complete sentences, proper punctuation, and clear phrasing helps the AI grasp the intended meaning and generate a more coherent and relevant response.

• Style influences the output: Just as there are different styles of historical writing, there are different styles of prompting. A formal and precise prompt might be appropriate for factual information retrieval, while a more creative and open-ended prompt might be better for generating imaginative narratives.

• Rhetoric adds layers of meaning: Rhetorical devices like metaphors, analogies, and rhetorical questions can be used in prompts to guide the AI's reasoning and elicit more nuanced responses.

We would love to hear your own take on this topic.

---

Use case: narrative summarization

Colin Greenstreet, co-founder of MarineLives and convenor of the ai-and-history-collaboratory, is working with a third year undergraduate at the University of York, Abi Cunningham. Abi approached MarineLives three weeks ago to volunteer to assist with machine transcription.

• In one week, Abi went from no prior experience of machine transcription and LLM-based summarization, to writing and improving this prompt to generate on topic narrative summarization of English High Court of Admiralty depositions.

• Two weeks later she developed a further prompt to identify the start and end of depositions within a large text block using contextual probablistic logic together with narrative summarization.

Here is our challenge to collaboratory members in preparation for our first session of the collaboratory on Tuesday, November 26th, 2024:

• Take a primary manuscript source you are working with.

• Run it through a machine transcription service (like Transkribus, or Tesseract). If you don't have a Transkribus account, get in touch with Colin Greenstreet and you can load your document into his Transkribus account, and we will upload the manuscript images and run it together with a suitable model.

• Then, chose a frontier model of your choice - GPT-4o, Gemini, Claude Sonnet 3.5, or another provider like Meta or Mistral.

• Write a prompt to summarize your document, or parts of your document.

• Play with your prompt. Try to improve it. Tinker with it.

Bring your experience (and the full text of the prompt and its output) to our first session.

---

Use case: analytical ontological summarization

Let's take a look at a complex nested prompt designed to create analytical ontological summaries, rather than narrative summaries

• Try it out on some new inputs from the English High Court of Admiralty

• You can access sample depositions from the English High Court of Admiralty here

• How does this analytical ontological prompt differ from Abi's narrative summarization prompt?

• What techniques do the two prompts use?

• Try asking a frontiier LLM (Claude, GPT-4, or Gemini) what techniques the two prompts use.

• Try asking the same LLMs how the prompts mght be improved.

Come to the first session of our collaboratory prepared to discuss your findings and your own views.

---

Use case: batch processing of raw HTR for clean up and summarization

Now let's look at a different type of prompt, which combines two tasks. Task one is the clean up of raw machine transcription (HTR). Task two is the summarization of the cleaned up raw machine transcription. Here is the prompt.

Try pasting the prompt into a frontier LLM of your choice and asking it:

• What is this prompt designed to do?

• How well does the prompt achieve its probable aims?

• How does the prompt approach batch processing?

• Is there a maximum size to the batches of input data which can be handled with one prompt?

Come to the first session of our collaboratory prepared to discuss your findings and your own views.

---

History prompt library

History prompt library

This link takes you to an EMPTY PAGE full of place holders by prompt type.

Our goal, as a collaboratory, is to fill this History Prompt Library with prompts for real live use cases each of us is working on. It is our collective contribution to the Commons.

---

Useful prompt types for historical research

• Analyze

• Categorize

• Contextualize

• Expand

• Extract

• Geotag

• Interrogate

• Link

• Map

• Modernize

• Role play

• Simplify

• Structure

• Summarize

• Translate

---

Anthropic prompt engineering guide

Prompt generator

Be clear and direct

Use examples (multishot)

Let Claude think (chain of thought)

Use XML tags

Give Claude a role (system prompts)

Prefill Claude’s response

Chain complex prompts

Long context tips

---