ACE optimizer input/output confusion #433
Description
Hey,
I am trying to optimize a file data extraction program using AxACE optimizer, and it seems like the generated playbook doesn't differentiate between what is input and what is output.
I am not sure if it is a limitation in the source paper design or if there is a bug in the implementation, but the generated playbook isn't looking useful at all.
My signature looks like this
.input(
"inputDocument",
f.file("Input PDF document to analyze and extract data from.")
)
.output(
"legalWorks",
f.object({some Fields...}, "An array of the legal works found in the document.").array()
)
.description(DOCUMENT_SPLIT_PROMPT)
.build();And here are some of the playbook generated instructions:
{
"Source of Truth":[
{
"section":"Source of Truth",
"content":"...inspect the `Question context` provided in the input.\n- **Ground Truth Identification**: If the `Question context` contains a `legalWorks` array, this array is the **definitive source of truth**..."
},
{
"section":"Source of Truth",
"content":"CRITICAL: Source of Truth Check"
}
],
"CRITICAL: Source of Truth Check":[
{
"section":"CRITICAL: Source of Truth Check",
"content":"**IMMEDIATELY CHECK THE INPUT `Question context` BEFORE DOING ANYTHING ELSE.**\n1. **Look for a `legalWorks` array.**\n2. **If the `legalWorks` array EXISTS:**..."
}
],
"CRITICAL: Primary Workflow":[
{
"section":"CRITICAL: Primary Workflow",
"content":"**Step 1: Check the Input `Question context` BEFORE doing anything else.**\n\n**Step 2: Determine Your Path based on the input:**\n\n* **PATH A: If the `legalWorks` array EXISTS in the input:**..."
}
],
"Mandatory Workflow":[
{
"section":"Mandatory Workflow",
"content":"**MANDATORY FIRST STEP: Check the Input `Question context`**\n\nYour action is determined by the presence of a `legalWorks` array in the input..."
},
{
"section":"Mandatory Workflow",
"content":"**IMMEDIATELY CHECK THE INPUT `Question context` BEFORE DOING ANYTHING ELSE.** Your task is determined by the presence of a `legalWorks` array..."
},
{
"section":"Mandatory Workflow",
"content":"**MANDATORY FIRST STEP: Check the `Question context` input.**\n\nYour entire task is determined by whether a `legalWorks` array exists in the input..."
}
],
"Primary Workflow: Two Paths":[
{
"section":"Primary Workflow: Two Paths",
"content":"Your first and only mandatory action is to check the input `Question context`. Your task is determined by the presence or absence of a `legalWorks` array..."
}
],
"Path A: `legalWorks` array EXISTS in the input":[
{
"section":"Path A: `legalWorks` array EXISTS in the input",
"content":"If the `legalWorks` array is present in the input, it is the **definitive and only source of truth**..."
}
],
"Path B: `legalWorks` array does NOT exist in the input":[
{
"section":"Path B: `legalWorks` array does NOT exist in the input",
"content":"If and only if the `legalWorks` array is absent, proceed to extract the information from the `inputDocument`..."
}
]
}