[JS] definePrompt returns undefined outputSchema

[ariel-pettyjohn]

Describe the bug
definePrompt returns an undefined outputSchema even when a valid outputSchema is provided.

To Reproduce
I'm trying to run the below test code but I keep getting this error: Uncaught (in promise) Error: No JSON object or array found in model output. I tried passing both outputSchema and output to definePrompt to test (just in case).

Code:

export async function runTestFlow({ property }) {
    const inputSchema  = z.object({ property: z.string() });
    const outputSchema = z.object({ type    : z.string() });

    const flow = defineFlow({
        name        : "testFlow",
        inputSchema : inputSchema,
        outputSchema: outputSchema,
    }, async ({ property }) => {        
        const prompt = definePrompt({
            name        : "testPrompt",
            inputSchema : inputSchema,
            outputSchema: outputSchema,
            output      : { schema: outputSchema }
        }, async ({ property }) => ({
            messages: [{ 
                role   : "user",
                content: [{ 
                    text: `What type is the ${property} property on a color?` 
                }] 
            }],
            config: { temperature: 0.3 }
        }));
        console.log("PROMPT:", prompt);
        const response = await generate(
            renderPrompt({
                prompt: prompt,
                input : { property },
                model : gemini15Flash,
            })
        );
        return response.output();
    });

    return await runFlow(flow, { property });
}

When I inspect the logged prompt object the outputSchema is undefined:

[Function: actionFn] {
  __action: {
    name: 'testPrompt',
    description: undefined,
    inputSchema: ZodObject {
      spa: [Function: bound safeParseAsync] AsyncFunction,
      _def: [Object],
      parse: [Function: bound parse],
      safeParse: [Function: bound safeParse],
      parseAsync: [Function: bound parseAsync] AsyncFunction,
      safeParseAsync: [Function: bound safeParseAsync] AsyncFunction,
      refine: [Function: bound refine],
      refinement: [Function: bound refinement],
      superRefine: [Function: bound superRefine],
      optional: [Function: bound optional],
      nullable: [Function: bound nullable],
      nullish: [Function: bound nullish],
      array: [Function: bound array],
      promise: [Function: bound promise],
      or: [Function: bound or],
      and: [Function: bound and],
      transform: [Function: bound transform],
      brand: [Function: bound brand],
      default: [Function: bound default],
      catch: [Function: bound catch],
      describe: [Function: bound describe],
      pipe: [Function: bound pipe],
      readonly: [Function: bound readonly],
      isNullable: [Function: bound isNullable],
      isOptional: [Function: bound isOptional],
      _cached: [Object],
      nonstrict: [Function: passthrough],
      augment: [Function: extend]
    },
    inputJsonSchema: undefined,
    outputSchema: undefined,
    outputJsonSchema: undefined,
    metadata: { prompt: {}, type: 'prompt' },
    actionType: 'prompt'
  }
}

Expected behavior
If I run the below code instead, with the property rgb, I get the expected response:

{
    "type": "array"
}

Code:

export async function runTestFlow({ property }) {
    const inputSchema  = z.object({ property: z.string() });
    const outputSchema = z.object({ type    : z.string() });

    const flow = defineFlow({
        name        : "testFlow",
        inputSchema : inputSchema,
        outputSchema: outputSchema,
    }, async ({ property }) => {       
        const response = await generate({
            prompt: `What type is the ${property} property on a color?`,
            model : gemini15Flash,
            output: { schema: outputSchema }
        });
        return response.output();
    });

    return await runFlow(flow, { property });
}

Runtime

• OS: Windows 11
• Home Edition

Node version

• 20.12.2

Additional context
Note that I tried providing an outputSchema to renderPrompt as well, with the same result. I also tried awaiting renderPrompt and logged the result, but it doesn't look like that's where the outputSchema is designed to be provided. So, I'm presuming it should be provided to definePrompt instead, which does seem more intuitive.

Thanks, and please let me know if I can provide additional context!

[pavelgj]

This is WAI -- definePrompt doesn't actually allow specifying the output schema: https://github.com/firebase/genkit/blob/main/js/ai/src/prompt.ts#L58-L62

The two options is either using the generate function (as you did) or using dotprompt: https://firebase.google.com/docs/genkit/dotprompt.md#defining_inputoutput_schemas

import { defineDotprompt } from '@genkit-ai/dotprompt';

const prompt = defineDotprompt({
  name: "testPrompt",
  input: {
    schema: inputSchema
  },
  output: {
    format: 'json',
    schema: outputSchema,
  },
}, `What type is the  {{property}} property on a color?` );

[ariel-pettyjohn]

Ah-ha, that defineDotPrompt example is perfect for my use-case, thanks @pavelgj 💯 I didn't notice it in the docs before, but I see it now!

For context: I'm working on an object-oriented wrapper around Genkit to complement an ODM that I already built around Firestore, so I'm trying to avoid forcing developers to write Dotprompt files in case they'd rather keep everything defined in-code. defineDotPrompt looks like it should give me the best of both worlds 👍

Thanks again!

Edit: I almost forgot to ask, will defineDotPrompt also register the prompt with Genkit like definePrompt, as mentioned in the docs? I presume it does since we're assigning it a name. @pavelgj

[MichaelDoyle]

Yes, exactly, defineDotPrompt will register the prompt with Genkit!