Base prompt

[JesseVent]

Were any evals ran on the base prompt itself?

Saying it's solving a 'Supabase eval task' potentially has the following impacts:

• nudges it toward Supabase tools instead of generic ones
• leaks context a user would never provide
• each mode gets a differently worded prompt
• instructing the AI to end its turn with a short summary

Could shape how long it runs and when it stops which also could skew comparisons.

You can view the trace here https://supabase-evals-two.vercel.app/?trace=build-storage-001-private-bucket-access

evals/apps/framework/harness/run-eval.ts

Lines 309 to 322 in 0f19647

	function basePromptFor(mode: EvalMode): string {
	if (mode === 'local-stack') {
	return (
	'You are an agent solving a Supabase eval task in a Linux workspace. ' +
	'Use the provided tools to inspect and modify the workspace and run commands. ' +
	'When you are done, end your turn with a short summary of what you did.'
	);
	}
	return (
	'You are an agent solving a Supabase eval task. ' +
	'Use the provided tools to inspect and modify the project. ' +
	'When you are done, end your turn with a short summary of what you did ' +
	'(or for audit tasks, your findings).'
	);

Could be better to just leave it default.

You can view the trace here https://supabase-evals-two.vercel.app/?trace=build-storage-001-private-bucket-access

Trace from one of the failed ones #161