Skip to content

[Tech Debt] Prompt SafeFormatter brace handling is undocumented and inconsistent with Python f-string convention #781

@Sxnan

Description

@Sxnan

Search before asking

  • I searched in the issues and found nothing similar.

Description

Found during YAML doc verification (#742) while running an end-to-end agent
with a real LLM.

SafeFormatter in python/flink_agents/api/prompts/utils.py:22-41 uses the
regex \{([^{}]+)\} to substitute {key} placeholders, and offers no
escape syntax for emitting a literal { or }:

  • {name} → substituted with the value of name
  • {{, }}, {{name}} → all preserved as-is in the output

Python f-strings and str.format use {{ and }} for literal { and }.
A user porting prompt content from f-string-shaped examples to a Flink
Agents prompt template will reasonably double the braces in JSON examples
to "escape" them — and find that the LLM sees literal {{ and }} in the
prompt, then sometimes mimics the doubled-brace form in its response,
breaking downstream JSON parsing.

This was hit during end-to-end YAML verification: a system prompt with
double-brace JSON examples occasionally produced LLM responses with
{{ ... }} instead of { ... }, which then failed json.loads in the
downstream action and silently dropped the output (related: #780).

The framework's behavior is reasonable from an implementation perspective,
but it is undocumented and diverges from a common convention.

Suggested fix

Pick one:

  1. Document only — add a short note to
    docs/content/docs/development/prompts.md (and a cross-reference from
    yaml.md's prompt section) stating:

    • {name} is the only substitution syntax
    • { and } characters that are not placeholders pass through verbatim
    • there is no {{ / }} escape — if you need a literal { or }, write
      it as-is
    • reminder that LLM examples inside the prompt should use single braces
  2. Document + code — extend SafeFormatter (and the Java equivalent in
    Prompt.formatMessages) to recognize {{{ and }}}, matching
    Python f-string / str.format conventions, then update the doc
    accordingly. This is a behavior change; consider a deprecation cycle if
    anyone might depend on {{ being literal today.

How to reproduce

In a YAML prompt, write a JSON example with doubled braces (a natural copy
from Python f-string examples):

prompts:
  - name: p
    messages:
      - role: system
        content: |
          Respond as JSON like:
          {{
            "result": "ok"
          }}
      - role: user
        content: "{input}"

The rendered prompt the LLM sees contains literal {{ and }}. The LLM
may then mimic the doubled-brace form in its response.

Version and environment

Flink Agents 0.3.0 (main). Python SafeFormatter; the Java equivalent
Prompt.formatMessages should be checked for parity.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

Labels

fixVersion/0.3.0The feature or bug should be implemented/fixed in the 0.3.0 version.priority/blockerIndicates the PR or issue that should block the release until it gets resolved.tech debt[Issue Type] User-unaware issues, such as code refactor and infrastructure maintenance.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions