When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of `":"` as json properties #94

sgsdxzy · 2024-04-29T18:01:37Z

The json template to send as "guided_json" in the request to oai api of vLLM/Aphrodite:

json_template = {
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "tool_name": {"type": "string"},
            "parameters": {
                "type": "object",
                "additionalProperties": {
                    "anyOf": [{"type": "string"}, {"type": "number"}, {"type": "boolean"}]
                },
            },
        },
        "required": ["tool_name", "parameters"],
        "additionalProperties": {},
    },
    "minItems": 1,
    "maxItems": 1,
}

Model output without any constrant

[
    {
        "tool_name": "internet_search",
        "parameters": {
            "query": "biggest penguin species",
            "provider": "Google"
        }
    }
]

Model output with "guided_json" and "guided_decoding_backend": "lm-format-enforcer"

[
    {
        "tool_name": "internet_search",
        "parameters": {
           ":": "biggest penguin species world"
        }
    }
]

Model output with "guided_json" and "guided_decoding_backend": "outlines"

[
    {
        "tool_name": "internet_search",
        "parameters": {
            "query": "biggest penguin species",
            "provider": "Google"
        }
    }
]

The model: CohereForAI/c4ai-command-r-plus
The prompt: (adapted from the function calling example of c4ai-command-r-plus)

The text was updated successfully, but these errors were encountered:

noamgat · 2024-04-30T05:47:58Z

How are you running this? Through the latest vLLM image? The latest vLLM version does not yet contain a few JSON Schema bug fixes that were made in the past weeks. (The PR was approved, but not in 0.4.1). Can you make sure you are on the latest version?

…

On Mon, Apr 29, 2024 at 9:01 PM sgsdxzy ***@***.***> wrote: The json template to send as "guided_json" in the request to oai api of vLLM/Aphrodite: json_template = { "type": "array", "items": { "type": "object", "properties": { "tool_name": {"type": "string"}, "parameters": { "type": "object", "additionalProperties": { "anyOf": [{"type": "string"}, {"type": "number"}, {"type": "boolean"}] }, }, }, "required": ["tool_name", "parameters"], "additionalProperties": {}, }, "minItems": 1, "maxItems": 1, } Model output without any constrant [ { "tool_name": "internet_search", "parameters": { "query": "biggest penguin species", "provider": "Google" } } ] Model output with "guided_json" and "guided_decoding_backend": "lm-format-enforcer" [ { "tool_name": "internet_search", "parameters": { ":": "biggest penguin species world" } } ] Model output with "guided_json" and "guided_decoding_backend": "outlines" [ { "tool_name": "internet_search", "parameters": { "query": "biggest penguin species", "provider": "Google" } } ] The model: CohereForAI/c4ai-command-r-plus The prompt <https://pastebin.com/rtx2PcVv>: (adapted from the function calling example of c4ai-command-r-plus) — Reply to this email directly, view it on GitHub <#94>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKFA2FKG5GD7CUV2OUYQH3Y72DJPAVCNFSM6AAAAABG6ZKI7OVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DSNRTGQ2TSNI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

sgsdxzy · 2024-04-30T07:45:27Z

Yeah I was using the release version of 0.4.1, will try latest, thanks. Could you point me to the related vLLM PR?

sgsdxzy · 2024-04-30T11:33:24Z

I am using vllm main with lmfe==0.9.8, and during the same request I encountered:

ERROR:root:Unknown LMFormatEnforcer Problem. Prefix: '[
    {
        "tool_name": "internet_search",
        "parameters": {
           "hquery": "biggest penguin in the world",
           "hprovider": "Google"
        }
'
Terminating the parser. Please open an issue at
https://github.com/noamgat/lm-format-enforcer/issues with the prefix and CharacterLevelParser parameters
Traceback (most recent call last):
  File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 96, in _compute_allowed_tokens
    self._collect_allowed_tokens(state.parser, self.tokenizer_tree.root, allowed_tokens, shortcut_key)
  File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 144, in _collect_allowed_tokens
    self._collect_allowed_tokens(next_parser, next_tree_node, allowed_tokens, None)
  File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 144, in _collect_allowed_tokens
    self._collect_allowed_tokens(next_parser, next_tree_node, allowed_tokens, None)
  File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 142, in _collect_allowed_tokens
    next_parser = parser.add_character(character)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/jsonschemaparser.py", line 63, in add_character
    while new_character not in self.object_stack[receiving_idx].get_allowed_characters():
                               ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
IndexError: list index out of range

Any idea why?

noamgat · 2024-04-30T11:47:58Z

Interesting. This looks closer to a good completion so the version upgrade fixed something, but this looks like a new bug. I hope to get to it in the coming days. It would be best if you created a unit test that reproduces the problem, but if not I will try based on the information you provided here. My guess is that its related to the fact that the top-level object is an array, which should be supported but is less battle tested than the top level object being a dict.

…

On Tue, Apr 30, 2024 at 2:33 PM sgsdxzy ***@***.***> wrote: I am using vllm main with lmfe==0.9.8, and during the same request I encountered: ERROR:root:Unknown LMFormatEnforcer Problem. Prefix: '[ { "tool_name": "internet_search", "parameters": { "hquery": "biggest penguin in the world", "hprovider": "Google" } ' Terminating the parser. Please open an issue athttps://github.com/noamgat/lm-format-enforcer/issues with the prefix and CharacterLevelParser parameters Traceback (most recent call last): File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 96, in _compute_allowed_tokens self._collect_allowed_tokens(state.parser, self.tokenizer_tree.root, allowed_tokens, shortcut_key) File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 144, in _collect_allowed_tokens self._collect_allowed_tokens(next_parser, next_tree_node, allowed_tokens, None) File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 144, in _collect_allowed_tokens self._collect_allowed_tokens(next_parser, next_tree_node, allowed_tokens, None) File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py", line 142, in _collect_allowed_tokens next_parser = parser.add_character(character) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/sgsdxzy/micromamba/envs/vllm/lib/python3.11/site-packages/lmformatenforcer/jsonschemaparser.py", line 63, in add_character while new_character not in self.object_stack[receiving_idx].get_allowed_characters(): ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^ IndexError: list index out of range Any idea why? — Reply to this email directly, view it on GitHub <#94 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKFA2GKOIUY56KPUL3FBC3Y756RVAVCNFSM6AAAAABG6ZKI7OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGA2TOOJRGM> . You are receiving this because you commented.Message ID: ***@***.***>

sgsdxzy · 2024-04-30T12:22:49Z

Yes change the outmost array to object works fine.

sgsdxzy · 2024-04-30T18:08:25Z

I might be having some lucky draw. If I fix the random number seed I am still getting

[
    {
        "tool_name": "internet_search",
        "parameters": {
           ":": "biggest penguin species world"
        }
    }
]

as response as of lmfe 0.9.8, alongside the Unknown LMFormatEnforcer Problem. Error above.

noamgat · 2024-05-01T19:42:47Z

Thanks! I hope to look at this in the coming days.

noamgat · 2024-05-03T12:43:24Z

I just released 0.9.10 with a fix that should remove the Unknown LMFormatEnforcer Problem issue. However, I'm not sure it will solve your problem, as the last response you posted conforms to the json schema you posted (obviously its not a good one, but thats not LMFE's job).

sgsdxzy · 2024-05-04T05:53:25Z

I can confirm 0.9.10 fixed the ERROR:root:Unknown LMFormatEnforcer Problem.
However I think lmfe gives the LLM wrong logits that prevents some valid responses from generation. It made conforming to the schema necessary but insufficient: It does enforce the json schema so all generations are valid, but not all valid generations are allowed.

[
    {
        "tool_name": "internet_search",
        "parameters": {
            "query": "biggest penguin species",
            "provider": "Google"
        }
    }
]

also conforms to the schema and should be a much more likely generation (it will be generated without any contraint by the LLM, and by outlines) but prevented by lmfe somehow.

noamgat · 2024-05-04T06:35:52Z

I can confirm 0.9.10 fixed the ERROR:root:Unknown LMFormatEnforcer Problem. However I think lmfe gives the LLM wrong logits that prevents some valid responses from generation. It made conforming to the schema necessary but insufficient: It does enforce the json schema so all generations are valid, but not all valid generations are allowed.
[
    {
        "tool_name": "internet_search",
        "parameters": {
            "query": "biggest penguin species",
            "provider": "Google"
        }
    }
]
also conforms to the schema and should be a much more likely generation (it will be generated without any contraint by the LLM, and by outlines) but prevented by lmfe somehow.

I now see the problem - this language model prefers to use spaces instead of tab, and uses indentation of length 4. This causes the completion to contain 13 consecutive whitespaces (newline + 12). LMFE has a heuristic const MAX_CONSECUTIVE_WHITESPACES=12 in order to avoid infinite whitespace loops (Which are legal jsons, but probably unwanted). If you are able to test this, can you try increasing this constant and trying again?

One of the upcoming features I plan to add is to allow environment variables to modify some of these configurations, to make it easier to change some of LMFE's heuristics in non-code environments (such as vLLM OpenAI server)

sgsdxzy · 2024-05-04T13:09:41Z

I can confirm setting MAX_CONSECUTIVE_WHITESPACES in consts.py to some larger value completely fixes this.
Yes making it configurable through environment variables is a better solution to this.

noamgat · 2024-05-04T13:57:11Z

#97 - Coming very soon :)

noamgat · 2024-05-04T14:52:30Z

Released in v0.10.1. Can you check if you can now solve the problem via Configuration Options?

sgsdxzy · 2024-05-04T15:26:15Z

v0.10.1 fixes this issue.

sgsdxzy mentioned this issue May 4, 2024

Fix minor bugs in outlines and lmfe. PygmalionAI/aphrodite-engine#449

Merged

sgsdxzy closed this as completed May 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of `":"` as json properties #94

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of `":"` as json properties #94

sgsdxzy commented Apr 29, 2024

noamgat commented Apr 30, 2024 via email

sgsdxzy commented Apr 30, 2024 •

edited

sgsdxzy commented Apr 30, 2024

noamgat commented Apr 30, 2024 via email

sgsdxzy commented Apr 30, 2024

sgsdxzy commented Apr 30, 2024

noamgat commented May 1, 2024

noamgat commented May 3, 2024

sgsdxzy commented May 4, 2024

noamgat commented May 4, 2024

sgsdxzy commented May 4, 2024

noamgat commented May 4, 2024 •

edited

noamgat commented May 4, 2024

sgsdxzy commented May 4, 2024

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of ":" as json properties #94

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of ":" as json properties #94

Comments

sgsdxzy commented Apr 29, 2024

noamgat commented Apr 30, 2024 via email

sgsdxzy commented Apr 30, 2024 • edited

sgsdxzy commented Apr 30, 2024

noamgat commented Apr 30, 2024 via email

sgsdxzy commented Apr 30, 2024

sgsdxzy commented Apr 30, 2024

noamgat commented May 1, 2024

noamgat commented May 3, 2024

sgsdxzy commented May 4, 2024

noamgat commented May 4, 2024

sgsdxzy commented May 4, 2024

noamgat commented May 4, 2024 • edited

noamgat commented May 4, 2024

sgsdxzy commented May 4, 2024

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of `":"` as json properties #94

When using json schema in vLLM/Aphrodite-engine, lmfe generates a lot of `":"` as json properties #94

sgsdxzy commented Apr 30, 2024 •

edited

noamgat commented May 4, 2024 •

edited