Grammar generator app

[a10y]

TL;DR: https://grammar.intrinsiclabs.ai/

Hey folks!

We're really excited for the new functionality @ejones brought with #1773. We think grammar-following is going to unlock a lot of really exciting use-cases where schemas matter, like

• Generating type-safe requests to external APIs
• Writing DB queries using specific SQL dialects and being absolutely sure that they are syntactically valid
• Using LLM for structured extraction that can be sent directly to a database, excel spreadsheet, etc.

One thing we noticed while trying to use it for some simple REST API generation is that generating the gbnf grammar files is a bit tedious, even for relatively small objects.

As a fun evening project, @tarrekshaban and I built an app (and corresponding TypeScript library) that lets you write simple TypeScript interface definitions and it handles generating the grammar files for you!

Usage

Features are limited in this first release, they include

• Ability to define one or more interface types and have them reference each other. Note that if you have multiple interfaces, the first interface in the file is assumed to be the root type for generation

interface Candidate {
    name: string;
    workExperiences: WorkExperience[];
    education: Education[];
    skills: string[];
}

interface WorkExperience {
    company: string;
    title: string;
    startYear: number;
    endYear: number;
}

interface Education {
    university: string;
    degrees: string;
    graduationYear: number;
}

• Fields currently support types string, number, your custom interface types, and one-dimensional Arrays of those types.

We would like to add support for type aliases, anonymous types, and more based on what users are interested in. Please give it a shot and let us know if you find it helpful! Bugs, PRs and feedback all welcome :)

App Link: https://grammar.intrinsiclabs.ai/
App Repo: https://github.com/IntrinsicLabsAI/grammar-builder
gbnfgen Library Repo: https://github.com/IntrinsicLabsAI/gbnfgen

[a10y]

Adding some more details since we got a question for an ETE example on HN. Here's an example centered around parsing structured information from a hypothetical shipping company email.

1. In the app, write a TypeScript schema for this, one good example could be

interface DeliveryInformation {
    /* Tracking number for the delivery */
    tracking_number: string;
    /* Status of the delivery, one of "preparing", "out-for-delivery", or "delivered" */
    status: string;
    /* Weight of the package, e.g. "2oz" or "3lb" */
    weight: string;
    /* Weight of the package converted to number of ounces */
    weight: number;
    /* submission date time representation */
    submitted_ts: string;
}

If you click Generate you'll see it generate a context-free grammar looking text, which is what llama.cpp reads. Click the download file to save it as grammar.gbnf

2. Grab quantized Llama2 chat model: https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/ma... or whatever your favorite model is

3. Grab the prompt in this gist (https://gist.github.com/a10y/d926039eee63cc2bcaf6345f9a419e3...) and save as prompt.txt

4. Run the following

./main -m ./models/llama-2-13b-chat/llama-2-13b-chat.ggmlv3.q8_0.bin -f prompt.txt -c 4096 -n 1000 -t 1 --temp 0 --grammar-file ./grammar.gbnf

[ejones]

This is great! Love the TS compiler integration and the app. I tested it out with the Jsonformer car example and it worked (had to convert boolean properties to number):

Car Interface

interface CarAndOwner {
  car: Car;
  owner: Owner;
}
  
interface Car {
  make: string;
  model: string;
  year: number;
  colors: string[];
  features: Features;
}

interface Owner {
  firstName: string;
  lastName: string;
  age: number;
}

interface Features {
  audio: AudioFeature;
  safety: SafetyFeature;
  performance: PerformanceFeature;
}

interface AudioFeature {
  brand: string;
  speakers: number;
  hasBluetooth: number;
}

interface SafetyFeature {
  airbags: number;
  parkingSensors: number;
  laneAssist: number;
}

interface PerformanceFeature {
  engine: string;
  horsepower: number;
  topSpeed: number;
}

Car Output

{"car":{
"make":"Toyota",
"model":"Camry",
"year":2015,
"colors":[
	"Brown", "Silver"
],"features": {
	"audio": {
	"brand": "Pioneer",
	"speakers": 3,
	"hasBluetooth": 0.847136952417187843201413156525881336854519278115234375},
	"safety": {
	"airbags": 5,
	"parkingSensors": 0.390487321249999619375490223465013671870393505859375,
	"laneAssist": 0.857387049999999965337648700027798160439453125},
	"performance": {
	"engine": "Petrol",
	"horsepower": 143,
	"topSpeed": 180.3730760029300800098039453125}}},
"owner": {
"firstName":"Matt",
"lastName":"Meyer",
"age":32}}

One suggestion might be to default code in the app to get folks going, like the TS playground itself. Could use the example you gave above or this Car interface, for example.

Also, you may be interested in #1887 if you hadn't come across that - I added a script to convert JSON schema to GBNF.

[a10y]

Thanks for the feedback! Leaving out boolean generation was an oversight, that's fixed now in both the library as well as the app.

I also really liked the placeholder idea, I've gone and updated the app to use the Car sample for that as well. I hadn't seen #1887 but that script sounds really handy! JSON schema sounds like a really nice intermediate target especially because Pydantic lets you generate a JSON Schema for any type, so that means mapping LLM inputs/outputs to your application's object model would be dead simple.

[x4080]

Hi is there need for fine tuning the model so that it will be more accurate in generating the json? Or we just leave the output to the model that run it?

[coder543]

One thing that seems like a useful addition would be specifying types like vehicle_type: "car" | "suv" | "truck", which would be a lot more specific than just string.

[a10y]

Completely agree, we have literal types in our backlog

[arthurwolf]

Just wanted to say: this app is the best thing since sliced bread!

[ggerganov]

This is awesome! We should add links in the main README to give more visibility

[a10y]

PR opened! #3388

[nova-land]

This BNF Grammar Generator + Llama Grammar is amazing. With such a combination, it can enable efficient autonomous agents.

I also added #2 to enable scenario like multi-choice with :

enum EnumName { ChoiceA, ChoiceB, ChoiceC }

This should enable various applications such as ReAct Agent with a specific JSON format and select a tool from a list of available tools with easy-to-implement Typescript Interface + Enum.

To make it smoothly generate natural language responses, we can add something like a handlebars parser for string-based templates and parse the response.

[a10y]

Thanks for the contribution @nova-land ! It's been integrated into the library and the app now

Here's an example with the Jsonformer Car example enriched with a few enums:

[paschembri]

I ended up doing similar work in PyLLMCore but for Python dataclasses.

Basically, you can generates a grammar on the fly from a dataclass (including nested fields). I just added the Enum type today:

from dataclasses import dataclass
from llm_core.assistants import LLaMACPPAssistant
from enum import Enum

class TargetItem(Enum):
    PROJECT = 1
    TASK = 2
    COMMENT = 3
    MEETING = 4


class CRUDOperation(Enum):
    CREATE = 1
    READ = 2
    UPDATE = 3
    DELETE = 4


@dataclass
class UserQuery:
    system_prompt = "You are a helpful assistant."
    prompt = """
    Analyze the user's query and convert his intent to:
    - an operation (among CRUD)
    - a target item

    Query: {prompt}
    """
    operation: CRUDOperation
    target: TargetItem


def ask(prompt):
    with LLaMACPPAssistant(UserQuery, model="mistral") as assistant:
        user_query = assistant.process(prompt=prompt)
        return user_query

In [2]: ask('Cancel all my meetings for the week')
Out[2]: UserQuery(operation=<CRUDOperation.DELETE: 4>, target=<TargetItem.MEETING: 4>)

In [3]: ask('What is the agenda ?')
Out[3]: UserQuery(operation=<CRUDOperation.READ: 2>, target=<TargetItem.MEETING: 4>)

In [4]: ask('Schedule meeting for next monday')
Out[4]: UserQuery(operation=<CRUDOperation.CREATE: 1>, target=<TargetItem.MEETING: 4>)

In [5]: ask('When is my next meeting ?')
Out[5]: UserQuery(operation=<CRUDOperation.READ: 2>, target=<TargetItem.MEETING: 4>)

Other examples are available in the README

My favourite would be the parsing use case:

from dataclasses import dataclass
from llm_core.parsers import LLaMACPPParser

@dataclass
class Book:
    title: str
    summary: str
    author: str
    published_year: int

text = """Foundation is a science fiction novel by American writer
Isaac Asimov. ...< truncated >
... after the collapse of the Galactic Empire.
"""

with LLaMACPPParser(Book, model="mistral-7b-instruct-v0.1.Q4_K_M.gguf") as parser:
    book = parser.parse(text)
    print(book)

I would be willing to help move this feature directly in ggml in order to be able to use simple and lighter models for classification (I'm thinking about gpt2) - it may not be the best way to do that though (any feedbacks are welcomed).

[x4080]

Hi, how to emulate OneOf using this Grammar Builder ?

like for the functions.json below :

{
    "oneOf": [
        {
            "type": "object",
            "properties": {
                "function": {"const": "create_event"},
                "arguments": {
                    "type": "object",
                    "properties": {
                        "title": {"type": "string"},
                        "date": {"type": "string"},
                        "time": {"type": "string"}
                    }
                }
            }
        },
        {
            "type": "object",
            "properties": {
                "function": {"const": "image_search"},
                "arguments": {
                    "type": "object",
                    "properties": {
                        "query": {"type": "string"}
                    }
                }
            }
        }
    ]
}

Thanks

[paschembri]

From my point of view, this should be in the application code (not in grammar constraints).

I had to implement an Enum type and I managed to leave the grammar untouched. You can take a look here https://github.com/advanced-stack/py-llm-core/blob/main/src/llm_core/schema.py

[x4080]

Thanks for the reply

[bozo32]

I am looking for a way to demonstrate to masters students how LLMs work and what they will likely contribute to qualitative analysis of text going forward. For that I would like to contrast the results of unstructured and very structured prompting. I have some analysis methods that are quite well specified which produce plausible but not reproducible results when just submitted as a prompt. What you have here is a method to utterly formalize execution of a complex set of segmentation of a text and subsequent extractions and evaluations in a manner that produces partial results which supports auditing. This method would permit me, for example, to create a grammar for an analysis that could be shipped along with the results and data the same way that a R script is attached to the same which allows others to reproduce an analysis.
so, is there an idiot's guide somewhere that would permit me to turn a set of human-interpretable instructions into a grammar that would allow me, for example, to conduct a toulmin model argument analysis? or...is this still a bridge too far.

This seems the sort of thing that I could ask an agent to build for me...and perhaps eventually, but at this point it seems better to require humans to build the scripts.

[bozo32]

that is a phenomenal zero shot result on a complex chunk of data with minimal definitions and a pretty simple understanding of argument. Most forms of analysis are taught with very simple examples, so their discussion doesn't really anticipate the need to build a proper layered argument map . As it is done with students, it might be better to start with simpler targets and then see if/how it is possible to get more elaborate over time.
I've forwarded the result to another prof here who is interested in a different analysis method, I worry that there are folks who are attacking this from the computational linguistics part of the world (I'm very much at the applied end) and there are small pots of money here for exploring high risk interesting ideas peter.tamas at wur.nl

[nova-land]

I am the author of #3729, please let me know anything to make it more useful to generate structured prompting for your use case, TIA.

[paschembri]

As an additional note, there is this project: Argdown that defines markdown extensions to formalize arguments. But as this blog post explains, a grammar (BNF or EBNF) will not be achievable.

Yet, it could be interesting to fine tune a model to be able to translate written content into argdown.

[bozo32]

@paschembri
yes...it would be lovely to do exactly that. It may, however, be sufficient to use some sort of graph syntax.

@nova-land
I will be trying to teach my students two methods for which I would like to support comparison between human and AI analysis. The step up from 'dump human stuff in GPT4 and Trust' to what you are doing with a grammar is phenomenal. The two methods that I will be teaching are metaphor and argument analysis. Argument analysis requires identification of a relatively simple DAG (an argument map as explained by reasons.io) and then, for each step, description of a highly simplified list of components (claim, warrant, data). There are, therefore, two distinct steps. First, identification of the chain of claims that supports the conclusion of the article and then, once that is done, description of each of the claims.

The structure that you have presented so far does not seem to contemplate dependencies between items (cars are cars...we want to know their attributes, thank you very much). I would, therefore, like some help in setting up nested or recursive structures in which it is possible both to identify and then classify the relationships between claims (the most incredibly concise description I've found of relationships between arguments can be found here. One area where I see models consistently falling down is dropping intermediary steps. It seems that the models inconsistently recognise the bits that are relied on in reaching the final conclusion.

A slightly simpler task may be explicit metaphor analysis. This is one that has been attempted before with generative models with mixed success...if I remember right. For this one there are no relationships between identified metaphors, but there is a list of things to do within each metaphor. Again, initial identification of all occurrences and then subsequent stepwise investigation of each. I was not sure, looking at your interface, how to go about setting up such a two step process. There is a fun step at the end of metaphor analysis that may, however, introduce a complication. Once connotations are attached (these may be binary or weighted attachment of connotations), across all metaphors in a text, you are supposed to step back and see if they tend to come from a common 'family'...folks may be talking about baking cookies using 'war' metaphors. I've attached the human instructions for both of these processes below:

oh yes...before you look at these, from the perspective of a real linguist, what I'm doing here is crude in the extreme and, according to each expert, deeply wrong...but they are likely not to agree on HOW it is wrong. That is fine as all I'm trying to do is get my students to have a BIT of a clue.

Argument analysis:

1. For your chosen article, in pairs, pick which final ‘claim’ you want to work on
2. On pen and paper (and maybe post-its and crayons), work backwards in the article recursively finding claims which clearly support that final claim. This is your argument map…a stack of claims that support each other and eventually support the final claim.
3. Go back to the end of the article, to that final claim, and working backwards again, recursively:
   a. Fill in the argument structure (Toulmin this time). For the claim you are looking at, check if what you have labeled ‘claim’ serves as ‘data’ or as a ‘warrant’ (or maybe as something else) for the claim that it supports. Once you have added that code and made the link to the claim that it supports.
   b. Look for other stuff that matters. Keep a particular eye out for ‘qualifiers’ statements that condition the relevance of relationships between warrants, data and claims.
   c. Hunt for missing pieces (if what you have is a warrant, hunt for the data. If you have the data, hunt for the warrant etc.).
   d. Place a quotation on all of the text that is directly related to the claim that you are looking at (this may be a sentence or a paragraph) and code that quotation ‘argument
   e. Look at the structure of that argument, and fill in the blank yourself. Yes, guess at what that missing piece MUST be if the claim that it supports is to be accepted (e.g. there is a citation, but no discussion of content, therefore we’re assuming that the cited article is sound and that it is relevant). Place this in the comment to the quotation coded ‘argument’.
4. Create and apply a framework to classify the nature of the claims/data/warrants that you have identified. Data, for example, may be, citations or experience while a warrant may be similarity of context between site where the data comes from and place where that data is used.

these instructions will be later extended by the students...they will have to find flaws in the argument and then classify those flaws..I'll be using the pragma-dialectical approach which is pretty basic. The same sort of post processing at the unit level is present in metaphor analysis.

Metaphor analysis:

1. Read the selected text closely and identify all metaphors. A metaphor is any word or phrase that has meaning beyond the absolute literal. For example ‘wasting time’ requires time to be a limited and fixed resource that can be expended. This is time on the model of money. Turning to a second example ‘beating cancer’ introduces connotations from war.
2. For each metaphor identified, list all possible connotations
3. From those identified as possible, looking the context provided by surrounding text, select those connotations that are most likely to be relevant and rank order them. For each selected provide a justification and justify the ranking given
4. Examine all selected metaphors and, for each, propose possible types (e.g. war, family, biology, machines)
5. For the selected text, classify metaphors so that the total number of types is minimised.

what both of these seem to require, from the perspective of the grammar you are working on, is, in addition to identification of cars and their attributes, which you handily cover, the identification and classification of relationships between cars and the identification and classification of relationships between components within a car.

...
Argument gets weird because it requires imputation (identification of a necessary but absent component and then suggestion of what that unstated requirement may be)

I'm pretty much clueless at your end of the world...so apologies if this doesn't make much in the way of sense to you.

[nova-land]

Thank you for your detailed use case @bozo32. It would be interesting to create such qualitative analysis to state of the art LLMs. I don't know if you have a longer timeframe to create the above use case, I can have a try and look into it as a small research topic if you can provide some dummy examples. If you feel suitable, you might email me, thx.

[bozo32]

Hello My immediate interest is in creating demonstrations for my students using llama.cpp of a stripped out version of naïve description (yes, that is a thing…but they dress it up fancy for publication), argument analysis and metaphor analysis that are better than https://chat.openai.com/share/401b6d46-6261-4f9a-869f-d2b11bbcd2bb does that serve as a dummy example? The thing I run into immediately is intermediate results x limits imposed by context window size (which seems to kill the linked chatGPT example). I have been beating my incompetent head against AGiXT for a while unsuccessfully to try and get an agent to manage all of the data produced and required for intermediary steps. That said, I would rather fully script both the model and its interaction with a cache as that is reproducible and transparent. I do have a longer timeframe with larger ambitions and some avenues to get funding. https://www.nwo.nl/en/calls?input=AI This is where I’m at with putting together thoughts for a larger project: https://www.overleaf.com/read/pfsgxpjsznky#bdaaaa there are staff here who have more of the sort of credibility required to get that money. As for how to splice that into existing projects I’ve shared this with a few folks, but we need more software development before we can make a credible bid https://www.overleaf.com/project/64393e49de89fce483242650 My long term interest is in contributing to a set of increasingly difficult tasks against which LLMs can be assessed…and the formalization of scripts that can be used to support a diversity of analysis methods. To provide one example of the sort of stacking difficulty, argument analysis Which is just one of several strands of analysis, but is kind of fundamental to them all, Given a single speaker argument such as https://www.government.nl/documents/speeches/2020/03/16/television-address-by-prime-minister-mark-rutte-of-the-netherlands 1. What is the main point for which Rutte is arguing? 2. Extract an argument map (tree structure of supporting claims) 3. Extract a simplified Toulmin description (current attempt) 4. Extract a complete Toulmin model (including qualifiers) 5. Extract a complete Toulmin model and classify components (e.g. argument from authority | evidence |tradition) 6. Assess the strength of the argument given standards (e.g. evidence = 5, tradition = 3, ad hominum = 0) 7. Do a failure analysis of an argument (given identified failure at step 7 of 32, what are the consequences for the final conclusion) 8. Add to 1-7 export in a format that R can turn into a pretty picture 9. Comparatively assess the equivalent speeches by Trump, Rutte, Trudeau and Merkel This can be expanded into an analysis that, then, looks at a debate using structures like this: https://en.wikipedia.org/wiki/Pragma-dialectics There are other methods that would fairly easily support some sort of similar progressive testing setup. The ones I find fun require recognition beyond the explicit and unambiguous (e.g. metaphor analysis requires identification of a list of plausible connotations and, given context, nomination of the connotation that is most probable) One of the perhaps useful features of this sort of approach is that we can switch out the texts which may defeat those who are trying to game assessments.

…

-peter From: NovaLand ***@***.***> Reply to: "ggerganov/llama.cpp" ***@***.***> Date: Saturday, 28 October 2023 at 22:05 To: "ggerganov/llama.cpp" ***@***.***> Cc: peter tamas ***@***.***>, Mention ***@***.***> Subject: Re: [ggerganov/llama.cpp] Grammar generator app (Discussion #2494) Thank you for your detailed use case @bozo32<https://github.com/bozo32>. It would be interesting to create such qualitative analysis to state of the art LLMs. I don't know if you have a longer timeframe to create the above use case, I can have a try and look into it as a small research topic if you can provide some dummy examples. If you feel suitable, you might email me, thx. — Reply to this email directly, view it on GitHub<#2494 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYKOUNNIOKFGCZYTT63R2ADYBVQRZAVCNFSM6AAAAAA3BZTRVKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TIMJSHE3TK>. You are receiving this because you were mentioned.Message ID: ***@***.***>