LLM support

[kaihendry]

I noticed that using popular AI tools to work with PKL leads to poor results:

llm-test $ cat docker-compose.yml | llm -x -m claude-3.7-sonnet -s "convert this YAML to PKL https://pkl-lang.org/" | tee test.pkl
module docker {
  version = "3"

  services {
    alpine-curl {
      image = "alpine/curl"
      command = ["-fsSL", "http://manager:8080"]
      networks = ["duser_default"]
    }
  }

  networks {
    duser_default {
      external = true
    }
  }
}

llm-test $ pkl eval test.pkl
–– Pkl Error ––
Invalid token at position. Expected a class, typealias, method, or property.

1 | module docker {
                  ^
at test (file:///Users/hendry/tmp/llm-test/test.pkl, line 1)

llm-test $ cat docker-compose.yml | llm -x -m gpt-4o-mini -s "convert this YAML to PKL https://pkl-lang.org/" | tee test.pkl
version: "1.0"

services:
  - name: alpine-curl
    image: "alpine/curl"
    command: ["-fsSL", "http://manager:8080"]
    networks:
      - duser_default

networks:
  - name: duser_default
    external: true

llm-test $ pkl eval test.pkl
–– Pkl Error ––
Unexpected token `-`. Expected a type.

4 | - name: alpine-curl
    ^
at test (file:///Users/hendry/tmp/llm-test/test.pkl, line 4)

Do you have an MCP in mind or something to help models generate PKL code?

[bioball]

To be honest, I haven't thought about this much, and I'm also a novice when it comes to LLMs.

Curious what thoughts other people have here.

[jjuliano]

I am not sure if there is a direct support to LLM inside PKL.
However, I made an AI agent framework powered by PKL.

https://github.com/kdeps/kdeps

[StefMa]

LLMs doesn't know "anything" about pkl because "the world" doesn't know anything about pkl 😉.

Pkl is, compared to other languages, just not present. From where should LLM learn things?
Further, even Kotlin itself is "so new", that JetBrains decided actively to push training data for LLMs on the net (check their huggingface account).

As long as pkl stays like it is now, it will never©️ going to be useful in LLMs, I think.

But of course, if it's growth and more people adopt it, it is easier for LLMs to understand it.
Just because GitHub itself (their repositories with a permissive license) are useful training data.

So what can pkl do about it?
As mentioned, going a similar way as JetBrains and actively providing training data for pkl. Not saying that "the big foundation models" will definitely using it for training or fine tuning. But the changes is higher if there is something, right? 😁

You mention MCP. Yes, probably a MCP server would also help. But which tools or resources should it provide? Resources might be obvious, the super nice pkl documentation. But since it contains "only" code snippets, I would assume that this is too less for the LLM to map it to real world scenarios to provide useful code.
So it might help a bit, especially in making the documentation more "live" (cause you can go into an Q&A conversation with the LLM) but I think it won't help in generating pkl code.