prompty: adding Microsoft langchain_prompty package #21346

quchuyuan · 2024-05-06T21:06:23Z

No description provided.

vercel · 2024-05-06T21:06:26Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		May 11, 2024 4:01am

quchuyuan · 2024-05-07T19:18:17Z

@efriis Hi, I know there are some conversations going on on the business side. But could please you give me some review to help me better prepare this PR so that we can merge it as soon as we made a deal?

efriis

Hey team! I don't think this is doing what we discussed. Wasn't the plan to reconstruct a prompty spec as a prompt or prompt | chat_model runnable in the langchain runtime?

This seems to be doing a great deal of bespoke converting to and from some different format.

efriis · 2024-05-06T22:37:31Z

libs/partners/prompty/langchain_prompty/__init__.py

@@ -0,0 +1 @@
+from .langchain import *


switch to explicit import, define __all__, and add newline

this is fiexed

efriis · 2024-05-06T22:39:46Z

libs/partners/prompty/langchain_prompty/langchain.py

+from typing import Dict
+from .utils import load, prepare
+
+def create_chat_prompt(path: str, input_name_agent_scratchpad = "agent_scratchpad") -> Runnable[Dict[str, ChatPromptTemplate], str]:


Why does the returned runnable take a dictionary mapping str -> ChatPromptTemplate as input?

my bad, typo here, it should be

Runnable[Dict[str, any], ChatPromptTemplate]

I'm basically constructing a ChatPromptTemplate using input and prompty. Prompty will render and parse based on user inputs.

What would a user pass in that's different than what they'd pass into a ChatPromptTemplate? Would it be possible to have it return a ChatPromptTemplate instead of a RunnableLambda that returns a ChatPromptTemplate?

efriis · 2024-05-06T22:40:42Z

libs/partners/prompty/langchain_prompty/renderers.py

@@ -0,0 +1,26 @@
+from pydantic import BaseModel
+from .core import Invoker, InvokerFactory, Prompty, SimpleModel
+from jinja2 import DictLoader, Environment


cc @eyurtsev - will this trigger the same vulnerabilities in Jinja2 as we've seen in the past?

@efriis is langchain removing jinja for all?
happy to support additional formats too as discussed.

In general, I think it would make the most sense for the LangChain runtime for prompty to support the prompt template styles that langchain supports, and not introduce a new one just for prompty.

We removed jinja2 because of code sandboxing concerns when reading from files (similar to this). See #10252 for more details. See this example for injecting for RCE: https://github.com/langchain-ai/langchain/pull/10252/files#diff-a0bd39be3ca1018bba7dcb089c148cb0083fddbf7eb95d374bf1d86ecfcd1093

For the initial release, I would recommend supporting f-strings and mustache by default, so the jinja2 vulnerabilities don't attract CVEs. In general, it's very difficult to guarantee that people are only loading .prompty files that are static/controlled by the developer, and if you want to proceed with Jinja2, I'll probably recommend some updates to docstrings and documentation to reflect the danger of loading anything user-defined with these functions.

Thanks @efriis, I fully agree with you. Judging on the timeline, here's my proposal, let me know what you think:

We will start with Jinja2, since that's mostly where people are using today. And like you said, will update docstrings and doc to reflect the danger of it.

Start supporting mustache by default, we also need to coordinate with Microsoft orchestration frameworks to support those.

Does this sound good?

Let me chat with the team and get back to you. Because this package has our name on it, I am hesitant to merge something in that introduces the same vulnerability as this 9.8/10 CVE from last year that caused us to strip out Jinja2 in the first place: GHSA-7gfq-f96f-g85j

If there's any way to switch to Mustache for launch (pretty straightforward, and similar syntax to Jinja. Support in most languages.), that would be make releasing this much easier.

Ok how about this:

insert "danger arbitrary code execution" warnings into docstrings, readme, and any docs (to prevent people misusing and parsing user input or anything from an unsandboxed filesystem)

moving to a partner repo (langchain-ai/langchain-prompty) so github doesn't flag security in this monorepo if CVEs get filed for it

publish from partner repo

@efriis we discussed internally a little bit, here's the initial thinking:

In our "template.type" field, we will add "mustache" support. In our documentation, we will suggest user to use mustache instead of jinja.

In this langchain-prompty package, we will support mustache by default if that option isn't set by user.

In our langchain template samples, we will also use mustache.

But if user sets type to jinja explicitly, it'll use jinja.

In our VSCode extension, we would support mustache if user sets that "type" field. But by default still support jinja, until other Microsoft frameworks support mustache (we need some time to coordinate).

So basically, all langchain interface are mustache, except .4 if user explicitly specifies it. Or do you think we should delete 4 too? I'm ok to delete 4 but there would be some interoperability issues for the initial period.

@efriis

also is there any discussion somewhere comparing different options of templating engine? Just curious why the consensus is mustache, vs others? For example, mustache can't handle dictionary object well, handlebar seems easier to use, which extends from mustache

@efriis we also handle security pretty seriously in Microsoft. So I've changed to support only mustache. I think it does has its merits by being simple, less capability but more secure. And probably mostly enough for prompt workload already. Can you take a look and see if this is ready to merge?

wayliums · 2024-05-08T20:09:38Z

Hey team! I don't think this is doing what we discussed. Wasn't the plan to reconstruct a prompty spec as a prompt or prompt | chat_model runnable in the langchain runtime?

This seems to be doing a great deal of bespoke converting to and from some different format.

@efriis The experience in this proposal is

    prompt = langchain_prompty.create_chat_prompt('chat.prompty')
    prompt | chat_model

Any suggestion on how it should look like?

efriis · 2024-05-09T22:22:35Z

yes! The experience should ideally be:

prompt = langchain_prompty.create_chat_prompt('chat.prompty')
type(prompt) # -> ChatPromptTemplate
chain = prompt | model
chain.invoke({"variable": "value"}) # AIMessage(...)

wayliums · 2024-05-09T22:41:30Z

yes! The experience should ideally be:

prompt = langchain_prompty.create_chat_prompt('chat.prompty')
type(prompt) # -> ChatPromptTemplate
chain = prompt | model
chain.invoke({"variable": "value"}) # AIMessage(...)

So my understanding is you are proposing the same code experience, but the return type is ChatPromptTemplate, right? I actually started with that and switched to the Runnable<input, ChatPromptTemplate> approach.

Prompty's main value proposition is offering a serialized and easy to write prompty format. Basically Prompy+Input => Messages

We handle rendering (expand the template) + parsing (convert to Message list that LLM can take). In your suggestion, I just converted it to ChatPromptTemplate once in the beginning, then I'm losing the ability to render again based on different inputs.

Langchain has a special MessagesPlaceholder, but we actually give the freedom back to user. For example

{% for item in FewShotStarterMessages %}
  {{item.role}}:
  {{item.content}}
{% endfor %}

@efriis I hope this make sense? Maybe there are other ways that I don't know. Would love to hear your thoughts.

efriis · 2024-05-09T23:08:01Z

Is the distinction that you want to expand the Jinja2 template before parsing it as messages? Such that a user could wrap a for loop that ends up generating multiple messages?

You can actually do what you described in normal ChatPromptTemplate as long as it's either

within a single message (iterating a list input variable in a mustache template)
each entry is its own message (chat history)

efriis · 2024-05-09T23:10:17Z

The main features you miss out on by returning a custom runnable as opposed to a ChatPromptTemplate is general compatibility with the langchain ecosystem. These runnables

won't be usable in LangSmith (either in playground or hub)
can't do .partial - would use generic .bind instead

wayliums · 2024-05-09T23:36:07Z

within a single message (iterating a list input variable in a mustache template)
Yes, that's what we want to do. And we want the format to work across language and frameworks too. That's why we are handling the render+parsing

Can you give me an example of this?

within a single message (iterating a list input variable in a mustache template)

wayliums · 2024-05-09T23:43:47Z

The main features you miss out on by returning a custom runnable as opposed to a ChatPromptTemplate is general compatibility with the langchain ecosystem. These runnables

won't be usable in LangSmith (either in playground or hub)

can't do .partial - would use generic .bind instead

By "usable in langsmith" you mean the prompts playground, not the langserve playground? Right? I think if we are integrating with langsmith playground (we are not target it now as there would be quite some business implication), we would look for directly supporting .prompty format, then use javascript to parse it. We are doing that to our VSCode extension already.
By .partial you mean this? https://python.langchain.com/v0.1/docs/modules/model_io/prompts/partial/#partial-with-strings. Interesting, I need to think about this a little bit. I guess I could call it out in docstring too and not be blocked by this yet?

efriis · 2024-05-10T01:03:45Z

within a single message (iterating a list input variable in a mustache template)
Yes, that's what we want to do. And we want the format to work across language and frameworks too. That's why we are handling the render+parsing

Can you give me an example of this?
within a single message (iterating a list input variable in a mustache template)

https://smith.langchain.com/prompts/eyfriis/mustache-example

And yes to partial and LangSmith tracing as well as playground as benefits of using the prompt format within the LangChain runtime! For example, you could explore traces that use prompty prompts like the run of one above: https://smith.langchain.com/public/dc6657c1-801f-496b-9e99-b27d3b1d976d/r

wayliums · 2024-05-10T02:29:02Z

within a single message (iterating a list input variable in a mustache template)
Yes, that's what we want to do. And we want the format to work across language and frameworks too. That's why we are handling the render+parsing

Can you give me an example of this?
within a single message (iterating a list input variable in a mustache template)
https://smith.langchain.com/prompts/eyfriis/mustache-example

And yes to partial and LangSmith tracing as well as playground as benefits of using the prompt format within the LangChain runtime! For example, you could explore traces that use prompty prompts like the run of one above: https://smith.langchain.com/public/dc6657c1-801f-496b-9e99-b27d3b1d976d/r

In your example above, looks like it's not populating multiple message? It's still the same single message with user role but just the content is looped.

@efriis, but I feel this is something we could iterate on though, since the code user write isn't different between our proposals. So it could be a seamless update later too. What do you think?

efriis · 2024-05-10T21:14:36Z

libs/partners/prompty/tests/unit_tests/prompts/basic_chat.prompty

+{{#chat_history}}
+{{role}}:
+{{content}}
+{{/chat_history}}


Suggested change

{{#chat_history}}

{{role}}:

{{content}}

{{/chat_history}}

placeholder:

{{chat_history}}

Would you consider something like this for chat history in the langchain runtime? If so, we can get rid of the custom parsing and interpret this as a MessagesPlaceholder object, which does the same thing as this.

I'm not sure it makes sense to generate mixed sequence messages and iterated contents of messages because it's typically one or the other. Are you imagining a case where users want to prompty-format a generic list of inputs into multiple messages, instead of specifically

role: content

blocks?

The reason I ask is because chat histories in the langchain runtime will typically be passed in as a List[BaseMessage] instead of List[Dict] where dict matches the format {"role": x, "content": y}

This is rendered well by MessagesPlaceholder, but would not be rendered well for other messages cases (e.g. ones with tool_calls or ToolMessage).

You mean using placeholder as a special keyword in prompty? We want prompty to be usable across orchestration framework though (Promptflow, SemanticKernel etc), so I'm reluctant to add framework specific keywords. Maybe we can add a special langchain parser to support that later? But for //build we want to stick to a more generic approach.

That's also the main reason why I'm not ready to switch from returning Runnable<> to ChatTemplate. Coz then I would need something special in prompty to indicate it should be swapped to MessagesPlaceholder. And that concept doesn't exist in other places we want to support.

efriis · 2024-05-10T21:34:25Z

libs/partners/prompty/langchain_prompty/renderers.py

+
+    def invoke(self, data: BaseModel) -> BaseModel:
+        assert isinstance(data, SimpleModel)
+        generated = chevron.render(self.prompty.content, data.item)


this will actually be a security issue too because of chevron partials. Would be better if you just constructed a ChatPromptTemplate with type as mustache!

As an alternative, you can also use the partial-free renderer in langchain_core that mustache ChatPromptTemplates rely on https://github.com/langchain-ai/langchain/pull/19980/files#diff-c591786bc4be9ea7fab44200ea051b2e29ad2cda359a8cd3577eac4d0b2b0c7bR386

ah nice! Didn't know you have this, switching to your partial free renderer now

quchuyuan and others added 4 commits April 26, 2024 11:19

init

5fbc641

added test files

0a6cdb9

working prompty package initial checkin

724bc7e

making sure the package is working

ba3c657

efriis added the partner label May 6, 2024

efriis self-assigned this May 6, 2024

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. 🤖:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features labels May 6, 2024

Delete unnecessary files

6d9b07d

quchuyuan changed the title ~~Adding Microsoft langchain_prompty package~~ langchain_prompty: adding Microsoft langchain_prompty package May 6, 2024

quchuyuan added 2 commits May 6, 2024 14:09

Merge branch 'master' into microsoft/chuyuan/prompty

20f1b7d

Merge branch 'master' into microsoft/chuyuan/prompty

b3bf03d

efriis reviewed May 8, 2024

View reviewed changes

Update langchain.py

4626f94

wayliums added 2 commits May 10, 2024 20:43

switch to mustache

c5658a5

change to explict export

b94eb69

efriis reviewed May 10, 2024

View reviewed changes

switch to langchain's partial free renderer

be46d2f

efriis approved these changes May 10, 2024

View reviewed changes

dosubot bot added the lgtm PR looks good. Use to confirm that a PR is ready for merging. label May 10, 2024

efriis enabled auto-merge (squash) May 10, 2024 23:28

efriis added 4 commits May 10, 2024 16:31

x

78e9acd

x

0b35a91

x

2a685ba

x

525cd6c

efriis disabled auto-merge May 10, 2024 23:41

x

8596f1d

vercel bot deployed to Preview May 10, 2024 23:54 View deployment

test fix linting errors

c25069f

vercel bot deployed to Preview May 11, 2024 00:17 View deployment

finished core changes

46711f3

vercel bot deployed to Preview May 11, 2024 00:45 View deployment

wayliums added 2 commits May 11, 2024 03:16

fix style errors

8af8222

fix more linting erors

cc584e0

vercel bot deployed to Preview May 11, 2024 03:40 View deployment

efriis added 3 commits May 10, 2024 20:46

Merge branch 'master' into microsoft/chuyuan/prompty

331ab58

x

8a4b93c

x

645da01

efriis enabled auto-merge (squash) May 11, 2024 03:54

efriis disabled auto-merge May 11, 2024 03:54

efriis changed the title ~~langchain_prompty: adding Microsoft langchain_prompty package~~ prompty: adding Microsoft langchain_prompty package May 11, 2024

efriis enabled auto-merge (squash) May 11, 2024 03:54

efriis added 3 commits May 10, 2024 20:58

x

a79411e

x

cfd0b0a

x

8757813

efriis disabled auto-merge May 11, 2024 04:01

efriis enabled auto-merge (squash) May 11, 2024 04:01

efriis merged commit af875cf into langchain-ai:master May 11, 2024
21 checks passed

matthieucx mentioned this pull request Aug 23, 2024

prompty: Superfluous f-string templating causes prompt templaing to fail #25703

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prompty: adding Microsoft langchain_prompty package #21346

prompty: adding Microsoft langchain_prompty package #21346

quchuyuan commented May 6, 2024 •

edited by efriis

Loading

vercel bot commented May 6, 2024 •

edited

Loading

quchuyuan commented May 7, 2024

efriis left a comment

efriis May 6, 2024

wayliums May 10, 2024

efriis May 6, 2024

wayliums May 8, 2024

efriis May 9, 2024

efriis May 6, 2024

wayliums May 8, 2024 •

edited

Loading

efriis May 9, 2024

wayliums May 9, 2024

efriis May 9, 2024

efriis May 10, 2024

wayliums May 10, 2024 •

edited

Loading

wayliums May 10, 2024 •

edited

Loading

wayliums May 10, 2024

wayliums commented May 8, 2024 •

edited

Loading

efriis commented May 9, 2024

wayliums commented May 9, 2024 •

edited

Loading

efriis commented May 9, 2024

efriis commented May 9, 2024

wayliums commented May 9, 2024

wayliums commented May 9, 2024

efriis commented May 10, 2024

wayliums commented May 10, 2024 •

edited

Loading

efriis May 10, 2024

efriis May 10, 2024

wayliums May 10, 2024

efriis May 10, 2024

wayliums May 10, 2024

prompty: adding Microsoft langchain_prompty package #21346

prompty: adding Microsoft langchain_prompty package #21346

Conversation

quchuyuan commented May 6, 2024 • edited by efriis Loading

vercel bot commented May 6, 2024 • edited Loading

quchuyuan commented May 7, 2024

efriis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wayliums May 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wayliums May 10, 2024 • edited Loading

Choose a reason for hiding this comment

wayliums May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wayliums commented May 8, 2024 • edited Loading

efriis commented May 9, 2024

wayliums commented May 9, 2024 • edited Loading

efriis commented May 9, 2024

efriis commented May 9, 2024

wayliums commented May 9, 2024

wayliums commented May 9, 2024

efriis commented May 10, 2024

wayliums commented May 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quchuyuan commented May 6, 2024 •

edited by efriis

Loading

vercel bot commented May 6, 2024 •

edited

Loading

wayliums May 8, 2024 •

edited

Loading

wayliums May 10, 2024 •

edited

Loading

wayliums May 10, 2024 •

edited

Loading

wayliums commented May 8, 2024 •

edited

Loading

wayliums commented May 9, 2024 •

edited

Loading

wayliums commented May 10, 2024 •

edited

Loading