Update conversational widget to use text-generation (+ remove `conversational` task) #457

Wauplin · 2024-01-25T15:59:52Z

Done as part of https://github.com/huggingface/moon-landing/issues/8578. Should be merged before (or at the same time) as https://github.com/huggingface/moon-landing/pull/8723. This is only a first draft to check if we have everything we need.

From https://github.com/huggingface/moon-landing/issues/8578:

In huggingface.js and api-inference

Models that are secondary tagged as conversational will get the ConversationalWidget

The ConversationalWidget will call the text-generation API under the hood. The widget needs to take care of all prompt formatting (using the recent jinja work in huggingface.js)

Should we just kill the conversational API in the inference API with the APIs unification?

This would break use cases such as pipeline("microsoft/DialoGPT-medium") in transformers

Result:

All models with conversational capabilities will have a nice widget

We eliminate the fragmentation of tasks (conversational vs text generation)

We remove the confusing conv pipeline

Currently in this PR:

✔️ Models that are secondary tagged as conversational will get the ConversationalWidget
✔️ The ConversationalWidget will call the text-generation API under the hood. (automatic in inference API if pipeline_tag gets updated by https://github.com/huggingface/moon-landing/pull/8723)
✔️ The widget needs to take care of all prompt formatting (not complete)

cc @xenova @osanseviero @SBrandeis @coyotte508

Still unsure how to proceed:

how to handle the transition period? => EDIT: no transition period
what to do if we don't have a chat_template? => EDIT: raise error
what if we have a chat_template but no eos_token / bos_token? => EDIT: should be ok
should we keep the Conversation structure in the widget (with generated_responses / past_user_inputs / generated_text) ? If not, would need more svelte expertise 😄 => EDIT: ok

xenova · 2024-01-25T21:27:56Z

(my first commits just fixed some of the dependencies to @huggingface/jinja).

Let me try to answer (or at least provide discussion for) your questions:

how to handle the transition period?

We'll get this PR ready before merging https://github.com/huggingface/moon-landing/issues/8578, and then hopefully there should be no "transition period" (each text-generation model will be assigned a conversational or text-generation widget). Sure, there might be issues with compatibility, but we'll add some checks to default back to previous functionality if something is missing (like bos_token or something).

what to do if we don't have a chat_template?

The way transformers.js handles this, is that (like the transformers library) we define the default chat templates associated for each model type (e.g., here for llama). If no chat_template is found and the user tries to run apply_chat_template, we would use the default one. However, for widgets, I think it's best that we only set it to conversational if chat_template is defined in the tokenizer_config.json.

what if we have a chat_template but no eos_token / bos_token?

This is most likely due to an outdated tokenizer_config.json (like this), but is also possible if the model doesn't have a tokenizer_config.json. Fortunately, when saving newer tokenizers, the required special tokens are saved. However, in cases where something is not defined, we can either (1) use pre-defined defaults, or (2) disable the widget, or (3) default to text-generation.

should we keep the Conversation structure in the widget (with generated_responses / past_user_inputs / generated_text) ? If not, would need more svelte expertise 😄

I think we will need to convert it to the ChatML format (JSON array of messages, each with at least role and content values), since this is how the templates expect the conversation to be provided.

xenova · 2024-01-25T21:45:47Z

We also need to decide how users should define system prompts/messages. This might be something we need to pass in from the yaml in the model card (i.e., in the same way as examples are specified)

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte

packages/widgets/src/lib/components/InferenceWidget/InferenceWidget.svelte

@Wauplin

# TL;DR - Update `text-generation` spec to match TGI API - ~Add `conversational` spec, heavily inspired by TGI messages API (cc @Wauplin @osanseviero @Narsil )~ - ~Relevant related work: #457 & huggingface/moon-landing#8723 - regenerate typescript code for those tasks

…-text-generation

…b.com:huggingface/huggingface.js into 8578-switch-conversational-to-text-generation

Wauplin · 2024-02-12T16:38:15Z

I've updated this PR with feedback and improved the logic to handle chat format.
@xenova is finishing the templating part (discussed in dm) and after that we should be good to try / review this PR.

cc @julien-c @osanseviero I have also merged work from #479 that will remove the conversational task from hf.co/models + define the WidgetType.

packages/widgets/src/routes/+page.svelte

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte

julien-c · 2024-02-13T17:34:47Z

packages/widgets/src/lib/components/InferenceWidget/InferenceWidget.svelte

-		model.pipeline_tag && model.pipeline_tag in WIDGET_COMPONENTS
-			? WIDGET_COMPONENTS[model.pipeline_tag as keyof typeof WIDGET_COMPONENTS]
-			: undefined;
+		model.pipeline_tag === "text-generation" && model.tags?.includes("conversational")


maybe also model.pipeline_tag === text2text-generation btw @osanseviero no?

Not needed imo. https://huggingface.co/models?pipeline_tag=text2text-generation&other=conversational&sort=trending only has 29 models and they don't have templates (vs 9300 public models in https://huggingface.co/models?pipeline_tag=text-generation&other=conversational&sort=trending )

(also moot if we merge text2text-generation into text-generation like i'd like to 😁)

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte

osanseviero

Looking nice!

osanseviero · 2024-02-13T20:13:39Z

packages/tasks/src/tasks/conversational/about.md

-
-## Useful Resources
-
- Learn how ChatGPT and InstructGPT work in this blog: [Illustrating Reinforcement Learning from Human Feedback (RLHF)](https://huggingface.co/blog/rlhf)


FYI @merveenoyan as we'll want to move some of this to text-generation task page

osanseviero · 2024-02-13T20:15:39Z

packages/widgets/src/lib/components/InferenceWidget/InferenceWidget.svelte

-		model.pipeline_tag && model.pipeline_tag in WIDGET_COMPONENTS
-			? WIDGET_COMPONENTS[model.pipeline_tag as keyof typeof WIDGET_COMPONENTS]
-			: undefined;
+		model.pipeline_tag === "text-generation" && model.tags?.includes("conversational")


Not needed imo. https://huggingface.co/models?pipeline_tag=text2text-generation&other=conversational&sort=trending only has 29 models and they don't have templates (vs 9300 public models in https://huggingface.co/models?pipeline_tag=text-generation&other=conversational&sort=trending )

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte

Co-authored-by: Julien Chaumond <julien@huggingface.co>

SBrandeis

Cool stuff 🔥

xenova · 2024-02-16T16:37:25Z

I also just added some error checking to the compilation and render steps of the chat template:

Invalid chat templates

The error message is supplied by @huggingface/jinja. For example:

{{ test
{% for %}

Valid template, but error during runtime

(To test, I just deleted bos_token)

Finally, I updated it so that we only add message on first call (not also if model loading response received). Otherwise we get:

xenova

Looks good to go!

small nit: maybe we should clear the text input once we hit enter? (maybe only necessary once we add streaming)

Wauplin

Can't approve my own PR but I reviewed @xenova last changes. Thanks for making the extra step ❤️ Everything looks good for a merge now :)

Will merge it on tomorrow if no-one complains until then (not sure anymore who should/want to review it one more time 😄 )

osanseviero · 2024-02-19T14:47:07Z

🚀 🚀

julien-c

on my side i'll check in prod 🙂

Wauplin · 2024-02-20T09:31:32Z

And... finally merged! 😄 🚀

mishig25 · 2024-02-21T14:56:54Z

...widgets/src/lib/components/InferenceWidget/shared/WidgetOutputConvo/WidgetOutputConvo.svelte

-		input: string;
-		response: string;
+	export let messages: Array<{
+		role: string;


could we make the typing even stronger here?

role: "user" | "assistant" | "system"

?

@mishig25

following #457 (comment) cc @mishig25 --------- Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>

Wauplin and others added 6 commits January 25, 2024 16:48

first draft

5912d96

wip

eaa4808

Add @huggingface/jinja as dependency

4b49456

Update pnpm-lock.yaml

4c93818

format

6596be5

formatting

188c0e6

Fix errors

4340d70

Rocketknight1 reviewed Jan 29, 2024

View reviewed changes

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte Outdated Show resolved Hide resolved

coyotte508 force-pushed the main branch from 7c653d5 to 0f29277 Compare February 6, 2024 14:41

SBrandeis mentioned this pull request Feb 6, 2024

♻️ [Tasks] JSON spec: text-generation #468

Merged

julien-c reviewed Feb 8, 2024

View reviewed changes

packages/widgets/src/lib/components/InferenceWidget/InferenceWidget.svelte Outdated Show resolved Hide resolved

SBrandeis reviewed Feb 8, 2024

View reviewed changes

packages/widgets/src/lib/components/InferenceWidget/InferenceWidget.svelte Outdated Show resolved Hide resolved

Introduce WidgetType = PipelineType | "conversational";

32b6820

julien-c mentioned this pull request Feb 8, 2024

Introduce WidgetType = PipelineType | "conversational"; #479

Closed

Wauplin added 2 commits February 12, 2024 11:57

Merge branch 'main' into 8578-switch-conversational-to-text-generation

af99492

feedback

4b6b60c

Wauplin added 5 commits February 12, 2024 15:30

Merge branch 'kill-conversational' into 8578-switch-conversational-to…

f463fe2

…-text-generation

update chat logic

cd17cf8

Merge branch 'main' into 8578-switch-conversational-to-text-generation

bdb1f82

revert

3e469e6

Merge branch '8578-switch-conversational-to-text-generation' of githu…

43d2708

…b.com:huggingface/huggingface.js into 8578-switch-conversational-to-text-generation

Wauplin changed the title ~~[WIP] Prepare conversational widget to use text-generation~~ [WIP] Switch conversational widget to use text-generation Feb 12, 2024

Wauplin changed the title ~~[WIP] Switch conversational widget to use text-generation~~ [WIP] Update conversational widget to use text-generation Feb 12, 2024

Wauplin added 3 commits February 12, 2024 17:42

style

18d9bee

stuff that should be moved back to other PR

41ccfcc

lint

b677d69

julien-c reviewed Feb 13, 2024

View reviewed changes

packages/widgets/src/routes/+page.svelte Outdated Show resolved Hide resolved

SBrandeis reviewed Feb 13, 2024

View reviewed changes

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte Outdated Show resolved Hide resolved

julien-c reviewed Feb 13, 2024

View reviewed changes

.../src/lib/components/InferenceWidget/widgets/ConversationalWidget/ConversationalWidget.svelte Show resolved Hide resolved

osanseviero reviewed Feb 13, 2024

View reviewed changes

xenova and others added 4 commits February 14, 2024 01:23

Update packages/widgets/src/routes/+page.svelte

ce7e7cd

Co-authored-by: Julien Chaumond <julien@huggingface.co>

adapt WidgetOutputConvo to chat messages

05b333a

compile template onMount

abf7f8a

remove unecessary lines

25c77c8

SBrandeis approved these changes Feb 14, 2024

View reviewed changes

SBrandeis requested review from coyotte508, osanseviero and julien-c February 14, 2024 15:06

xenova added 2 commits February 16, 2024 16:34

Display error if invalid template

9acfe0b

Only update messages on first call

e2b87e5

This was referenced Feb 18, 2024

General Maintenance + Deprecate Conversational Endpoint (for now) Kardbord/hfapigo#32

Merged

Errors with Conversational Endpoint #488

Closed

xenova approved these changes Feb 19, 2024

View reviewed changes

Wauplin commented Feb 19, 2024

View reviewed changes

julien-c reviewed Feb 20, 2024

View reviewed changes

Merge branch 'main' into 8578-switch-conversational-to-text-generation

076169e

Wauplin merged commit 802e164 into main Feb 20, 2024
2 checks passed

Wauplin deleted the 8578-switch-conversational-to-text-generation branch February 20, 2024 09:31

mishig25 reviewed Feb 21, 2024

View reviewed changes

Wauplin mentioned this pull request Feb 21, 2024

Stronger typing for convo messages #493

Merged

Wauplin added a commit that referenced this pull request Feb 22, 2024

Stronger typing for convo messages (#493)

bab7c35

following #457 (comment) cc @mishig25 --------- Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>

coyotte508 mentioned this pull request Mar 27, 2024

[Conversational] Property conversational does not exist on type HfInference #586

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update conversational widget to use text-generation (+ remove `conversational` task) #457

Update conversational widget to use text-generation (+ remove `conversational` task) #457

Wauplin commented Jan 25, 2024 •

edited

xenova commented Jan 25, 2024

xenova commented Jan 25, 2024

Wauplin commented Feb 12, 2024

julien-c Feb 13, 2024

osanseviero Feb 13, 2024

julien-c Feb 14, 2024

osanseviero left a comment

osanseviero Feb 13, 2024

osanseviero Feb 13, 2024

SBrandeis left a comment

xenova commented Feb 16, 2024

xenova left a comment •

edited

Wauplin left a comment

osanseviero commented Feb 19, 2024

julien-c left a comment

Wauplin commented Feb 20, 2024 •

edited

mishig25 Feb 21, 2024

Wauplin Feb 21, 2024


		## Useful Resources

		- Learn how ChatGPT and InstructGPT work in this blog: [Illustrating Reinforcement Learning from Human Feedback (RLHF)](https://huggingface.co/blog/rlhf)

Update conversational widget to use text-generation (+ remove conversational task) #457

Update conversational widget to use text-generation (+ remove conversational task) #457

Conversation

Wauplin commented Jan 25, 2024 • edited

xenova commented Jan 25, 2024

xenova commented Jan 25, 2024

Wauplin commented Feb 12, 2024

julien-c Feb 13, 2024

Choose a reason for hiding this comment

osanseviero Feb 13, 2024

Choose a reason for hiding this comment

julien-c Feb 14, 2024

Choose a reason for hiding this comment

osanseviero left a comment

Choose a reason for hiding this comment

osanseviero Feb 13, 2024

Choose a reason for hiding this comment

osanseviero Feb 13, 2024

Choose a reason for hiding this comment

SBrandeis left a comment

Choose a reason for hiding this comment

xenova commented Feb 16, 2024

Invalid chat templates

Valid template, but error during runtime

xenova left a comment • edited

Choose a reason for hiding this comment

Wauplin left a comment

Choose a reason for hiding this comment

osanseviero commented Feb 19, 2024

julien-c left a comment

Choose a reason for hiding this comment

Wauplin commented Feb 20, 2024 • edited

mishig25 Feb 21, 2024

Choose a reason for hiding this comment

Wauplin Feb 21, 2024

Choose a reason for hiding this comment

Update conversational widget to use text-generation (+ remove `conversational` task) #457

Update conversational widget to use text-generation (+ remove `conversational` task) #457

Wauplin commented Jan 25, 2024 •

edited

xenova left a comment •

edited

Wauplin commented Feb 20, 2024 •

edited