Fix plugins

[olliestanley]

Plugins was compatible only with text-generation-inference based workers and therefore worked on Dragan's machines but did not work on OA prod. This resolves the incompatibility.

[olliestanley]

Marking this as draft as Dragan has identified another problem which will need to be remedied for plugins to work. No point merging this until that is fixed

[andreaskoepf]

lgtm!

[pevogam]

@olliestanley Can you elaborate a bit more on what is being fixed or extended here regarding the original pull request? I noticed the merged original pull request no longer introduces the sample calculator plugin, what are the plans regarding this?

[olliestanley]

@olliestanley Can you elaborate a bit more on what is being fixed or extended here regarding the original pull request? I noticed the merged original pull request no longer introduces the sample calculator plugin, what are the plans regarding this?

There were three problems in prod which did not show up in local testing, because prod uses basic_hf_server worker whereas testing was using text-generation-inference worker:

• the chain code uses generate endpoint which only exists in text-generation-inference rather than generate_stream
• basic HF server does not support custom stopping sequences
• basic HF server does not support fixed random seeds

This PR resolves all three of those

[flozi00]

Am i missing something to get it working ?

{
  "model_config": {
    "model_id": "OpenAssistant/oasst-sft-7-llama-30b",
    "max_input_length": 1024,
    "max_total_length": 1792,
    "quantized": false
  },
  "sampling_parameters": {
    "top_k": 50,
    "top_p": null,
    "typical_p": null,
    "temperature": 0.35,
    "repetition_penalty": 1.1111111111111112,
    "max_new_tokens": 1024
  },
  "do_sample": true,
  "plugins": [
    {
      "url": "https://www.klarna.com/.well-known/ai-plugin.json",
      "enabled": true,
      "plugin_config": {
        "schema_version": "v1",
        "name_for_model": "KlarnaProducts",
        "name_for_human": "Klarna Shopping",
        "description_for_human": "Search and compare prices from thousands of online shops.",
        "description_for_model": "Assistant uses the Klarna plugin to get relevant product suggestions for any shopping or product discovery purpose. Assistant will reply with the following 3 paragraphs 1) Search Results 2) Product Comparison of the Search Results 3) Followup Questions. The first paragraph contains a list of the products with their attributes listed clearly and concisely as bullet points under the product, together with a link to the product and an explanation. Links will always be returned and should be shown to the user. The second paragraph compares the results returned in a summary sentence starting with \"In summary\". Assistant comparisons consider only the most important features of the products that will help them fit the users request, and each product mention is brief, short and concise. In the third paragraph assistant always asks helpful follow-up questions and end with a question mark. When assistant is asking a follow-up question, it uses it's product expertise to provide information pertaining to the subject of the user's request that may guide them in their search for the right product.",
        "api": {
          "type": "openapi",
          "url": "https://www.klarna.com/us/shopping/public/openai/v0/api-docs/",
          "has_user_authentication": false,
          "is_user_authenticated": false
        },
        "auth": {
          "type": "none"
        },
        "logo_url": "https://www.klarna.com/assets/sites/5/2020/04/27143923/klarna-K-150x150.jpg",
        "contact_email": "openai-products@klarna.com",
        "legal_info_url": "https://www.klarna.com/us/legal/",
        "endpoints": null
      },
      "trusted": false
    }
  ]
}