Release 1.5.3 (#147)

* Pagination bug * Bug fix * chore: add docker cmd * Compatibility fixes for SDK version 2.0.0 (#69) * Pagination bug * Bug fix * Fix for schema changes * Render tool calling * Support for Langgraph, Qdrant & Groq (#73) * Pagination bug * Bug fix * Add langgraph support * QDrant support * Add Groq support * update README * update README * feat: optimise docker image for self host setup * adding api access to traces endpoint * clean up * refactor * feat: add clickhouse db create on app start (#79) * docs: add railway deploy, fix sdk badges (#81) * untrack .env * Revert "untrack .env" This reverts commit 4551d7e. * Playground and Prompt Management (#83) * Pagination bug * Bug fix * Playground - basic implementation * Playground - streaming and nonstreaming * Move playground inside project * API key flow * Api key * Playground refactor * Add chat hookup * anthropic streaming support * Bug fixes to openai playground * Anthropic bugfixes * Anthropic bugfix * Cohere first iteration * Cohere role fixes * Cohere api fix * Parallel running * Playground cost calculation non streaming * playground - streaming token calculation * latency and cost * Support for Groq * Add model name * Prompt management views * Remove current promptset flow * Prompt management API hooks * Prompt registry final * Playground bugfixes * Bug fix playground * Rearrange project nav * Fix playground * Fix prompts * Bugfixes * Minor fix * Prompt versioning bugfix * Bugfix * fix: clickhouse table find queries (#82) * Fix to surface multiple LLM requests inside LLM View (#84) * Pagination bug * Bug fix * Fix for surfacing multiple LLM requests in LLMView * Minor bugfixes (#86) * Pagination bug * Bug fix * Bugfixes * api to fetch promptset with prompt filters * bug fixes * fix invalid redirect * fix invalid status code * Project Switcher (#90) * Pagination bug * Bug fix * Project switcher * Feat: dataset download (#60) * API: download dataset * API: Download dataset * updated download-dataset api * Updated: download_dataset api * Updated download dataset API * Updated Download API: changed Response to Next Response, add a condition to ensure max page size is 500 * updated the download-dataset API: fixed the format and removed redundant lines of code * Updated download_daatset API: file name and removed 'id' param * Added the Download dataset button. * Merged developemnt into my branch * Updated button size * Fixes --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * Update prompt registry with instructions to fetch prompts (#91) * Pagination bug * Bug fix * Update prompt registry * Minor bugfix (#94) * Pagination bug * Bug fix * Minor bugfix * chore: update github repo badges * optimizing token count function * Add GPT4-O Pricing and Playground (#98) * Pagination bug * Bug fix * Add GPT4-O support * Add GPT4-O support * Update cost * Dylan/s3en 2234 add perplexity support to playground (#89) * adding perplexity to playground types * adding ui stuff:' * adding perplexity chat api * fixing perplexity model dropdown --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * api changes * add api access to get api and fix all bugs * bug fix * bug fix * updating descriptions to optional * prio python * cleanup and fixes * more bug fixes * more fixes * remove console log * updating trace_service functions * add migration * add format function, updating from day to hour * adding dropwdown menu * updating query key * updating query keys v2 * clean up * fix bug * Minor bugfix (#102) * Pagination bug * Bug fix * Minor bugfix * fixes * Bug fixes to Evaluations (#111) * Pagination bug * Bug fix * Bug fixes to evals v2 * Minor fix * upgrading tiktoken, adding omni to pricing (#110) Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * Update package * Dataset bugfix (#113) * Pagination bug * Bug fix * Dataset bugfix * Minor linking fix (#114) * Pagination bug * Bug fix * Dataset bugfix * Minor fix * api bug fix (#116) * fixing dropdown color (#117) * Support for Image inputs (#119) * Pagination bug * Bug fix * Support image inputs * Bug fix Spans with status_code = "ERROR" (#120) * Pagination bug * Bug fix * Support image inputs * Bug fixes * Dylan/s3en 2265 save prompt and model settings directly from playground (#99) * bug fixes, adding save prompt button in playground * adding save prompt dialog * adding create registry dialog * working registry selection * Adding new prompt to exist registry * storing * button padding fix * refactor, adding combobox * increasing dropdwon width --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * adding image display * adding property filtering (#123) * optimizing cost function * optimizing per cost per hour function * removing console statements * updating queryKey for datasets * init advanced filter dialog * adding operator combobox * updating ui * adding attribute type * bug fixes * cleanup * remove console * updating queryKeys * adding missing table checks * adding more checks * chore: update clickhouse docker for user management * style: remove blank space * updating save prompt button * Redesign of Manual Evaluations (#134) * Pagination bug * Bug fix * Schema changes for evals * Fix APIs for evals * Minor bugfixes * Bugfix * Evaluation changes * Manual eval simplification * Minor fix * Manual evaluations UX changes * Chart title fix * Disable experiments * fix * Dylan/s3en 2318 add the ability to import a conversation from the traces to (#133) * adding import conversation button * init import conversation dialog * adding trace rows * fixing scroll bug * adding import prompt to playground * fixes * Minor * Bugfixes --------- Co-authored-by: Karthik Kalyanaraman <karthik@scale3labs.com> * storing * remove import for images * importing attribute stuff * fixing operations display * cleaning up advanced filters * cleanup * docs: update docker instructions * removing console, fixing button * add stars history (#139) * Dylan/s3en 2401 ability to filter by user for metrics and traces (#138) * adding user id combobox, api * fixing sql query * removing user filter on deselect{ * moving user combobox into filter dialog * adding filtering by user for token charts * bug fixes, more userId route, function updates * latency filtering * updating to alphanumeric * removing console, comment * updating default filter operation to and * adding clear all button, cleanup * Dylan/s3en 2350 aggregate metrics by models v2 (#141) * adding user id combobox, api * fixing sql query * removing user filter on deselect{ * moving user combobox into filter dialog * adding filtering by user for token charts * bug fixes, more userId route, function updates * latency filtering * updating to alphanumeric * removing console, comment * updating default filter operation to and * adding clear all button, cleanup * init commit model combobox * adding api, updating function, adding to metrics page * adding model to charts, apis, service * remove consoles * Automated Evaluations - Support for Inspect AI (#142) * Pagination bug * Bug fix * Experiments view * re-order * Comparison view * Experiment report * Experiments - hook up APIs * Experiments pagination * deduplicate paginated data * Fix loading and error states - experiments * Rename experiments and evaluations * Add loading spinner * Link evaluation * Link to evals * Fixes * Minor UI fixes * Bugfixes * DSPy support (#144) * Pagination bug * Bug fix * Bugfixes * DSPy support * Update README * Prompt registry and Playground integration (#146) * Pagination bug * Bug fix * Bugfixes * Playground enhancements * Fix * Playground improvment * Bug fix --------- Co-authored-by: Darshit Suratwala <darshit@scale3labs.com> Co-authored-by: darshit-s3 <119623510+darshit-s3@users.noreply.github.com> Co-authored-by: dylan <dylan@scale3labs.com> Co-authored-by: dylanzuber-scale3 <116033320+dylanzuber-scale3@users.noreply.github.com> Co-authored-by: Rohit Kadhe <rohit@scale3labs.com> Co-authored-by: Rohit Kadhe <113367036+rohit-kadhe@users.noreply.github.com> Co-authored-by: MayuriS24 <159064413+MayuriS24@users.noreply.github.com> Co-authored-by: yemiscale3 <111706151+yemiadej@users.noreply.github.com>
Scale3-Labs · Jun 20, 2024 · bbb6a2b · bbb6a2b
1 parent e16d627
commit bbb6a2b
Show file tree

Hide file tree

Showing 12 changed files with 714 additions and 352 deletions.
diff --git a/app/(protected)/project/[project_id]/prompts/[prompt_id]/page.tsx b/app/(protected)/project/[project_id]/prompts/[prompt_id]/page.tsx
@@ -2,10 +2,10 @@
 import CreatePromptDialog from "@/components/shared/create-prompt-dialog";
 import { PromptInstructions } from "@/components/shared/setup-instructions";
 import { Button } from "@/components/ui/button";
-import { Checkbox } from "@/components/ui/checkbox";
 import { Label } from "@/components/ui/label";
 import { Separator } from "@/components/ui/separator";
 import { Skeleton } from "@/components/ui/skeleton";
+import { Switch } from "@/components/ui/switch";
 import { cn } from "@/lib/utils";
 import { Prompt } from "@prisma/client";
 import CodeEditor from "@uiw/react-textarea-code-editor";
@@ -149,6 +149,48 @@ export default function Page() {
             ))}
           </div>
           <div className="flex flex-col gap-8 w-full">
+            <div className="flex flex-col gap-2">
+              <Label>Go Live</Label>
+              <div className="flex items-center gap-2 w-fit">
+                <Switch
+                  checked={live}
+                  onCheckedChange={async (checked) => {
+                    setLive(checked as boolean);
+                    try {
+                      const payload = {
+                        ...selectedPrompt,
+                        live: checked,
+                      };
+                      await fetch("/api/prompt", {
+                        method: "PUT",
+                        headers: {
+                          "Content-Type": "application/json",
+                        },
+                        body: JSON.stringify(payload),
+                      });
+                      await queryClient.invalidateQueries({
+                        queryKey: ["fetch-prompts-query", promptsetId],
+                      });
+                      toast.success(
+                        checked
+                          ? "This prompt is now live"
+                          : "This prompt is no longer live. Make sure to make another prompt live"
+                      );
+                    } catch (error) {
+                      toast.error("Failed to make prompt live", {
+                        description:
+                          error instanceof Error
+                            ? error.message
+                            : String(error),
+                      });
+                    }
+                  }}
+                />
+                <p className="text-sm text-muted-foreground">
+                  Make this version of the prompt live
+                </p>
+              </div>
+            </div>
             <div className="flex flex-col gap-2">
               <Label>Prompt Registry ID</Label>
               <p className="p-2 rounded-md border-2 border-muted font-semibold text-md">
@@ -194,48 +236,6 @@ export default function Page() {
                 }}
               />
             </div>
-            <div className="flex flex-col gap-2">
-              <Label>Go Live</Label>
-              <div className="flex items-center gap-2 w-fit">
-                <Checkbox
-                  checked={live}
-                  onCheckedChange={async (checked) => {
-                    setLive(checked as boolean);
-                    try {
-                      const payload = {
-                        ...selectedPrompt,
-                        live: checked,
-                      };
-                      await fetch("/api/prompt", {
-                        method: "PUT",
-                        headers: {
-                          "Content-Type": "application/json",
-                        },
-                        body: JSON.stringify(payload),
-                      });
-                      await queryClient.invalidateQueries({
-                        queryKey: ["fetch-prompts-query", promptsetId],
-                      });
-                      toast.success(
-                        checked
-                          ? "This prompt is now live"
-                          : "This prompt is no longer live. Make sure to make another prompt live"
-                      );
-                    } catch (error) {
-                      toast.error("Failed to make prompt live", {
-                        description:
-                          error instanceof Error
-                            ? error.message
-                            : String(error),
-                      });
-                    }
-                  }}
-                />
-                <p className="text-sm text-muted-foreground">
-                  Make this version of the prompt live
-                </p>
-              </div>
-            </div>
             <div className="flex flex-col gap-2">
               <Label>Use the Live prompt directly in your code</Label>
               <PromptInstructions id={promptsetId} />

diff --git a/app/(protected)/settings/keys/page.tsx b/app/(protected)/settings/keys/page.tsx
@@ -8,12 +8,6 @@ export default async function Page() {
   if (!session || !session.user) {
     redirect("/login");
   }
-  const email = session?.user?.email as string;
-
-  const resp = await fetch(
-    `${process.env.NEXTAUTH_URL_INTERNAL}/api/user?email=${email}`
-  );
-  const user = await resp.json();
 
   return (
     <>

diff --git a/app/api/promptset/route.ts b/app/api/promptset/route.ts
@@ -1,6 +1,10 @@
 import { authOptions } from "@/lib/auth/options";
 import prisma from "@/lib/prisma";
-import { authApiKey, fillPromptStringTemplate, parseQueryString } from "@/lib/utils";
+import {
+  authApiKey,
+  fillPromptStringTemplate,
+  parseQueryString,
+} from "@/lib/utils";
 import { getServerSession } from "next-auth";
 import { redirect } from "next/navigation";
 import { NextRequest, NextResponse } from "next/server";
@@ -25,7 +29,10 @@ export async function GET(req: NextRequest) {
         include: {
           Prompt: {
             where: {
-              OR: version !== undefined ? [{ version: version }] : [{ live: true }],
+              OR:
+                version !== undefined
+                  ? [{ version: version }]
+                  : [{ live: true }],
             },
           },
         },
@@ -57,26 +64,39 @@ export async function GET(req: NextRequest) {
 
         livePromptVariables.forEach((key) => {
           const value =
-            variables !== null ? variables[key as keyof typeof variables] ?? "" : "";
-          if (!variablesSet.has(`${key},${value.length > 0 ? value : "undefined"}`)) {
+            variables !== null
+              ? variables[key as keyof typeof variables] ?? ""
+              : "";
+          if (
+            !variablesSet.has(
+              `${key},${value.length > 0 ? value : "undefined"}`
+            )
+          ) {
             errors.push(key);
           }
         });
         if (errors.length > 0) {
           const moreThanOneError = errors.length > 1;
           return NextResponse.json(
             {
-              error: `${moreThanOneError ? "Variables" : "Variable"} ${errors.join(", ")} ${moreThanOneError ? "are" : "is"} missing`,
+              error: `${
+                moreThanOneError ? "Variables" : "Variable"
+              } ${errors.join(", ")} ${
+                moreThanOneError ? "are" : "is"
+              } missing`,
             },
             { status: 400 }
           );
         }
-        prompts[0].value = fillPromptStringTemplate(prompts[0].value, variables as Record<string, string>);
+        prompts[0].value = fillPromptStringTemplate(
+          prompts[0].value,
+          variables as Record<string, string>
+        );
       }
       return NextResponse.json({
         ...promptSet,
         Prompt: undefined,
-        prompts: prompts
+        prompts: prompts,
       });
     } else {
       const session = await getServerSession(authOptions);

diff --git a/components/playground/common.tsx b/components/playground/common.tsx
@@ -1,9 +1,8 @@
-"use client";
-
 import PromptRegistryDialog from "@/components/playground/prompt-registry-dialog";
 import LLMPicker from "@/components/shared/llm-picker";
 import { Button } from "@/components/ui/button";
 import { Separator } from "@/components/ui/separator";
+import { LLM_VENDORS, LLM_VENDOR_APIS } from "@/lib/constants";
 import {
   CohereAIRole,
   Conversation,
@@ -12,6 +11,7 @@ import {
 import { cn } from "@/lib/utils";
 import { ArrowTopRightIcon } from "@radix-ui/react-icons";
 import { MinusCircleIcon, PlusIcon } from "lucide-react";
+import Link from "next/link";
 import { useEffect, useRef, useState } from "react";
 import { useQuery } from "react-query";
 import { toast } from "sonner";
@@ -84,7 +84,7 @@ export function ExpandingTextArea({
         ref={textAreaRef}
         defaultValue={value}
         onChange={handleChange}
-        style={{ overflowY: "auto", resize: "none", height: "auto" }}
+        style={{ overflowY: "auto", height: "auto" }}
       />
       <div className="absolute right-0 top-[-10px] py-2">
         <Button
@@ -223,13 +223,45 @@ export function Message({
 
 export function AddLLMChat({ onAdd }: { onAdd: (vendor: string) => void }) {
   const [vendor, setVendor] = useState("");
+  const [busy, setBusy] = useState(false);
+  const [vendorKey, setVendorKey] = useState<string>("");
+
+  useEffect(() => {
+    if (typeof window === "undefined") return;
+    const keys = LLM_VENDORS.map((vendor) => {
+      const keyName = LLM_VENDOR_APIS.find(
+        (api) => api.label.toUpperCase() === vendor.value.toUpperCase()
+      );
+      if (!keyName) return null;
+      const key = window.localStorage.getItem(keyName.value.toUpperCase());
+      if (!key) return null;
+      return { value: keyName.value.toUpperCase(), label: vendor.label, key };
+    });
+    const venKey = keys.find((key) => key?.label.toLowerCase() === vendor);
+    setVendorKey(venKey?.key || "");
+  }, [busy, vendor]);
+
   return (
-    <div className="w-[450px] h-[600px] rounded-lg border border-dashed border-muted-foreground flex items-center justify-center gap-1">
-      <LLMPicker setVendor={setVendor} />
-      <Button size={"sm"} onClick={() => onAdd(vendor)}>
-        Add
-        <PlusIcon className="ml-2 h-4 w-4" />
-      </Button>
+    <div className="w-[450px] h-[600px] rounded-lg border border-dashed border-muted-foreground flex flex-col items-center justify-center gap-4">
+      <div className="flex items-center gap-2">
+        <LLMPicker setVendor={setVendor} />
+        <Button
+          size={"sm"}
+          onClick={() => onAdd(vendor)}
+          disabled={vendorKey === "" || vendor === "" || busy}
+        >
+          Add
+          <PlusIcon className="ml-2 h-4 w-4" />
+        </Button>
+      </div>
+      {vendorKey === "" && (
+        <span className="text-sm font-semibold w-72">
+          Please add your {vendor} API key in the{" "}
+          <Link href="/settings/keys" className="underline text-blue-400">
+            settings page.
+          </Link>
+        </span>
+      )}
     </div>
   );
 }
diff --git a/components/playground/import-messages.tsx b/components/playground/import-messages.tsx
@@ -0,0 +1,36 @@
+import {
+  DropdownMenu,
+  DropdownMenuContent,
+  DropdownMenuTrigger,
+} from "@/components/ui/dropdown-menu";
+import { cn } from "@/lib/utils";
+import { DotsHorizontalIcon } from "@radix-ui/react-icons";
+import { Button } from "../ui/button";
+import ImportPrompt from "./import-prompt";
+import ImportTraceConversation from "./import-trace-conversation";
+
+export default function ImportMessages({
+  setMessages,
+  className = "",
+}: {
+  setMessages: any;
+  className?: string;
+}) {
+  return (
+    <DropdownMenu>
+      <DropdownMenuTrigger asChild>
+        <Button
+          variant="outline"
+          size={"icon"}
+          className={cn(className, "bg-muted")}
+        >
+          <DotsHorizontalIcon />
+        </Button>
+      </DropdownMenuTrigger>
+      <DropdownMenuContent className="w-full flex flex-col gap-1">
+        <ImportTraceConversation setMessages={setMessages} />
+        <ImportPrompt setMessages={setMessages} />
+      </DropdownMenuContent>
+    </DropdownMenu>
+  );
+}