feat: support RAG

chore: Update new model.json with multiple binaries feat: Add updates for handling multiple model binaries chore: jan can see Update Model.json (#1005) * add(mixtral): add model.json for mixtral * archived some models + update the model.json * add(model): add pandora 10.7b * fix(model): update description * fix(model): pump vers and change the featured model to trinity * fix(model): archive neuralchat * fix(model): decapriated all old models * fix(trinity): add cover image and change description * fix(trinity): update cover png * add(pandora): cover image * fix(pandora): cover image * add(mixtral): add model.json for mixtral * archived some models + update the model.json * add(model): add pandora 10.7b * fix(model): update description * fix(model): pump vers and change the featured model to trinity * fix(model): archive neuralchat * fix(model): decapriated all old models * fix(trinity): add cover image and change description * fix(trinity): update cover png * add(pandora): cover image * fix(pandora): cover image * chore: model desc nits * fix(models): adjust the size for solars and pandoras * add(mixtral): description --------- Co-authored-by: 0xSage <n@pragmatic.vc> chore: reformat model.json and use new template fix(Model): download/abort model (#1163) * fix(Model): download/abort model * fix: image preview Signed-off-by: James <james@jan.ai> --------- Signed-off-by: James <james@jan.ai> Co-authored-by: James <james@jan.ai> Co-authored-by: Louis <louis@jan.ai> add preview and reduce time re-render all chat screen Signed-off-by: James <james@jan.ai> store files under thread_id/files Signed-off-by: James <james@jan.ai> fix: Update llava 1.5 size fix: Nitro extension path resolver feat: Add upload preview clearance chore: Update FileType to multiple targets fix: delete file preview once new thread created chore: Add langchain import support storing pdf file Signed-off-by: James <james@jan.ai> feat: add retrieval tool in node runtime fix: import module done Co-authored-by: Louis <louis-jan@users.noreply.github.com> feat: Add type assistant tool chore: Add tool_retrieval_enabled to InferenceEngine chore: Add AssistantTool to thread entity chore: refactor tool retrieval base class feat: Add handler for assistant with rag enabled chore: Update inferenceEngine type properly chore: Update inferenceEngine type properly fix: Update retrieval tool chore: main entry correction refactor: tsconfig files chore: Update ModelRuntimeParams type refactor: Remove unused files fix: wip chore: remove unused console.log for FileUploadPreview fix: Update mapping correctly for engine and proxyEngine feat: Add proxyEngine to type ModelInfo fix: WIP with test route fix: Add bundleDependencies to package.json chore: remove conversational history memory fix: refactor data passing reafactor: remove unused code fix: Update module chore: export import correction fix conflict Signed-off-by: James <james@jan.ai> fix: resolve after rebased fix: Update llava 1.5 model json feat: Add bakllava 1 model json refactor: node module export, ES syntax and langchain import fix: WIP fix: WIP fix: WIP fix: external module import fix: WIP Add UI attachment fot file upload Prepare Thumbnail UI image chore: rebase model folder to dev branch chore: remove multiple binaries related commits fix: remove multiple binaries related commits part 2 fix: Remove transformer.js related deps Fix truncate file name attachment remove unused code image preview attachment fix: remove multi binaries error chore: remove commented code for ModelArtifacts type Dropzone for drag and drop attachment Avoid conditional showing 0 using length fix symbol windows avoid undefined tools fix: add tool retrieval to true by default and disable the change chore: remove unused code fix: Enable nitro embedding by default fix: Update code WIP with nitro embedding chore: remove unused running function fix: assistant extension missing module feat: Retrieval ingest, query and reforward fix: Update hnswlib version conflict fix: Add tool settings fix: Update path to thread_id/memory fix: Add support for nitro embedding usage fix: RAG does not work with plain content message fix(Model): #1662 imported model does not use gpu (#1723) Signed-off-by: James <james@jan.ai> Co-authored-by: James <james@jan.ai> feat: allow users to update retrieval settings chore: pass thread assistant settings to assistant extensions chore: eslint fix fix bug border right panel showing in thread while not have active thread Update setting layout retrieval assistant Renaming file settingcomponent change default value in core extention add fake loader generate response fix conditional fake loader remove unused import Proper error message on file typr fix: loading indicator fix: chunk size and overlap constraint conditional drag and drop when retrieval off fix: enable retrieval middleware as soon as its tool is enabled fix: configure embedding engine according to request fix: Retrieval false by default fix: engine json chore: migrate assistant disabled collapse panel when retrieval or children null chore: remove unused log chore: Bump nitro version to 0.2.14 for batch embedding chore: remove unused console.log
janhq · Jan 26, 2024 · 28e4405 · 28e4405
1 parent 38f757d
commit 28e4405
Show file tree

Hide file tree

Showing 68 changed files with 1,953 additions and 519 deletions.
diff --git a/core/src/api/index.ts b/core/src/api/index.ts
@@ -62,6 +62,7 @@ export enum FileManagerRoute {
   getJanDataFolderPath = 'getJanDataFolderPath',
   getResourcePath = 'getResourcePath',
   fileStat = 'fileStat',
+  writeBlob = 'writeBlob',
 }
 
 export type ApiFunction = (...args: any[]) => any

diff --git a/core/src/fs.ts b/core/src/fs.ts
@@ -1,11 +1,20 @@
-import { FileStat } from "./types"
+import { FileStat } from './types'
 
 /**
  * Writes data to a file at the specified path.
  * @returns {Promise<any>} A Promise that resolves when the file is written successfully.
  */
 const writeFileSync = (...args: any[]) => global.core.api?.writeFileSync(...args)
 
+/**
+ * Writes blob data to a file at the specified path.
+ * @param path - The path to file.
+ * @param data - The blob data.
+ * @returns
+ */
+const writeBlob: (path: string, data: string) => Promise<any> = (path, data) =>
+  global.core.api?.writeBlob(path, data)
+
 /**
  * Reads the contents of a file at the specified path.
  * @returns {Promise<any>} A Promise that resolves with the contents of the file.
@@ -60,7 +69,6 @@ const syncFile: (src: string, dest: string) => Promise<any> = (src, dest) =>
  */
 const copyFileSync = (...args: any[]) => global.core.api?.copyFileSync(...args)
 
-
 /**
  * Gets the file's stats.
  *
@@ -70,7 +78,6 @@ const copyFileSync = (...args: any[]) => global.core.api?.copyFileSync(...args)
 const fileStat: (path: string) => Promise<FileStat | undefined> = (path) =>
   global.core.api?.fileStat(path)
 
-
 // TODO: Export `dummy` fs functions automatically
 // Currently adding these manually
 export const fs = {
@@ -84,5 +91,6 @@ export const fs = {
   appendFileSync,
   copyFileSync,
   syncFile,
-  fileStat
+  fileStat,
+  writeBlob,
 }
diff --git a/core/src/node/api/routes/fs.ts b/core/src/node/api/routes/fs.ts
@@ -2,6 +2,7 @@ import { FileSystemRoute } from '../../../api'
 import { join } from 'path'
 import { HttpServer } from '../HttpServer'
 import { getJanDataFolderPath } from '../../utils'
+import { normalizeFilePath } from '../../path'
 
 export const fsRouter = async (app: HttpServer) => {
   const moduleName = 'fs'
@@ -13,10 +14,10 @@ export const fsRouter = async (app: HttpServer) => {
         const result = await import(moduleName).then((mdl) => {
           return mdl[route](
             ...body.map((arg: any) =>
-              typeof arg === 'string' && arg.includes('file:/')
-                ? join(getJanDataFolderPath(), arg.replace('file:/', ''))
-                : arg,
-            ),
+              typeof arg === 'string' && (arg.startsWith(`file:/`) || arg.startsWith(`file:\\`))
+                ? join(getJanDataFolderPath(), normalizeFilePath(arg))
+                : arg
+            )
           )
         })
         res.status(200).send(result)

diff --git a/core/src/types/assistant/assistantEntity.ts b/core/src/types/assistant/assistantEntity.ts
@@ -2,6 +2,13 @@
  * Assistant type defines the shape of an assistant object.
  * @stored
  */
+
+export type AssistantTool = {
+  type: string
+  enabled: boolean
+  settings: any
+}
+
 export type Assistant = {
   /** Represents the avatar of the user. */
   avatar: string
@@ -22,7 +29,7 @@ export type Assistant = {
   /** Represents the instructions for the object. */
   instructions?: string
   /** Represents the tools associated with the object. */
-  tools?: any
+  tools?: AssistantTool[]
   /** Represents the file identifiers associated with the object. */
   file_ids: string[]
   /** Represents the metadata of the object. */

diff --git a/core/src/types/inference/inferenceEntity.ts b/core/src/types/inference/inferenceEntity.ts
@@ -1,3 +1,5 @@
+import { ContentType, ContentValue } from '../message'
+
 /**
  * The role of the author of this message.
  */
@@ -13,7 +15,32 @@ export enum ChatCompletionRole {
  */
 export type ChatCompletionMessage = {
   /** The contents of the message. **/
-  content?: string
+  content?: ChatCompletionMessageContent
   /** The role of the author of this message. **/
   role: ChatCompletionRole
 }
+
+export type ChatCompletionMessageContent =
+  | string
+  | (ChatCompletionMessageContentText &
+      ChatCompletionMessageContentImage &
+      ChatCompletionMessageContentDoc)[]
+
+export enum ChatCompletionMessageContentType {
+  Text = 'text',
+  Image = 'image_url',
+  Doc = 'doc_url',
+}
+
+export type ChatCompletionMessageContentText = {
+  type: ChatCompletionMessageContentType
+  text: string
+}
+export type ChatCompletionMessageContentImage = {
+  type: ChatCompletionMessageContentType
+  image_url: { url: string }
+}
+export type ChatCompletionMessageContentDoc = {
+  type: ChatCompletionMessageContentType
+  doc_url: { url: string }
+}
diff --git a/core/src/types/message/messageEntity.ts b/core/src/types/message/messageEntity.ts
@@ -1,5 +1,6 @@
 import { ChatCompletionMessage, ChatCompletionRole } from '../inference'
 import { ModelInfo } from '../model'
+import { Thread } from '../thread'
 
 /**
  * The `ThreadMessage` type defines the shape of a thread's message object.
@@ -35,7 +36,10 @@ export type ThreadMessage = {
 export type MessageRequest = {
   id?: string
 
-  /** The thread id of the message request. **/
+  /**
+   * @deprecated Use thread object instead
+   * The thread id of the message request.
+   */
   threadId: string
 
   /**
@@ -48,6 +52,10 @@ export type MessageRequest = {
 
   /** Settings for constructing a chat completion request **/
   model?: ModelInfo
+
+  /** The thread of this message is belong to. **/
+  // TODO: deprecate threadId field
+  thread?: Thread
 }
 
 /**
@@ -62,7 +70,7 @@ export enum MessageStatus {
   /** Message loaded with error. **/
   Error = 'error',
   /** Message is cancelled streaming */
-  Stopped = "stopped"
+  Stopped = 'stopped',
 }
 
 /**
@@ -71,6 +79,7 @@ export enum MessageStatus {
 export enum ContentType {
   Text = 'text',
   Image = 'image',
+  Pdf = 'pdf',
 }
 
 /**
@@ -80,6 +89,8 @@ export enum ContentType {
 export type ContentValue = {
   value: string
   annotations: string[]
+  name?: string
+  size?: number
 }
 
 /**

diff --git a/core/src/types/model/modelEntity.ts b/core/src/types/model/modelEntity.ts
@@ -7,6 +7,7 @@ export type ModelInfo = {
   settings: ModelSettingParams
   parameters: ModelRuntimeParams
   engine?: InferenceEngine
+  proxyEngine?: InferenceEngine
 }
 
 /**
@@ -18,7 +19,8 @@ export enum InferenceEngine {
   nitro = 'nitro',
   openai = 'openai',
   triton_trtllm = 'triton_trtllm',
-  hf_endpoint = 'hf_endpoint',
+
+  tool_retrieval_enabled = 'tool_retrieval_enabled',
 }
 
 export type ModelArtifact = {
@@ -90,6 +92,13 @@ export type Model = {
    * The model engine.
    */
   engine: InferenceEngine
+
+  proxyEngine?: InferenceEngine
+
+  /**
+   * Is multimodal or not.
+   */
+  visionModel?: boolean
 }
 
 export type ModelMetadata = {
@@ -129,4 +138,5 @@ export type ModelRuntimeParams = {
   stop?: string[]
   frequency_penalty?: number
   presence_penalty?: number
+  engine?: string
 }
diff --git a/core/src/types/thread/index.ts b/core/src/types/thread/index.ts
@@ -1,2 +1,3 @@
 export * from './threadEntity'
 export * from './threadInterface'
+export * from './threadEvent'
diff --git a/core/src/types/thread/threadEntity.ts b/core/src/types/thread/threadEntity.ts
@@ -1,3 +1,4 @@
+import { AssistantTool } from '../assistant'
 import { ModelInfo } from '../model'
 
 /**
@@ -30,6 +31,7 @@ export type ThreadAssistantInfo = {
   assistant_name: string
   model: ModelInfo
   instructions?: string
+  tools?: AssistantTool[]
 }
 
 /**

diff --git a/core/src/types/thread/threadEvent.ts b/core/src/types/thread/threadEvent.ts
@@ -0,0 +1,4 @@
+export enum ThreadEvent {
+  /** The `OnThreadStarted` event is emitted when a thread is started. */
+  OnThreadStarted = 'OnThreadStarted',
+}
diff --git a/electron/handlers/fileManager.ts b/electron/handlers/fileManager.ts
@@ -59,4 +59,20 @@ export function handleFileMangerIPCs() {
       return fileStat
     }
   )
+
+  ipcMain.handle(
+    FileManagerRoute.writeBlob,
+    async (_event, path: string, data: string): Promise<void> => {
+      try {
+        const normalizedPath = normalizeFilePath(path)
+        const dataBuffer = Buffer.from(data, 'base64')
+        fs.writeFileSync(
+          join(getJanDataFolderPath(), normalizedPath),
+          dataBuffer
+        )
+      } catch (err) {
+        console.error(`writeFile ${path} result: ${err}`)
+      }
+    }
+  )
 }
diff --git a/electron/handlers/fs.ts b/electron/handlers/fs.ts
@@ -1,9 +1,9 @@
 import { ipcMain } from 'electron'
 
-import { FileSystemRoute } from '@janhq/core'
-import { join } from 'path'
 import { getJanDataFolderPath, normalizeFilePath } from '@janhq/core/node'
-
+import fs from 'fs'
+import { FileManagerRoute, FileSystemRoute } from '@janhq/core'
+import { join } from 'path'
 /**
  * Handles file system operations.
  */
@@ -15,7 +15,7 @@ export function handleFsIPCs() {
         mdl[route](
           ...args.map((arg) =>
             typeof arg === 'string' &&
-            (arg.includes(`file:/`) || arg.includes(`file:\\`))
+            (arg.startsWith(`file:/`) || arg.startsWith(`file:\\`))
               ? join(getJanDataFolderPath(), normalizeFilePath(arg))
               : arg
           )

diff --git a/extensions/assistant-extension/package.json b/extensions/assistant-extension/package.json
@@ -3,26 +3,46 @@
   "version": "1.0.0",
   "description": "This extension enables assistants, including Jan, a default assistant that can call all downloaded models",
   "main": "dist/index.js",
-  "module": "dist/module.js",
+  "node": "dist/node/index.js",
   "author": "Jan <service@jan.ai>",
   "license": "AGPL-3.0",
   "scripts": {
-    "build": "tsc -b . && webpack --config webpack.config.js",
+    "build": "tsc --module commonjs && rollup -c rollup.config.ts",
     "build:publish": "rimraf *.tgz --glob && npm run build && npm pack && cpx *.tgz ../../electron/pre-install"
   },
   "devDependencies": {
+    "@rollup/plugin-commonjs": "^25.0.7",
+    "@rollup/plugin-json": "^6.1.0",
+    "@rollup/plugin-node-resolve": "^15.2.3",
+    "@rollup/plugin-replace": "^5.0.5",
+    "@types/pdf-parse": "^1.1.4",
+    "cpx": "^1.5.0",
     "rimraf": "^3.0.2",
-    "webpack": "^5.88.2",
-    "webpack-cli": "^5.1.4"
+    "rollup": "^2.38.5",
+    "rollup-plugin-define": "^1.0.1",
+    "rollup-plugin-sourcemaps": "^0.6.3",
+    "rollup-plugin-typescript2": "^0.36.0",
+    "typescript": "^5.3.3"
   },
   "dependencies": {
     "@janhq/core": "file:../../core",
+    "@langchain/community": "0.0.13",
+    "hnswlib-node": "^1.4.2",
+    "langchain": "^0.0.214",
     "path-browserify": "^1.0.1",
+    "pdf-parse": "^1.1.1",
     "ts-loader": "^9.5.0"
   },
   "files": [
     "dist/*",
     "package.json",
     "README.md"
+  ],
+  "bundleDependencies": [
+    "@janhq/core",
+    "@langchain/community",
+    "hnswlib-node",
+    "langchain",
+    "pdf-parse"
   ]
 }