Morrigan

OpenAI/ChatGPT Backend with conversation and API

Features

• Import documents of knowledge base from a table (CSV), save them into PostgreSQL
• Based on the title and content of the document, generate and fill in QA items in the document using some APIs
• Summarize the questions and generate corresponding vectors
• Implement high-quality Q&A using vector search
• Welcome message and preset messages
• Chat History for Conversation (based on redis)
• RESTful API
• Support text/event-stream
• Login with OAuth2 client for general Security Provider

Supported Frontend

chatgpt-svelte based on Svelte ⤸

forked: https://github.com/liut/chatgpt-svelte

chatgpt-web based on Vue.js ⤸

forked: https://github.com/liut/chatgpt-web

APIs

Get welcome message and new conversation ID

GET /api/welcome

Parameters

None

Responses

http code	content-type	response
200	application/json	{"message": "welcome message", "id": "new-cid"}

Get user information of that has been verified or signed in

GET /api/me

Parameters

None

Responses

http code	content-type	response
200	application/json	{"data": {"avatar": "", "name": "name", "uid": "uid"}}
401	application/json	{"error": "", "message": ""}

Post chat prompt and return Streaming messages

POST /api/chat-sse or /api/chat with {stream: true}

Parameters

name	type	data type	description
csid	optional	string	conversation ID
prompt	required	string	message for ask
stream	optional	bool	enable event-stream, force on /api/chat-sse

Responses

http code	content-type	response
200	text/event-stream	{"delta": "message fragments", "id": "conversation ID"}
401	application/json	{"status": "Unauthorized", "message": ""}

POST /api/chat-process for chatgpt-web only

Parameters

name	type	data type	description
prompt	required	string	message for ask
options	optional	object	{ conversationId: "" }

Responses

http code	content-type	response
200	application/octet-stream	{"delta": "message fragments", "text": "message", "conversationId": ""}
401	application/json	{"status": "Unauthorized", "message": ""}

Getting started

test -e .env || cp .env.example .env
# Edit .env and change api key of OpenAI
# Embedding and replacing frontend resources

make deps

forego start

# or

make dist

Prepare preset data file in YAML

welcome:
  content: Hello, I am your virtual assistant. How can I help you?
  role: assistant

messages:
  - content: You are a helpful assistant.
    role: system
  - content: When is my birthday?
    role: user
  - content: How would I know?
    role: assistant

 # more messages

Prepare database

CREATE USER morrigan WITH LOGIN PASSWORD 'mydbusersecret';
CREATE DATABASE morrigan WITH OWNER = morrigan ENCODING = 'UTF8';
GRANT ALL PRIVILEGES ON DATABASE morrigan to morrigan;


\c morrigan

-- optional: install extension from https://github.com/pgvector/pgvector
CREATE EXTENSION vector;

Command line usage

USAGE:
   morrigan [global options] command [command options] [arguments...]

COMMANDS:
   usage, env                   show usage
   initdb                       init database schema
   import                       import documents from a csv
   fill-qa, fillQAs             fill QA in documents
   export-qa, exportQAs         export QA from documents
   embedding, embedding-pormpt  read prompt documents and embedding
   web, run                     run a web server
   help, h                      Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h  show help

Change settings with environment

Show all local settings

go run . usage

Example:

MORRIGAN_OPENAI_API_KEY=oo-xx
MORRIGAN_HTTP_LISTEN=:3002

# optional preset data
MORRIGAN_PRESET_FILE=./data/preset.yaml

# optional OAuth2 login
MORRIGAN_AUTH_REQUIRED=true
OAUTH_PREFIX=https://portal.my-company.xyz

# optional proxy
HTTPS_PROXY=socks5://proxy.my-company.xyz:1081

The operation steps for generating data.

1. Prepare a CSV file for the corpus document.
2. Import documents.
3. Generate Questions and Anwsers from documents with Completion.
4. Generate Prompts and vector from QAs with Embedding
5. Done and go to chat

CSV template of documents

title	heading	content
my company	introducion	A great company stems from a genius idea.

go run . initdb
go run . import mycompany.csv
go run . fill-qa
go run . embedding

Attach frontend resources

1. Go to frontend project directory
2. Build frontend pages and accompanying static resources.
3. Copy them into ./htdocs

Example:

cd ../chatgpt-svelte
npm run build
rsync -a --delete dist/* ../morrigan/htdocs/
cd -

During the development and debugging phase, you can still use with proxy to collaborate with the front-end project.