Skip to content

Commit

Permalink
chore: lfg
Browse files Browse the repository at this point in the history
  • Loading branch information
dankolesnikov committed Apr 27, 2023
1 parent dd9eb89 commit 7b3ef16
Show file tree
Hide file tree
Showing 6 changed files with 12 additions and 88 deletions.
13 changes: 9 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ The app can be used in two ways:

![image](https://user-images.githubusercontent.com/122662504/234627824-2304f741-9f7b-4252-bdb4-ef2bdfd8139a.png)

- `Playground`: Input a set of documents that you want to ask questions about. Optionally, also include your own test set of question-answer pairs related to the documents; see an example [here](https://github.com/langchain-ai/auto-evaluator/tree/main/api/docs/karpathy-lex-pod). If you do not supply a test set, the app will auto-generate one. If the test set is smaller than the desired number of eval questions specified in the top left, the app will auto-generate the remainder.
- `Playground`: Input a set of documents that you want to ask questions about. Optionally, also include your own test set of question-answer pairs related to the documents; see an example [here](https://github.com/langchain-ai/auto-evaluator/tree/main/api/docs/karpathy-lex-pod). If you do not supply a test set, the app will auto-generate one. If the test set is smaller than the desired number of eval questions specified in the top left, the app will auto-generate the remainder.

![image](https://user-images.githubusercontent.com/122662504/234629201-4c17b411-f910-476b-9bf6-1246c7c5a307.png)

Expand All @@ -41,7 +41,7 @@ The app can be used in two ways:

- For each question, we use a `RetrievalQA` chain to answer it.
- This will fetch chunks that are relevant to the question from the `retriever` and pass them to the LLM.
- We expose the `QA_CHAIN_PROMPT` used for to pass this context to the LLM [here](https://github.com/langchain-ai/auto-evaluator/blob/main/api/text_utils.py).
- We expose the `QA_CHAIN_PROMPT` used for to pass this context to the LLM [here](https://github.com/langchain-ai/auto-evaluator/blob/main/api/text_utils.py).

`Model-graded evaluation`:

Expand All @@ -52,7 +52,7 @@ The app can be used in two ways:
(2) The app will evaluate the `similarity of the LLM generated answer` relative to ground truth answer.

- The prompts for both can be seen [here](https://github.com/dankolesnikov/evaluator-app/blob/main/api/text_utils.py)
- Users can select which grading prompt to use. [Here](https://rlancemartin.notion.site/Auto-Evaluator-Opportunities-7b3459dc2ae34440ae3481fe6f43ba40) are some notes in prompt selection from our experience.
- Users can select which grading prompt to use. [Here](https://rlancemartin.notion.site/Auto-Evaluator-Opportunities-7b3459dc2ae34440ae3481fe6f43ba40) are some notes in prompt selection from our experience.

`Experimental results`:

Expand Down Expand Up @@ -109,10 +109,15 @@ Test the `api` locally:
curl -X POST -F "files=@Docs/0333_text.txt" -F "num_eval_questions=1" -F "chunk_chars=1000" -F "overlap=100" -F "split_method=RecursiveTextSplitter" -F "retriever_type=similarity-search" -F "embeddings=OpenAI" -F "model_version=gpt-3.5-turbo" -F "grade_prompt=Fast" -F "num_neighbors=3" http://localhost:8000/evaluator-stream
```

Run the frontend from `nextjs` folder and view web app at specified URL (e.g., `http://localhost:3001/`):
Run the frontend from `nextjs` folder and view web app at specified URL (e.g., `http://localhost:3000/`):

`yarn dev`

### Environment Variables

`EVALUATOR_API_URL=http://127.0.0.1:8000` - used by frontend.
`OPENAI_API_KEY=` - used by backend.

## Deployment

Doppler auth:
Expand Down
Binary file added api/__pycache__/evaluator_app.cpython-38.pyc
Binary file not shown.
Binary file added api/__pycache__/text_utils.cpython-38.pyc
Binary file not shown.
2 changes: 1 addition & 1 deletion nextjs/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"private": true,
"scripts": {
"dev": "doppler run -- next dev",
"dev": "next dev",
"build": "next build",
"start": "next start"
},
Expand Down
3 changes: 2 additions & 1 deletion nextjs/utils/variables.ts
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
export const IS_DEV = process.env.NODE_ENV === "development";
export const API_URL = process.env.NEXT_PUBLIC_API_URL;
export const API_URL =
process.env.NEXT_PUBLIC_API_URL ?? process.env.EVALUATOR_API_URL;
82 changes: 0 additions & 82 deletions pages/index.tsx

This file was deleted.

1 comment on commit 7b3ef16

@vercel
Copy link

@vercel vercel bot commented on 7b3ef16 Apr 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dankolesnikov is attempting to deploy a commit to the LangChain Team on Vercel.

To accomplish this, @dankolesnikov needs to request access to the Team.

Afterwards, an owner of the Team is required to accept their membership request.

If you're already a member of the respective Vercel Team, make sure that your Personal Vercel Account is connected to your GitHub account.

Please sign in to comment.