Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,7 @@ coverage/

# Logs
logs/
*.log
*.log

# Comprehensive testing examples internal use only
examples/comprehensive-testing.ts
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## Overview

`quotientai` is an SDK and CLI for logging data to [Quotient](https://quotientai.co), running hallucination and document attribution detections for retrieval and search-augmented AI systems, and **automatically tracing AI/ML applications**.
`quotientai` is an SDK and CLI for logging data to [Quotient](https://quotientai.co), running hallucination and document relevancy detections for retrieval and search-augmented AI systems, and **automatically tracing AI/ML applications**.

## Installation

Expand All @@ -14,14 +14,14 @@ npm install quotientai

## Usage

### Logging and Hallucination Detection
### Logging and Detection

Create an API key on [Quotient](https://app.quotientai.co) and set it as an environment variable called `QUOTIENT_API_KEY`. Then follow the examples below or see our [docs](https://docs.quotientai.co) for a more comprehensive walkthrough.

Send your first log and detect hallucinations. Run the code below and see your Logs and Detections on your [Quotient Dashboard](https://app.quotientai.co/dashboard).
Send your first log and run detections (hallucination detection, document relevancy). Run the code below and see your Logs and Detections on your [Quotient Dashboard](https://app.quotientai.co/dashboard).

```typescript
import { QuotientAI } from 'quotientai';
import { QuotientAI, DetectionType } from 'quotientai';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

quotientai/types for DetectionType?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i see we can have both?


const quotient = new QuotientAI(apiKey?: string);

Expand All @@ -30,12 +30,12 @@ const quotientLogger = quotient.logger.init({
appName: 'my-app',
environment: 'dev',
sampleRate: 1.0,
hallucinationDetection: true,
hallucinationDetectionSampleRate: 1.0,
detections: [DetectionType.HALLUCINATION, DetectionType.DOCUMENT_RELEVANCY],
detectionSampleRate: 1.0,
});

// create a log
const logId = await quotientLogger.log({
const logId = await quotient.log({
userQuery: 'How do I cook a goose?',
modelOutput: 'The capital of France is Paris',
documents: ['Here is an excellent goose recipe...'],
Expand Down
12 changes: 5 additions & 7 deletions examples/example_logs.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
import { QuotientAI } from 'quotientai';
import { QuotientAI, DetectionType } from 'quotientai';

async function main() {
const quotient = new QuotientAI();
console.log('QuotientAI client initialized');

// configure the logger
const quotientLogger = quotient.logger.init({
quotient.logger.init({
appName: 'my-app',
environment: 'dev',
sampleRate: 1.0,
tags: { model: 'gpt-4o', feature: 'customer-support' },
hallucinationDetection: true,
hallucinationDetectionSampleRate: 1.0,
detections: [DetectionType.HALLUCINATION, DetectionType.DOCUMENT_RELEVANCY],
detectionSampleRate: 1.0,
});

console.log('Logger initialized');
Expand All @@ -38,13 +38,11 @@ async function main() {
'You are a helpful assistant that answers questions about the world.',
"Answer the question in a concise manner. If you are not sure, say 'I don't know'.",
],
hallucinationDetection: true,
inconsistencyDetection: true,
});
console.log('pollForDetections with logId: ', logId);

// poll for the detection results
const detectionResults = await quotientLogger.pollForDetections(logId);
const detectionResults = await quotientLogger.pollForDetection(logId);
console.log('detectionResults', detectionResults);
} catch (error) {
console.error(error);
Expand Down
22 changes: 16 additions & 6 deletions quotientai/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,18 +53,27 @@ export class QuotientAI {
}

/**
* Log the model interaction.
*
* Args:
* user_query: The user's input query
* model_output: The model's response
* # New detection parameters (recommended):
* detections: List of detection types to run (replaces hallucinationDetection/inconsistencyDetection)
* detectionSampleRate: Sample rate for all detections 0-1 (replaces hallucinationDetectionSampleRate)
*
* # Deprecated detection parameters:
* hallucinationDetection: [DEPRECATED in 0.0.9] Use detections=[DetectionType.HALLUCINATION] instead
* inconsistencyDetection: [DEPRECATED in 0.0.9] Use detections=[DetectionType.INCONSISTENCY] instead
*
* # Common input parameters:
* userQuery: The user's input query
* modelOutput: The model's response
* documents: Optional list of documents (strings or LogDocument objects)
* message_history: Optional conversation history
* messageHistory: Optional conversation history
* instructions: Optional list of instructions
* tags: Optional tags to attach to the log
* hallucination_detection: Override hallucination detection setting
* inconsistency_detection: Override inconsistency detection setting
*
* Returns:
* Log ID if successful, None otherwise
* Log ID if successful, null otherwise
*/
async log(params: Omit<LogEntry, 'appName' | 'environment'>): Promise<any> {
return this.logger._internalLog(params);
Expand Down Expand Up @@ -92,3 +101,4 @@ export class QuotientAI {

// Export types that users need
export { TracingConfig } from './tracing';
export { DetectionType } from './types';
Loading