erc3-js

JavaScript ESM client for the ERC3 benchmark evaluation platform.

Overview

erc3-js is a pure ESM (ECMAScript Module) JavaScript client for interacting with the ERC3 benchmark platform. It provides a clean, modern API for managing evaluation sessions, running tasks, and interacting with benchmark-specific APIs.

Features

• Pure ESM: Modern JavaScript modules with full tree-shaking support
• Multiple Benchmarks: Support for Store, Demo, and other benchmarks
• Type-Safe: Comprehensive JSDoc annotations for editor autocomplete
• Separate APIs: Store and Demo APIs are independently accessible
• Error Handling: Custom ApiException class with detailed error information
• Session Management: Complete workflow for sessions and tasks

Installation

npm install erc3-js

Quick Start

import { ERC3 } from 'erc3-js';

// Create client (uses ERC3_API_KEY env var)
const client = new ERC3();

// Or provide API key explicitly
const client = new ERC3({ apiKey: 'your-api-key' });

// List available benchmarks
const benchmarks = await client.listBenchmarks();
console.log(benchmarks);

// Start a session
const session = await client.startSession({
  benchmark: 'store',
  workspace: 'my-workspace',
  name: 'Test Run 1'
});

// Get session status
const status = await client.sessionStatus(session.session_id);
console.log('Tasks:', status.tasks);

// Work with a task
const task = status.tasks[0];
await client.startTask(task.task_id);

// Get specialized client for the benchmark
const storeClient = client.getStoreClient(task.task_id);
const products = await storeClient.listProducts();
console.log(products);

// Complete the task
const result = await client.completeTask(task.task_id);
console.log('Evaluation:', result.eval);

Command-Line Interface (CLI)

The package includes three separate CLI utilities for interacting with the ERC3 platform:

1. erc3 - Core API (benchmarks, sessions, tasks)
2. erc3-store - Store API (products, basket, checkout)
3. erc3-demo - Demo API (secrets, answers)

This separation allows you to use each CLI independently, which is especially useful for code agents that only need access to specific benchmark APIs.

Installation

# Install globally
npm install -g erc3-js

# After installation, three commands are available:
erc3 --help
erc3-store --help
erc3-demo --help

Running from Git Clone

If you've cloned this repository instead of installing via npm:

# Core API
ERC3_API_KEY='your-api-key' node bin/erc3.js benchmarks

# Store API
ERC3_API_KEY='your-api-key' node bin/erc3-store.js products task-123 --limit 3

# Demo API
ERC3_API_KEY='your-api-key' node bin/erc3-demo.js secret task-456

# Or use npm link to install globally from the repo:
npm link
erc3 benchmarks
erc3-store products task-123 --limit 3
erc3-demo secret task-456

Environment Setup

Set your API key as an environment variable:

export ERC3_API_KEY='your-api-key'

CLI 1: Core API (erc3)

Handles benchmarks, sessions, and tasks.

# List benchmarks
erc3 benchmarks

# View benchmark details
erc3 benchmark store

# Start a session
erc3 session:start store --workspace my-workspace --name "Test Run"

# Get session status
erc3 session:status ssn-123

# Search sessions
erc3 session:search --workspace my-workspace

# Start a task
erc3 task:start tsk-456

# View task details
erc3 task:view tsk-456

# Complete a task
erc3 task:complete tsk-456

# Full help
erc3 --help

CLI 2: Store API (erc3-store)

Standalone CLI for the Store benchmark API. Can be used completely independently.

# List products (note: max limit is 3)
erc3-store products task-456 --limit 3

# View basket (formatted output)
erc3-store basket task-456

# Add product to basket
erc3-store add task-456 --sku gpu-h100 --quantity 1

# Remove product from basket
erc3-store remove task-456 --sku gpu-h100 --quantity 1

# Apply coupon
erc3-store coupon:apply task-456 --coupon SAVE20

# Remove coupon
erc3-store coupon:remove task-456

# Checkout
erc3-store checkout task-456

# Get JSON output for programmatic use
erc3-store products task-456 --json

# Full help
erc3-store --help

CLI 3: Demo API (erc3-demo)

Standalone CLI for the Demo benchmark API. Can be used completely independently.

# Get secret
erc3-demo secret task-789

# Submit answer
erc3-demo answer task-789 --answer "secret-value"

# Complete workflow with shell scripting
SECRET=$(erc3-demo secret task-789 --json | jq -r '.value')
erc3-demo answer task-789 --answer "$SECRET"

# Full help
erc3-demo --help

Why Separate CLIs?

The separation allows:

• Code agents to receive only the CLI they need (e.g., just erc3-store)
• Independent usage without core API dependencies
• Cleaner permissions when sharing tools with automated systems
• Easier testing of individual benchmark APIs

API Surface

Core Client (ERC3)

Constructor

new ERC3({ apiKey?, baseUrl? })

• apiKey (string, optional): API key (defaults to ERC3_API_KEY env var)
• baseUrl (string, optional): Base URL (defaults to https://erc.timetoact-group.at)

Benchmark Methods

listBenchmarks()

Lists all available benchmarks.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{}' \
  https://erc.timetoact-group.at/benchmarks/list

JavaScript:

const benchmarks = await client.listBenchmarks();
// Returns: { benchmarks: [...] }

viewBenchmark(benchmark)

Views detailed information about a specific benchmark.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"benchmark": "store"}' \
  https://erc.timetoact-group.at/benchmarks/view

JavaScript:

const benchmark = await client.viewBenchmark('store');
// Returns: { id: 'store', description: '...', specs: [...], routes: [...] }

Session Methods

startSession({ benchmark, workspace, name, architecture? })

Starts a new evaluation session.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "account_key": "YOUR_API_KEY",
    "benchmark": "store",
    "workspace": "my-workspace",
    "name": "Test Run 1",
    "architecture": "x86_64"
  }' \
  https://erc.timetoact-group.at/sessions/start

JavaScript:

const session = await client.startSession({
  benchmark: 'store',
  workspace: 'my-workspace',
  name: 'Test Run 1',
  architecture: 'x86_64' // optional, defaults to 'x86_64'
});
// Returns: { session_id: '...', task_count: 15 }

sessionStatus(sessionId)

Gets the current status of a session.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"session_id": "SESSION_ID"}' \
  https://erc.timetoact-group.at/sessions/status

JavaScript:

const status = await client.sessionStatus('session-123');
// Returns: { session_id: '...', tasks: [...], ... }

searchSessions(criteria)

Searches for sessions based on criteria.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "account_key": "YOUR_API_KEY",
    "workspace": "my-workspace",
    "benchmark": "store"
  }' \
  https://erc.timetoact-group.at/sessions/search

JavaScript:

const sessions = await client.searchSessions({
  workspace: 'my-workspace',
  benchmark: 'store'
});

submitSession(sessionId)

Submits a completed session for evaluation.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"session_id": "SESSION_ID"}' \
  https://erc.timetoact-group.at/sessions/submit

JavaScript:

const result = await client.submitSession('session-123');

Task Methods

startTask(taskOrId)

Starts a task.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"task_id": "TASK_ID"}' \
  https://erc.timetoact-group.at/tasks/start

JavaScript:

await client.startTask('task-123');
// Or with task object:
await client.startTask(task);

completeTask(taskOrId)

Completes a task and retrieves evaluation results.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"task_id": "TASK_ID"}' \
  https://erc.timetoact-group.at/tasks/complete

JavaScript:

const result = await client.completeTask('task-123');
console.log('Evaluation:', result.eval);

viewTask(taskId, since?)

Views task details and logs.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"task_id": "TASK_ID"}' \
  https://erc.timetoact-group.at/tasks/view

JavaScript:

const task = await client.viewTask('task-123');
// Or get logs since timestamp:
const task = await client.viewTask('task-123', 1234567890);

logLLM({ taskId, model, usage, durationSec })

Logs LLM usage for a task.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "task_id": "TASK_ID",
    "model": "gpt-4",
    "usage": {
      "prompt_tokens": 100,
      "completion_tokens": 50,
      "total_tokens": 150
    },
    "duration_sec": 2.5
  }' \
  https://erc.timetoact-group.at/tasks/log

JavaScript:

await client.logLLM({
  taskId: 'task-123',
  model: 'gpt-4',
  usage: {
    prompt_tokens: 100,
    completion_tokens: 50,
    total_tokens: 150
  },
  durationSec: 2.5
});

Factory Methods

getStoreClient(taskOrId)

Creates a Store API client for a specific task.

const storeClient = client.getStoreClient('task-123');
// See Store API documentation below

getDemoClient(taskOrId)

Creates a Demo API client for a specific task.

const demoClient = client.getDemoClient('task-123');
// See Demo API documentation below

Helper Functions

getApiKey(email, baseUrl?)

Gets an API key from the ERC3 service.

cURL Example:

curl -X POST \
  -H 'Content-Type: application/json' \
  -d '{"email": "your@email.com"}' \
  https://erc.timetoact-group.at/get_key

JavaScript:

import { getApiKey } from 'erc3-js';

const result = await getApiKey('your@email.com');
console.log('API Key:', result.account_key);

Store API

The Store API provides e-commerce functionality for the "store" benchmark. See Store API Documentation for detailed information.

Quick Example:

const storeClient = client.getStoreClient('task-123');

// List products
const products = await storeClient.listProducts({ offset: 0, limit: 20 });

// Add to basket
await storeClient.addToBasket('gpu-h100', 1);

// View basket
const basket = await storeClient.viewBasket();
console.log('Total:', basket.total / 100); // Prices are in cents

// Apply coupon
await storeClient.applyCoupon('DOGGY25');

// Checkout
const result = await storeClient.checkout();

See Store API README for complete documentation and curl examples.

Demo API

The Demo API provides a simple secret-retrieval workflow for the "demo" benchmark. See Demo API Documentation for detailed information.

Quick Example:

const demoClient = client.getDemoClient('task-123');

// Get secret
const secret = await demoClient.getSecret();
console.log('Secret:', secret.value);

// Submit answer
await demoClient.submitAnswer(secret.value);

See Demo API README for complete documentation and curl examples.

Error Handling

The SDK uses a custom ApiException class for error handling:

import { ERC3, ApiException } from 'erc3-js';

try {
  const result = await client.completeTask('task-123');
} catch (error) {
  if (error instanceof ApiException) {
    console.error('API Error:', error.message);
    console.error('Status:', error.status);
    console.error('Code:', error.code);
    console.error('Detail:', error.detail);
  } else {
    console.error('Unexpected error:', error);
  }
}

Environment Variables

• ERC3_API_KEY: Default API key (optional if provided in constructor)

Module Exports

The package provides multiple export paths:

// Main exports
import { ERC3, StoreClient, DemoClient, ApiException, getApiKey } from 'erc3-js';

// Or import from specific paths
import { StoreClient } from 'erc3-js/store';
import { DemoClient } from 'erc3-js/demo';
import { ApiException } from 'erc3-js/common';

Requirements

• Node.js >= 18.0.0
• ESM-compatible environment

Examples

See the examples/ directory for complete working examples:

• basic-usage.js - Complete session workflow
• store-example.js - Store API usage
• demo-example.js - Demo API usage

License

MIT

Related Documentation

• Store API README - Complete Store API documentation
• Demo API README - Complete Demo API documentation