Skip to content

purvik6062/smart-cache-script

BranchesTags

Repository files navigation

SmartCache Enhanced Backend

A sophisticated Node.js backend service for automated Arbitrum Stylus contract caching with ROI-based bidding logic, real-time blockchain monitoring, and intelligent contract discovery.

🎯 Overview

SmartCache Enhanced Backend automatically discovers new Stylus contracts on Arbitrum, detects SmartCache opt-ins, and places profitable bids for contract caching based on real-time ROI analysis. The system features comprehensive logging, error handling, and performance monitoring.

✨ Key Features

🔍 Real-time Contract Discovery

  • WebSocket Event Monitoring: Live monitoring of ProgramActivated events for new Stylus deployments
  • Automatic Opt-in Detection: Real-time detection via isCacheable() function calls and AutoCacheOptIn events
  • Contract Classification: Automatic categorization (discovered, checked, opted-in, cached)
  • Automatic Caching: When isCacheable() returns true, automatically attempts to place profitable bids
  • Manual Contract Checking: API endpoint for manual opt-in verification and caching

💰 ROI-Based Bidding Logic

  • Dynamic market analysis using latest bid value data
  • Configurable ROI thresholds (default: 10%)
  • Intelligent bid placement with gas estimation and balance validation
  • Comprehensive bid success/failure tracking

📊 Structured Logging & Monitoring

  • Winston-based structured logging with multiple transport layers
  • Request/response logging with unique request IDs
  • Blockchain-specific event logging (contract deployment, bid placement, opt-ins)
  • Database operation tracking and error monitoring

🌐 WebSocket Infrastructure

  • Real-time Event Streaming: Live monitoring of blockchain events via WebSocket providers
  • Environment Configuration: Uses WS_ARBITRUM_SEPOLIA_URL environment variable for flexible network setup
  • Provider Management: Centralized WebSocket provider handling with automatic reconnection
  • Event Topic Monitoring: Dedicated listeners for ProgramActivated and AutoCacheOptIn events
  • Graceful Shutdown: Proper WebSocket cleanup on application termination

🔄 Automatic Caching Flow

  1. Contract Deployment: WebSocket detects ProgramActivated event
  2. Opt-in Check: Calls isCacheable() on the new contract
  3. ROI Analysis: If cacheable, analyzes profitability using current market data
  4. Bid Placement: If profitable (ROI > threshold), automatically places bid
  5. Result Tracking: Logs success/failure and updates contract categories
  6. Continuous Monitoring: Process repeats for all new contract deployments

🛡️ Robust Error Handling

  • Custom error classes for different error types (Blockchain, Database, Validation)
  • Comprehensive error middleware with consistent API responses
  • Global error handlers for uncaught exceptions and unhandled rejections
  • Development-friendly error responses with stack traces

🗄️ Database Integration

  • MongoDB integration for contract and bid data storage
  • Automated contract metadata storage with eviction thresholds
  • Performance-optimized caching with background updates
  • Database health monitoring and connection management

🔗 Multi-Network Support

  • Arbitrum Sepolia (primary)
  • Arbitrum One and Nova support
  • Alchemy integration for deployer verification
  • Network-specific configuration management

🌐 Network Selection (Testnet/Mainnet)

The backend now supports fully dynamic network selection for both Arbitrum Sepolia (testnet) and Arbitrum One (mainnet).

Single Network Mode:

  • Default Network: The default network is set via the NETWORK environment variable. Supported values: arbitrum-sepolia, arbitrum-one.
  • Override Per Request: Most API endpoints (such as /place-bid) accept a network parameter in the request body to override the default for that operation.

Multi-Network WebSocket Monitoring:

  • Multiple Networks: You can monitor multiple networks simultaneously for WebSocket events using the NETWORKS environment variable.
  • Configuration: Set NETWORKS=arbitrum-sepolia,arbitrum-one to monitor both testnet and mainnet.
  • Event Processing: Events from both networks are processed independently and tracked separately.

Multi-Network Bid Value Caching:

  • Separate Documents: The bid_values collection now stores separate documents for each network:
    • { type: "latest_bid_value", network: "arbitrum-sepolia", ... }
    • { type: "latest_bid_value", network: "arbitrum-one", ... }
  • Background Updates: The background updater monitors both networks simultaneously and updates their respective cache documents.
  • Network-Specific Contracts: Each network uses its own CacheManager contract:
    • Arbitrum Sepolia: 0x0C9043D042aB52cFa8d0207459260040Cca54253
    • Arbitrum One: 0x51dEDBD2f190E0696AFbEE5E60bFdE96d86464ec - Block Range Optimization: - Arbitrum Sepolia: Uses 5000-block ranges for efficient event querying - Arbitrum One: Uses 400-block ranges with pagination (up to 20 chunks = 8000 blocks) to respect Alchemy's 500-block limit for eth_getLogs and ensure older bids are found

Environment Variables:

# Single Network Mode
NETWORK=arbitrum-sepolia

# Multi-Network Mode (for WebSocket monitoring and bid value caching)
NETWORKS=arbitrum-sepolia,arbitrum-one

# Required API Keys
ARBISCAN_KEY=your_arbiscan_api_key
ALCHEMY_API_KEY=your_alchemy_api_key

# Database
MONGODB_URI=your_mongodb_connection_string
DATABASE_NAME=smartcache

# WebSocket URLs (optional - defaults provided)
WS_ARBITRUM_SEPOLIA_URL=wss://sepolia-rollup.arbitrum.io/ws

Examples:

Single Network (Testnet):

NETWORK=arbitrum-sepolia

Single Network (Mainnet):

NETWORK=arbitrum-one

Multi-Network (Both):

NETWORKS=arbitrum-sepolia,arbitrum-one

Multi-Network (Mainnet Only):

NETWORKS=arbitrum-one

Supported Networks:

  • arbitrum-sepolia (testnet)
  • arbitrum-one (mainnet)

Example Configurations:

Single Network (WebSocket + API):

# Monitor only Arbitrum Sepolia
NETWORK=arbitrum-sepolia

# Monitor only Arbitrum One
NETWORK=arbitrum-one

Multi-Network WebSocket Monitoring:

# Monitor both testnet and mainnet simultaneously
NETWORKS=arbitrum-sepolia,arbitrum-one

# Monitor only mainnet
NETWORKS=arbitrum-one

# Monitor only testnet
NETWORKS=arbitrum-sepolia

API Request Examples:

{
  "contractAddress": "0x...",
  "network": "arbitrum-one" // or omit for default
}

Multi-Network Features:

  • Parallel Monitoring: All specified networks are monitored simultaneously
  • Network-Specific Events: Each network's events are tracked separately
  • Independent Contract Processing: Contracts discovered on each network are processed independently
  • Fault Tolerance: If one network fails to connect, others continue monitoring
  • Network-Aware Logging: All logs include the source network information

🏗️ Architecture

Modular Structure

src/
├── server.js              # Application entry point
├── app.js                 # Main application logic & event listeners
├── services/              # Business logic layer
│   ├── bidding-logic.js   # ROI-based bidding service
│   ├── bid-value.js       # Blockchain data fetching & caching
│   └── place-bid.js       # Bid placement functionality
├── routes/                # API endpoints
│   ├── contracts.js       # Contract management APIs
│   ├── performance.js     # Performance monitoring APIs
│   └── status.js          # System status & analytics
├── database/              # Database layer
│   └── connection.js      # MongoDB connection management
├── blockchain/            # Blockchain integration
│   └── providers.js       # WebSocket providers & utilities
├── middleware/            # Express middleware
│   ├── errorHandler.js    # Global error handling
│   └── requestLogger.js   # Request/response logging
└── utils/
    └── logger.js          # Winston logging configuration

🚀 Quick Start

Prerequisites

  • Node.js 18+
  • MongoDB database
  • Arbitrum Sepolia RPC access
  • Arbiscan API key
  • Private key for bid transactions (optional for read-only mode)

Installation

# Clone repository
git clone <repository-url>
cd smart-cache-script

# Install dependencies
npm install

# Set up environment variables
cp .env.example .env
# Edit .env with your configuration

Environment Configuration

Create a .env file with the following variables:

# Blockchain Configuration
WS_ARBITRUM_SEPOLIA_URL=wss://sepolia-rollup.arbitrum.io/ws
ARBISCAN_KEY=your_arbiscan_api_key
PRIVATE_KEY=your_private_key_for_bidding
ALCHEMY_API_KEY=your_alchemy_api_key

# Database Configuration
MONGODB_URI=mongodb://localhost:27017/smartcache
DATABASE_NAME=smartcache
COLLECTION_NAME=contracts

# Application Configuration
NODE_ENV=development
LOG_LEVEL=info
PORT=4000
NETWORK=arbitrum-sepolia # or arbitrum-one

Running the Application

# Development mode (JavaScript)
npm run dev

# Development mode (TypeScript)
npm run dev:ts

# Production build
npm run build
npm run start:prod

# Standard start
npm start

# Testing WebSocket functionality
npm run test:websocket

# Testing bid cache system
npm run test:bid-cache

🧪 Testing

WebSocket Monitoring Test

Test the real-time WebSocket monitoring functionality:

# Start the server (in a separate terminal)
npm run dev:ts

# Run WebSocket tests (in another terminal)
npm run test:websocket

This test will verify:

  • WebSocket connection status
  • Event listener activation
  • Contract tracking functionality
  • Real-time status endpoints
  • Manual contract checking
  • Automatic contract caching (checkAndCache functionality)

📚 API Documentation

Contract Management

Store Contract

POST /contracts/store
Content-Type: application/json

{
  "contractAddress": "0x...",
  "deployedBy": "0x...",
  "network": "arbitrum-sepolia"
}

List Contracts

GET /contracts/list?network=arbitrum-sepolia

Check Cache Status

GET /contracts/cache-status/0x...

Place Bid

POST /contracts/place-bid
Content-Type: application/json

{
  "contractAddress": "0x...",
  "network": "arbitrum-one" // or omit for default
}

Real-time Monitoring

Get Tracked Contracts

GET /api/tracked-contracts

Returns all contract categories with real-time status:

{
  "allContracts": ["0x123...", "0x456..."],
  "optedInContracts": ["0x123..."],
  "checkedContracts": ["0x123...", "0x456..."],
  "successfullyCachedContracts": ["0x123..."],
  "counts": {
    "total": 2,
    "optedIn": 1,
    "checked": 2,
    "cached": 1
  },
  "realtimeStatus": {
    "websocketConnected": true,
    "eventListenersActive": true,
    "providerConnected": true
  }
}

Real-time Status

GET /api/realtime-status

Detailed WebSocket and event monitoring status:

{
  "websocket": {
    "connected": true,
    "provider": "initialized"
  },
  "eventListeners": {
    "active": true,
    "programActivatedTopic": "0x...",
    "autoCacheOptInTopic": "0x..."
  },
  "contracts": {
    "arbWasm": "connected",
    "cacheManager": "connected",
    "smartCacheDetection": "connected"
  },
  "discovery": {
    "totalDiscovered": 5,
    "totalChecked": 4,
    "totalOptedIn": 2,
    "checkRate": 50
  }
}

Manual Contract Opt-in Check

POST /api/check-contract-optin
Content-Type: application/json

{
  "contractAddress": "0x..."
}

Manually trigger opt-in verification for a specific contract:

{
  "success": true,
  "contractAddress": "0x...",
  "message": "Contract opt-in check completed",
  "isOptedIn": true,
  "wasChecked": true
}

Manual Contract Caching

POST /api/cache-contract
Content-Type: application/json

{
  "contractAddress": "0x..."
}

Manually trigger caching (bid placement) for a specific contract:

{
  "success": true,
  "contractAddress": "0x...",
  "message": "Contract caching attempt completed",
  "wasCached": true,
  "isOptedIn": true,
  "totalCached": 3
}

Performance Monitoring

Bid Cache Performance

GET /performance/bid-cache

Database Health

GET /performance/database-health

Initialize Cache

POST /performance/initialize-bid-cache

System Status

System Overview

GET /status

Manual Contract Analysis

POST /status/analyze/0x...

🔧 Configuration

Bidding Logic Configuration

const biddingLogic = new BiddingLogic({
  roiThreshold: 0.1, // 10% ROI threshold
  defaultMarketBid: "0.005", // Default market bid in ETH
  logger: logger, // Winston logger instance
});

Logging Configuration

The application uses Winston for structured logging with multiple levels:

  • error: Error conditions
  • warn: Warning conditions
  • info: Informational messages
  • debug: Debug-level messages

Logs are stored in:

  • logs/error.log - Error messages only
  • logs/combined.log - All log levels
  • logs/exceptions.log - Uncaught exceptions
  • logs/rejections.log - Unhandled promise rejections

Database Schema

Contracts Collection

{
  contractAddress: String,
  deployedBy: String,
  network: String,
  minBidRequired: String,
  gasSaved: String,
  deployedAt: Date,
  evictionThresholdDate: Date,
  usingAutoCacheFlag: Boolean,
  txHash: String,
  roiAnalysis: Object
}

🧪 Testing

Bid Cache Testing

# Run bid cache tests
node tests/bid-cache.test.js

Manual Contract Testing

# Test specific contract
curl -X POST http://localhost:4000/status/analyze/0x...

🔍 Monitoring & Debugging

Log Analysis

# View real-time logs
tail -f logs/combined.log

# Filter by module
grep "Blockchain" logs/combined.log

# View errors only
tail -f logs/error.log

Health Checks

  • Database: GET /performance/database-health
  • WebSocket: Monitor connection status in logs
  • Bidding: GET /performance/bid-cache

🛠️ Development

Adding New Routes

  1. Create route file in src/routes/
  2. Import and register in src/app.js
  3. Use error handling middleware with asyncHandler

Adding New Services

  1. Create service file in src/services/
  2. Use structured logging with createLogger
  3. Implement proper error handling

Custom Error Types

const { BlockchainError } = require("./middleware/errorHandler");
throw new BlockchainError("Transaction failed", "bid_placement");

📈 Performance Optimization

Caching Strategy

  • MongoDB-based bid value caching (5-minute TTL)
  • Background cache updates every 5 minutes
  • Reverse chronological event processing for latest data

Connection Management

  • WebSocket reconnection with exponential backoff
  • MongoDB connection pooling
  • Request ID tracking for debugging

🔒 Security Considerations

  • Private keys stored in environment variables
  • Database connection strings sanitized in logs
  • Request/response size limiting
  • Input validation on all endpoints

📞 Support & Contributing

Common Issues

  1. WebSocket Connection Fails: Check WS_ARBITRUM_SEPOLIA_URL and network connectivity
  2. Database Connection Issues: Verify MONGODB_URI and database accessibility
  3. Bid Placement Fails: Check private key and account balance

Contributing

  1. Fork the repository
  2. Create feature branch
  3. Add tests for new functionality
  4. Submit pull request with detailed description

📝 License

[Your License Here]

Built with ❤️ for the Arbitrum Stylus ecosystem

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages