Data mapping from logistics sources

Data Mapping From Logistics Sources

This document explains how the Eagle1 Whereis API transforms raw carrier data into standardized, developer-friendly API responses. The mapping system is the core component that enables consistent tracking information across different logistics providers.

Overview

The mapping system consists of three main components:

1. Status Code Mapping - Converts carrier-specific status codes to standardized Eagle1 status codes
2. Data Transformation - Normalizes carrier data formats into consistent API responses
3. Event Generation - Creates structured event objects with standardized fields

Supported Carriers

Currently, the system supports:

• FedEx (fdx) - Full tracking with detailed scan events
• SF Express (sfex) - Comprehensive tracking with customs clearance support

Architecture

graph TD
    A[Raw Carrier Data] --> B[Operator Class]
    B --> C[Status Code Mapping]
    B --> D[Data Transformation]
    B --> E[Event Creation]
    C --> F[Standardized Entity]
    D --> F
    E --> F
    F --> G[API Response]

Loading

Status Code Mapping

Each carrier has its own status code mapping system that converts proprietary codes to Eagle1 standard codes:

FedEx Mapping Structure

// Example: FedEx IT.AR event mapping
IT: {
  AR: function(entity: Entity, sourceData: Record<string, unknown>): number {
    const locationType = sourceData["locationType"] as string;
    if (locationType === "DESTINATION_FEDEX_FACILITY") {
      return 3300; // At local FedEx facility
    }
    return 3002; // Arrived, In-Transit (default)
  }
}

SF Express Mapping Structure

// Example: SF Express status 201 mapping
"201": function(entity: Entity, sourceData: Record<string, unknown>): number {
  const map: Record<string, number> = {
    "30": 3001, // Logistics In-Progress
    "31": 3002, // Arrived, In-Transit
    "36": 3004, // Departed, In-Transit
  };
  return map[sourceData["opCode"] as string] || 3001;
}

Mapping Process Flow

1. Data Retrieval

• API authentication (OAuth for FedEx, signed digest for SF Express)
• Fetch raw tracking data from carrier APIs
• Handle API errors and rate limiting

2. Data Processing

For each carrier event:

1. Extract key fields: timestamp, location, status codes
2. Apply status mapping: Convert to Eagle1 standard codes
3. Normalize format: Consistent field names and data types
4. Generate event ID: Unique identifier based on tracking ID, timestamp, and status

3. Entity and Events Construction

{
  "entity": {
    "id": "fdx-888877776666",
    "type": "waybill",
    "uuid": "eg1_7e3f6f06-2710-4225-8067-62bebfc4x45c",
    "createdAt": "2024-11-11T14:16:48-06:00",
    "additional": {
      "origin": "San Francisco CA United States",
      "destination": "CENTRAL  Hong Kong SAR, China"
    }
  },
  "events": [{
    "status": 3000,
    "what": "Transport Bill Created",
    "whom": "FedEx",
    "when": "2024-11-11T14:16:48-06:00",
    "where": "Customer location",
    "notes": "Shipment information sent to FedEx",
    "additional": {
      "trackingNum": "888877776666",
      "operatorCode": "fdx",
      "dataProvider": "FedEx",
      "updateMethod": "manual-pull",
      "updatedAt": "2025-02-20T12:23:43.892Z"
    }
  }]
}

Standard Status Codes

Eagle1 uses a standard status code system:

Range	Category	Examples
3000-3009	Transport Start	3000: Transport Bill Created
3050-3099	Pickup	3050: Picked up
3100-3199	Received	3100: Received by Carrier
3200-3299	Export Processing	3200: Export Released
3300-3399	Arrival	3300: Arrived At Destination
3400-3449	Import Processing	3400: Import Released
3450-3499	Delivery	3450: Final Delivery In-Progress
3500+	Completion	3500: Delivered

Smart Event Generation

The system includes intelligent features:

Missing Event Detection

• FedEx: Automatically detects missing "Received by Carrier" (3100) events
• SF Express: Identifies missing "Import Released" (3400) events in customs scenarios

Supplement Event Creation

When critical events are missing, the system generates supplementary events:

// Example: Generate missing 3100 event 1 second before the next logical event
const supplementEvent = this.createSupplementEvent(
  trackingId,
  3100, // Status code
  baseEvent.when, // Timestamp reference  
  baseEvent.where // Location
);

Context-Aware Mapping

Mapping decisions consider:

• Location type (origin facility vs. destination facility)
• Event descriptions (keywords like "Export", "Import", "Departed")
• Sequence logic (what events should logically follow others)

Error Handling

The mapping system handles various error scenarios:

Authentication Errors

• Invalid API credentials → 500-01 error with specific operator code
• Token expiration → Automatic refresh with 5-second buffer

Data Quality Issues

• Missing route data → Logged warnings with context
• Malformed responses → Graceful degradation
• Empty event arrays → Proper logging and empty entity return

Configuration

Key configuration default parameters:

// FedEx Configuration
fdx: {
  apiUrl: "https://apis.fedex.com/oauth/token",
  trackApiUrl: "https://apis.fedex.com/track/v1/trackingnumbers"
}

// SF Express Configuration  
sfex: {
  apiUrl: "https://bspgw.sf-express.com/std/service",
  dataSourceTimezone: +8 // China timezone
}

Performance Optimizations

Batch Processing

• FedX: Support for multiple tracking numbers in single API call
• SF Express: Single tracking number per request (API limitation)

Caching Strategy

• Token caching with automatic refresh
• Event deduplication using unique event IDs

Rate Limiting

• Respects carrier API rate limits
• Graceful backoff on API errors

Extending the System

To add a new carrier:

1. Create operator class in /operators/[carrier].ts
2. Implement required methods: whereIs(), getStatusCode(), convert()
3. Define status mapping dictionary for carrier-specific codes
4. Add configuration in config.ts
5. Register operator in the gateway system

Testing and Validation

The mapping system includes comprehensive testing:

• Unit tests for each carrier mapping
• Integration tests with live API responses
• Status code validation against standard codes
• Event sequence validation

---

For technical implementation details, see the source code in:

• /operators/fdx.ts - FedEx implementation
• /operators/sfex.ts - SF Express implementation
• /metadata/status-codes.jsonc - Standard status code definitions
• /main/model.ts - Core data models