Skip to content

Data mapping optimization

Jump to bottom
Jack Duan edited this page Sep 13, 2025 · 2 revisions

Whereis API: Data Mapping Optimization

Executive Summary

The Eagle1 whereis-api-v0 implements an advanced two-stage data transformation system that converts carrier-specific tracking data into standardized, developer-friendly responses. This system goes beyond simple field mapping by incorporating intelligent optimization to enhance data completeness and accuracy.

Architecture Overview

Core Philosophy

  • Beyond Simple Mapping: Transform raw carrier data into meaningful, standardized information
  • Developer-Centric: Provide consistent, easy-to-understand API responses
  • Continuous Improvement: Mapping rules evolve based on real-world data patterns

Two-Stage Optimization Process

Stage 1: Rule-Based Status Code Mapping

Converts carrier-specific events to standardized Eagle1 status codes using sophisticated mapping logic that considers multiple contextual factors.

Stage 2: AI-Powered Data Enhancement

  • Missing Event Generation: Automatically fills gaps in tracking timelines (e.g., missing 3100 events for FedEx, 3400 events for SF Express)
  • Location Optimization: Enhances geographical and facility information, particularly for SF Express shipments
  • Data Quality Improvement: Ensures comprehensive shipment visibility

Implementation Details

FedEx Integration (operators/fdx.ts)

Data Source: FedEx API scanEvents array
Processing Order: Reverse chronological (latest events first, then reversed)

Key Decision Factors:

  1. derivedStatusCode - High-level status category (IN, IT, CD, PU, DL, DE, CA)
  2. eventType - Specific event classification (OC, DR, DP, AR, IT, etc.)
  3. locationType - Facility type context (ORIGIN_FEDEX_FACILITY, SORT_FACILITY, etc.)
  4. eventDescription - Textual event details for pattern matching
  5. exceptionCode - Special handling for exceptions (e.g., code "67")

Status Code Translation Logic (from actual implementation):

// Example of complex mapping logic
IT: {
  DP: function(entity, sourceData): number {
    const locationType = sourceData.locationType as string;
    const eventDescription = sourceData.eventDescription as string;
    
    if (locationType === "ORIGIN_FEDEX_FACILITY") {
      return 3100; // Received by Carrier
    }
    if (/Departed FedEx hub/i.test(eventDescription)) {
      return 3250; // In-Transit
    }
    return 3004; // Departed, In-Transit
  }
}

AI-Powered Missing Event Generation:

  • Missing 3100 Detection: isMissing3100() checks if "Received by Carrier" event is missing
  • Base Event Identification: get3100BaseEvent() finds the appropriate base event (status 3001)
  • Supplement Event Creation: createSupplementEvent() generates missing events with system attribution

SF Express Integration (operators/sfex.ts)

Data Source: SF Express API routes array
Processing Order: Chronological event sequence

Key Decision Factors:

  1. secondaryStatusCode - Primary status identifier (101, 201, 204, 205, 301, etc.)
  2. opCode - Operation code (30, 31, 36, 50, 54, etc.)
  3. remark - Chinese language descriptions with pattern matching
  4. secondaryStatusName - Status descriptions for customs processing

Complex Mapping Example (from actual implementation):

"201": function(entity, sourceData): number {
  const map = {
    "30": 3001, // Logistics In-Progress
    "31": 3002, // Arrived, In-Transit
    "36": 3004, // Departed, In-Transit
    "105": 3250, // In-Transit
    "106": 3300, // Arrived At Destination
  };
  
  const remark = sourceData["remark"] as string;
  if (/完成分拣/.test(remark)) {
    return 3003; // Scanned, In-Transit
  } else if (/快件离开/.test(remark)) {
    return 3004; // Departed, In-Transit
  }
  
  return map[sourceData["opCode"] as string] ?? 3001;
}

AI-Powered Customs Enhancement:

  • Missing 3400 Detection: isMissing3400() identifies missing "Import Released" events
  • Smart Base Event Selection: get3400BaseEvent() finds events after customs processing (3350)
  • Intelligent Timing: Supplements events 1 second before the base event

Location Optimization (SF Express Focus)

Location Processing Logic:

// Location enhancement example
const remark: string = (route["remark"] as string).trim();
if (remark.startsWith("快件途经")) {
  event.where = remark.substring(4); // Extract location after "package via"
} else {
  event.where = route["acceptAddress"] as string;
}

Optimization Benefits:

  • Standardizes Chinese location descriptions
  • Extracts meaningful location information from remarks
  • Provides consistent geographical context

Complete Status Code Mapping Tables

FedEx Status Code Mappings

derivedStatusCode eventType Condition/Logic Eagle1 Code Description
IN OC Any 3000 Transport Bill Created
IT DR Any 3250 In-Transit
IT DP Origin FedEx Facility 3100 Received by Carrier
IT DP "Departed FedEx hub" 3250 In-Transit
IT DP Default 3004 Departed, In-Transit
IT AR Sort facility + "destination" 3300 At destination sort facility
IT AR Origin FedEx Facility 3100 Received by Carrier
IT AR Destination FedEx Facility 3300 At local FedEx facility
IT AR Default 3002 Arrived, In-Transit
IT IT Exception code "67" 3450 Final Delivery In-Progress
IT IT Default 3001 Logistics In-Progress
IT AF Any 3001 Logistics In-Progress
IT CC "Export" in description 3200 Customs Clearance: Export Released
IT CC "Import" in description 3400 Customs Clearance: Import Released
IT OD Any 3450 Final Delivery In-Progress
IT RR Any 3450 Delivery option requested
CD CD "Import" in description 3350 Customs Clearance: Import In-Progress
CD CD Default 3150 Customs Clearance: Export In-Progress
PU PU Any 3050 Picked up
DL DL Any 3500 Delivered
DE DE Any 3450 Final Delivery In-Progress
CA CA Any 3009 Process Stopped

SF Express Status Code Mappings

secondaryStatusCode opCode Condition Eagle1 Code Description
101 50, 54 Any 3100 Received by Carrier
201 30 Any 3001 Logistics In-Progress
201 31 Any 3002 Arrived, In-Transit
201 36 Any 3004 Departed, In-Transit
201 105 Any 3250 In-Transit
201 106 Any 3300 Arrived At Destination
201 * "完成分拣" (sorting complete) 3003 Scanned, In-Transit
201 * "快件离开" (package departed) 3004 Departed, In-Transit
204 * "清关中" (clearing customs) 3350 Customs Clearance: Import In-Progress
204 * Default 3001 Logistics In-Progress
205 * "已清关" (customs cleared) 3400 Customs Clearance: Import Released
205 * Default 3001 Logistics In-Progress
301 * "派送中" (out for delivery) 3450 Final Delivery In-Progress
301 * Default 3001 Logistics In-Progress
1301 70 Any 3300 Arrived At Destination
401 80 Any 3500 Delivered

AI Optimization Features

1. Missing Event Generation

FedEx - Missing 3100 Events:

  • Detection Logic: Checks if events progress beyond 3100 without including a 3100 event
  • Base Event Selection: Identifies first 3001-04 event after pickup-related events
  • Timing: Creates supplement event 1 millisecond before base event

SF Express - Missing 3400 Events:

  • Detection Logic: Identifies customs scenarios (3350) followed by progression events without 3400
  • Smart Targeting: Focuses on events with status codes [3004, 3250, 3450, 3500]
  • Intelligent Placement: Inserts supplement event at logical timeline position

Reference: Issue 171

2. Location Enhancement

SF Express Specific Optimizations:

  • Chinese Text Processing: Extracts meaningful location from Chinese remarks
  • Pattern Recognition: Identifies "快件途经" (package via) patterns
  • Fallback Logic: Uses acceptAddress when remarks don't contain location patterns

Reference: SF Express Location Optimization Issue #32

Benefits and Impact

For Developers

  • Unified API Response: Consistent status codes across all carriers
  • Complete Timeline Coverage: No missing critical events
  • Enhanced Location Data: Clear, standardized facility information
  • Predictable Data Structure: Reliable response format

For End Users

  • Comprehensive Tracking: Complete shipment journey visibility
  • Clear Status Updates: Meaningful event descriptions
  • Accurate Location Information: Enhanced geographical context

Technical Implementation Highlights

Error Handling & Resilience

  • Graceful Degradation: Returns fallback status 3001 for unmapped events
  • Exception Processing: Specialized handling for FedEx exception codes
  • API Error Management: Comprehensive error handling for carrier API failures

Performance Optimizations

  • Efficient Mapping Algorithms: O(1) lookup for direct mappings
  • Minimal Processing Overhead: Optimized supplement event generation
  • Token Management: Efficient FedEx API authentication with automatic renewal

Data Integrity

  • Event ID Uniqueness: Timestamp-based event identification
  • Chronological Sorting: Automatic event ordering by timestamp
  • Duplicate Prevention: Event ID existence checking before addition

Future Roadmap

Planned Enhancements

  • Issue #171: Advanced missing event generation
  • Issue #179: AI LLM integration for enhanced mapping intelligence
  • Issue #180: Multi-version data service architecture

Continuous Improvement Process

  1. Data Pattern Analysis: Regular review of real-world tracking data
  2. Mapping Rule Refinement: Iterative improvement based on edge cases
  3. AI Enhancement Evolution: Progressive sophistication of optimization algorithms

This documentation demonstrates how Eagle1 transforms raw carrier tracking data into intelligent, developer-friendly responses through sophisticated rule-based mapping combined with AI-powered optimization, delivering complete shipment visibility and enhanced user experience.

Source reference

(C) 2025 Eagle1

Clone this wiki locally