Skip to content

Data mapping optimization

Jump to bottom
Jack Duan edited this page Jul 2, 2026 · 2 revisions

Whereis API: Data Mapping Optimization

Executive Summary

The Eagle1 whereis-api-v0 uses an advanced two-stage process to convert carrier tracking data into standardized, developer-friendly responses. Instead of relying on simple field mapping, this system uses intelligent optimization to ensure data is complete and accurate.

Architecture Overview

Core Philosophy

  • Beyond Simple Mapping: Transform raw carrier data into clear, standardized information.
  • Developer-Centric: Provide consistent API responses that are easy to understand.
  • Continuous Improvement: Update mapping rules based on real-world data patterns.

Two-Stage Optimization Process

Stage 1: Rule-Based Status Code Mapping

Converts carrier-specific events into standardized Eagle1 status codes using mapping logic based on multiple contextual factors.

Stage 2: AI-Powered Data Enhancement

  • Missing Event Generation: Automatically fills gaps in tracking timelines (e.g., missing 3100 events for FedEx, or 3400 events for SF Express).
  • Location Optimization: Improves geographical and facility information, especially for SF Express shipments.
  • Data Quality Improvement: Ensures comprehensive shipment visibility.

Implementation Details

FedEx Integration (operators/fdx.ts)

Data Source: FedEx API scanEvents array
Processing Order: Reverse chronological (latest events first, then reversed)

Key Decision Factors:

  1. derivedStatusCode - High-level status category (IN, IT, CD, PU, DL, DE, CA)
  2. eventType - Specific event classification (OC, DR, DP, AR, IT, etc.)
  3. locationType - Facility type (ORIGIN_FEDEX_FACILITY, SORT_FACILITY, etc.)
  4. eventDescription - Text details used for pattern matching
  5. exceptionCode - Special handling for exceptions (e.g., code "67")

Status Code Translation Logic (from actual implementation):

// Example of complex mapping logic
IT: {
  DP: function(entity, sourceData): number {
    const locationType = sourceData.locationType as string;
    const eventDescription = sourceData.eventDescription as string;

    if (locationType === "ORIGIN_FEDEX_FACILITY") {
      return 3100; // Received by Carrier
    }
    if (/Departed FedEx hub/i.test(eventDescription)) {
      return 3250; // In-Transit
    }
    return 3004; // Departed, In-Transit
  }
}

AI-Powered Missing Event Generation:

  • Missing 3100 Detection: isMissing3100() checks if the "Received by Carrier" event is missing.
  • Base Event Identification: get3100BaseEvent() finds the appropriate base event (status 3001).
  • Supplement Event Creation: createSupplementEvent() generates missing events and attributes them to the system.

SF Express Integration (operators/sfex.ts)

Data Source: SF Express API routes array
Processing Order: Chronological event sequence

Key Decision Factors:

  1. secondaryStatusCode - Primary status identifier (101, 201, 204, 205, 301, etc.)
  2. opCode - Operation code (30, 31, 36, 50, 54, etc.)
  3. remark - Chinese descriptions used for pattern matching
  4. secondaryStatusName - Status descriptions for customs processing

Complex Mapping Example (from actual implementation):

"201": function(entity, sourceData): number {
  const map = {
    "30": 3001, // Logistics In-Progress
    "31": 3002, // Arrived, In-Transit
    "36": 3004, // Departed, In-Transit
    "105": 3250, // In-Transit
    "106": 3300, // Arrived At Destination
  };

  const remark = sourceData["remark"] as string;
  if (/完成分拣/.test(remark)) {
    return 3003; // Scanned, In-Transit
  } else if (/快件离开/.test(remark)) {
    return 3004; // Departed, In-Transit
  }

  return map[sourceData["opCode"] as string] ?? 3001;
}

AI-Powered Customs Enhancement:

  • Missing 3400 Detection: isMissing3400() identifies missing "Import Released" events.
  • Smart Base Event Selection: get3400BaseEvent() finds events that occur after customs processing (3350).
  • Intelligent Timing: Inserts supplemented events 1 second before the base event.

Location Optimization (SF Express Focus)

Location Processing Logic:

// Location enhancement example
const remark: string = (route["remark"] as string).trim();
if (remark.startsWith("快件途经")) {
  event.where = remark.substring(4); // Extract location after "package via"
} else {
  event.where = route["acceptAddress"] as string;
}

Optimization Benefits:

  • Standardizes Chinese location descriptions.
  • Extracts meaningful location details from remarks.
  • Provides consistent geographical context.

Complete Status Code Mapping Tables

FedEx Status Code Mappings

derivedStatusCode eventType Condition/Logic Eagle1 Code Description
IN OC Any 3000 Transport Bill Created
IT DR Any 3250 In-Transit
IT DP Origin FedEx Facility 3100 Received by Carrier
IT DP "Departed FedEx hub" 3250 In-Transit
IT DP Default 3004 Departed, In-Transit
IT AR Sort facility + "destination" 3300 At destination sort facility
IT AR Origin FedEx Facility 3100 Received by Carrier
IT AR Destination FedEx Facility 3300 At local FedEx facility
IT AR Default 3002 Arrived, In-Transit
IT IT Exception code "67" 3450 Final Delivery In-Progress
IT IT Default 3001 Logistics In-Progress
IT AF Any 3001 Logistics In-Progress
IT CC "Export" in description 3200 Customs Clearance: Export Released
IT CC "Import" in description 3400 Customs Clearance: Import Released
IT OD Any 3450 Final Delivery In-Progress
IT RR Any 3450 Delivery option requested
CD CD "Import" in description 3350 Customs Clearance: Import In-Progress
CD CD Default 3150 Customs Clearance: Export In-Progress
PU PU Any 3050 Picked up
DL DL Any 3500 Delivered
DE DE Any 3450 Final Delivery In-Progress
CA CA Any 3009 Process Stopped

SF Express Status Code Mappings

secondaryStatusCode opCode Condition Eagle1 Code Description
101 50, 54 Any 3100 Received by Carrier
201 30 Any 3001 Logistics In-Progress
201 31 Any 3002 Arrived, In-Transit
201 36 Any 3004 Departed, In-Transit
201 105 Any 3250 In-Transit
201 106 Any 3300 Arrived At Destination
201 * "完成分拣" (sorting complete) 3003 Scanned, In-Transit
201 * "快件离开" (package departed) 3004 Departed, In-Transit
204 * "清关中" (clearing customs) 3350 Customs Clearance: Import In-Progress
204 * Default 3001 Logistics In-Progress
205 * "已清关" (customs cleared) 3400 Customs Clearance: Import Released
205 * Default 3001 Logistics In-Progress
301 * "派送中" (out for delivery) 3450 Final Delivery In-Progress
301 * Default 3001 Logistics In-Progress
1301 70 Any 3300 Arrived At Destination
401 80 Any 3500 Delivered

AI Optimization Features

1. Missing Event Generation

FedEx - Missing 3100 Events:

  • Detection: Checks if events progress past 3100 without a 3100 event being logged.
  • Base Event Selection: Identifies the first 3001-04 event after pickup.
  • Timing: Creates a supplement event 1 millisecond before the base event.

SF Express - Missing 3400 Events:

  • Detection: Finds customs events (3350) that are followed by progression events but lack a 3400 event.
  • Smart Targeting: Focuses on status codes like [3004, 3250, 3450, 3500].
  • Intelligent Placement: Inserts the supplement event at the most logical point in the timeline.

Reference: Issue 171

2. Location Enhancement

SF Express Specific Optimizations:

  • Chinese Text Processing: Extracts useful location data from Chinese remarks.
  • Pattern Recognition: Identifies "快件途经" (package via) patterns.
  • Fallback Logic: Uses acceptAddress if remarks do not contain recognized location patterns.

Reference: SF Express Location Optimization Issue #32

Benefits and Impact

For Developers

  • Unified API Response: Uses consistent status codes across all carriers.
  • Complete Timeline Coverage: Ensures no critical events are missing.
  • Enhanced Location Data: Provides clear and standardized facility information.
  • Predictable Data Structure: Offers a reliable and expected response format.

For End Users

  • Comprehensive Tracking: Gives full visibility into the shipment's journey.
  • Clear Status Updates: Provides meaningful event descriptions.
  • Accurate Location Information: Adds better geographical context to updates.

Technical Implementation Highlights

Error Handling & Resilience

  • Graceful Degradation: Defaults to status 3001 for any unmapped events.
  • Exception Processing: Specially handles FedEx exception codes.
  • API Error Management: Robust error handling for when carrier APIs fail.

Performance Optimizations

  • Efficient Mapping Algorithms: Uses O(1) lookups for direct mappings.
  • Minimal Processing Overhead: Optimizes the generation of supplement events.
  • Token Management: Automatically renews FedEx API authentication efficiently.

Data Integrity

  • Event ID Uniqueness: Uses timestamps to identify events uniquely.
  • Chronological Sorting: Automatically orders events by their timestamp.
  • Duplicate Prevention: Checks if an Event ID exists before adding it.

Future Roadmap

Planned Enhancements

  • Issue #171: Improve missing event generation.
  • Issue #179: Integrate an AI LLM to make mapping even smarter.
  • Issue #180: Build a multi-version data service architecture.

Continuous Improvement Process

  1. Data Pattern Analysis: Regularly review real-world tracking data.
  2. Mapping Rule Refinement: Iterate and improve rules based on edge cases.
  3. AI Enhancement Evolution: Progressively upgrade optimization algorithms.

This documentation shows how Eagle1 transforms raw carrier tracking data into smart, developer-friendly responses. By combining rule-based mapping with AI optimization, the API delivers complete shipment visibility and an improved user experience.

Source reference

(C) 2025 Eagle1

Clone this wiki locally