Skip to content

rohitiyer1234/UR5_Project_backend

BranchesTags

Repository files navigation

Backend (OCR + RoboHub proxy) — Detailed Documentation

This document describes the backend/ service in full: purpose, architecture, how it integrates with Robo Hub and the frontend, and per-file / per-function documentation down to the function level. It follows the same format used previously for robo_hub so you can drop it into your repo as backend/README.md or module-level docs.

Table of Contents

  1. Project overview & purpose
  2. Quickstart install & run
  3. High-level architecture & event flow
  4. Socket.IO events (frontend ↔ backend ↔ robo hub)
  5. Files & module-level documentation — function-level details
  • app.py
  • robohub_client.py
  • camera/capture.py
  • camera/preprocess.py
  • ocr/engine.py
  • ocr/compare.py
  • utils/draw.py
  • test_workflow.py
  • requirements.txt & backend_installation_packages.txt
  1. Integration, debugging & troubleshooting guide
  2. Design notes & suggested improvements
  3. Appendix — quick reference snippets

Project overview & purpose

This backend/ repo is the OCR backend and proxy between the web frontend and the Robo Hub (which coordinates the UR robot). It:

  • Provides a Flask + Flask-SocketIO server (app.py) that accepts WebSocket connections from the frontend (HMI).
  • Runs an OCR pipeline (camera capture → preprocess → PaddleOCR via ocr.engine) to read component text on demand.
  • Forwards job/trigger messages to the Robo Hub via a Socket.IO client (robohub_client.py).
  • Forwards inventory updates received from Robo Hub to the frontend, and exposes control events for camera, benchmark, capture, and robot triggers.

Intended ports (convention used in your codebase):

  • robo_hub (Robo Hub server): 5002

  • backend (this server): 5001

  • frontend (Flask UI): 5000

Quickstart install & run

  1. Create virtual environment (recommended):
python -m venv backend_env
# Windows
backend_env\Scripts\activate
# Linux / macOS
source backend_env/bin/activate
  1. Install dependencies:
pip install -r backend/requirements.txt
# or, if needed:
pip install -r requirements.txt

(If you use backend_installation_packages.txt, consult it for system-level packages such as libgl1, ffmpeg, or packages required by paddleocr.)

  1. Start Robo Hub first (the hub must be available for backend to connect):
# from robo_hub/ or wherever hub.py lives
python robo_hub/hub.py
# or
python app.py  # if hub entrypoint is named app.py
  1. Start backend:
python backend/app.py
# Backend will attempt to connect to Robo Hub (http://127.0.0.1:5002) via RoboHubClient
  1. Start frontend:
# from frontend/ (if you run a separate frontend Flask)
python frontend/app.py
# Open http://127.0.0.1:5000/control

High-level architecture & event flow

  1. Frontend (HMI) connects via Socket.IO to Backend (http://localhost:5001).

  2. Backend uses a RoboHubClient (Socket.IO client) to connect to Robo Hub (http://localhost:5002.

  3. Robo Hub sends update_inventory events (a mapping of PN → [t1,t2,t3,t4]) to any connected client (backend).

  4. Backend normalizes that inventory and emits update_inventory to the frontend with shape { inventory: [{ part_number, tray1, tray2, tray3, tray4}, ...] }.

  5. Frontend renders the table and also requests get_inventory on load. On refresh, the frontend should either (a) re-request inventory, or (b) backend emits inventory on new connections.

When the frontend requests actions (set_job, reset_inventory, send_trigger, etc.), the backend forwards (via robohub_client) to Robo Hub. Replies/acks come back through Robo Hub and are forwarded to the frontend.

Textual flow (simplified):

Frontend -> Backend (socket) -> RoboHub (socket) -> UR Robot (TCP)
UR -> RoboHub -> Backend (via RoboHubClient.sio.on('ack')) -> Frontend
RoboHub emits update_inventory -> Backend -> Frontend

Socket.IO events (frontend ↔ backend ↔ robo hub)

This table summarizes the main events used by the system.

Frontend → Backend (events frontend emits)

Event name Payload example Purpose
ping_test {msg: "Hello OCR backend!"} Keepalive / connectivity test
get_inventory null Ask backend to request inventory from robo hub
reset_inventory null Request Robo Hub to reset inventory to defaults
set_job { part_number: "EM1-2U1", quantity: 10 } Set a job (forward to Robo Hub)
start_robot { part_number: "EM1-2U1", quantity: 10 } (optional) Start robot / start program
start_camera null Start local camera capture
stop_camera null Stop camera feed
set_benchmark { part_number: "G8NB-17SR" } Move robot to benchmark + capture
capture_component null Capture a component image + run OCR
submitPartRequest or UI action {part_number, quantity} (UI convenience)

Backend → Frontend

Event name Payload example / shape Purpose
pong_test { msg: "Pong from OCR backend!" } Ping response
update_inventory { inventory: [ {part_number, tray1, tray2, tray3, tray4}, ... ] } OR mapping {PN: [n1,n2,n3,n4], ...} Inventory updates; frontend expects inventory array form
video_frame { frame: "<base64 jpg data>" } Live JPEG frames from camera
benchmark_set { image: "<b64>", part_number: "...", texts: [...] } Benchmark capture result
component_captured { image: "<b64>", texts: [...], match: bool, score: float } OCR result for a captured component
status { msg: "..." } Misc status messages
job_invalid { part_number, requested, available } Invalid job (Robo Hub validation)
reload_required { part: "<PN>" } Notify HMI that trays are empty

Backend ↔ Robo Hub (via robohub_client client)

Event name (emitted) Payload example Purpose
send_trigger { trigger: "D", event: "program_started", part_number: "..." } Ask Robo Hub to send letter to UR
get_inventory null Ask Robo Hub to emit update_inventory
reset_inventory null Ask Robo Hub to reset inventory
set_job { part_number: "...", quantity: N } Set job in Robo Hub
Robo Hub → Backend emits update_inventory mapping { PN: [n1,n2,n3,n4], ... } Backend uses this to emit to frontend

Files & module-level documentation — function-level details

Below are the files in backend/ with detailed documentation, function signatures, descriptions, parameters, returns and usage examples.

app.py — Flask + Socket.IO server (entry point)

Purpose Top-level Flask + Flask-SocketIO server. Hosts Socket.IO endpoints that the frontend connects to. Runs the OCR flow, camera capture, and forwards messages to/from Robo Hub via RoboHubClient.

Key components declared in app.py - app — Flask application - socketio — SocketIO server (flask_socketio.SocketIO) - robohub_client — instance of RoboHubClient (connects to Robo Hub) - camera — Camera() (from camera.capture) - ocr_engine — OCREngine() (from ocr.engine) - part_command_map — mapping PN → letter for UR triggers - last_inventory — (optional) cache for last inventory mapping sent by Robo Hub

Important functions / Socket.IO handlers (function-level details)

live_feed_loop()

def live_feed_loop():
    while _live_feed_running and camera.is_running:
        frame = camera.get_frame()
        if frame is None:
            socketio.sleep(0.03); continue
        processed = preprocess_for_ocr(frame)
        _, jpeg = cv2.imencode(".jpg", processed)
        b64_frame = base64.b64encode(jpeg.tobytes()).decode()
        socketio.emit("video_frame", {"frame": b64_frame}, namespace="/")
        socketio.sleep(0.03)
  • Purpose: background loop to send live camera JPEG frames to all connected frontends.
  • Side-effects: emits video_frame events frequently (≈30 fps if camera can).
  • Notes: uses socketio.sleep to cooperate with eventlet or other async workers.

start_live_feed() / stop_live_feed()

  • start_live_feed starts the background greenlet/thread with live_feed_loop.
  • stop_live_feed sets control flag to stop the loop.

run_capture_component()

  • Captures a frame, runs OCR, finds best match to part_number, logs results, and emits component_captured.
  • If match found, forwards START_PROGRAM trigger to Robo Hub.
  • Uses robohub_client.send_part_code() and robohub_client.wait_for_ack() to synchronize robot movement.

Socket.IO handlers (frontend → backend)

Handler Signature Purpose
handle_start_camera @socketio.on("start_camera") Start camera + begin live_feed_loop. Emits status.
handle_stop_camera @socketio.on("stop_camera") Stop camera/live feed. Emits status.
handle_set_benchmark @socketio.on("set_benchmark") Ask Robo Hub to position robot for benchmark, capture frame, run OCR, emit benchmark_set.
handle_capture_component @socketio.on("capture_component") Start run_capture_component in background.
handle_set_job @socketio.on("set_job") Forward job to Robo Hub via robohub_client.sio.emit("set_job", {...}).
handle_reset_inventory @socketio.on("reset_inventory") Forward reset command to robo hub (robohub_client.sio.emit("reset_inventory")).
handle_get_inventory @socketio.on("get_inventory") Forward get_inventory to Robo Hub (or emit cached last_inventory if not connected).
handle_start_robot @socketio.on("start_robot") Forward send_trigger start request to Robo Hub.
handle_reload_required @socketio.on("reload_required") Emit reload_required to frontends.
handle_ping @socketio.on("ping_test") Return pong_test for connectivity testing.

Robo Hub client → backend handlers

app.py registers robohub_client.sio.on("update_inventory") which normalizes the mapping and emits update_inventory to the frontend with shape:

{ "inventory": [ { "part_number": "...", "tray1": n, "tray2": n, "tray3": n, "tray4": n }, ... ] }

It keeps last_inventory as raw mapping and only broadcasts when the mapping changes.

Usage

  • Run python backend/app.py.
  • Frontend connects to http://127.0.0.1:5001 and will receive update_inventory, video_frame, etc.

robohub_client.py — Robo Hub Socket.IO client

Purpose

A small Socket.IO client used by the backend to talk to the Robo Hub server (which itself is a Socket.IO server). The class encapsulates connection, emit wrappers, ack handling, and helper methods for sending triggers/part codes and waiting for acknowledgments.

Class: RoboHubClient Constructor

def __init__(self, hub_url="http://127.0.0.1:5002"):
    # sets self.sio = socketio.Client(...)
    # sets self.hub_url, self.acks = {}, _connect_lock
    # registers callbacks on self.sio for "ack", "update_inventory", "reload_required"

Key Methods

Method Signature Description
connect connect(self, timeout=5.0) Connects to Robo Hub using self.sio.connect(hub_url, namespaces=['/']). Waits until connected or timeout. Returns True/False.
_ensure_connected — Attempts reconnection if sio is not connected. Called internally before emits.
send_trigger send_trigger(self, event_name) Map logical event name to single-letter trigger; emits send_trigger to Robo Hub and marks acks[event_name] = False to be awaited.
send_part_code send_part_code(self, part_letter, part_number="") Emit a send_trigger with trigger=part_letter and event=PART_{part_number}.
wait_for_ack wait_for_ack(self, event_name, timeout=10) Poll self.acks and wait until event ack is True, or timeout. Returns boolean.

Socket callbacks registered in constructor

  • @self.sio.on("ack") → sets self.acks[event] = True and prints log.
  • @self.sio.on("update_inventory") → prints debug.
  • @self.sio.on("reload_required") → prints debug.

Usage Example

client = RoboHubClient("http://127.0.0.1:5002")
client.connect()
client.send_part_code("A", "G8NB-17SR")
client.wait_for_ack("PART_G8NB-17SR", timeout=8)

Errors & pitfalls

  • Make sure the Robo Hub server is running and reachable. If connect() times out, emit calls will fail or be no-ops.
  • The client uses the root namespace '/'. If the server uses a different namespace, you must update both sides.

camera/capture.py — Camera threaded capture class

Class: Camera Purpose

Threaded camera capture wrapper using OpenCV VideoCapture. Keeps latest frame in memory protected by a lock, and supports start(), stop(), and get_frame().

Key attributes

  1. camera_index — index for cv2.VideoCapture (default 1 in your copy).
  2. cap — cv2.VideoCapture instance.
  3. frame — last captured frame (BGR numpy array).
  4. is_running — bool flag.
  5. lock — threading.Lock to guard frame.
  6. thread — background thread that runs update().

Methods

Method Signature Description
start() start(self) Opens the cv2 capture and starts a daemon thread calling update().
update() update(self) Reads frames in a loop; on success writes to self.frame under self.lock.
get_frame() get_frame(self) Returns a copy of the last frame or None. Copy prevents callers from mutating the internal array.
stop() stop(self) Releases capture, stops thread loop, clears frame.

Usage

cam = Camera(0)
cam.start()
time.sleep(1)
frame = cam.get_frame()
cam.stop()

Notes

  • Use cap = cv2.VideoCapture(index, cv2.CAP_DSHOW) on Windows to reduce lag and warnings (your code does this).

  • get_frame() returns None until the camera has produced a frame.

camera/preprocess.py — Preprocessing pipeline for OCR

Purpose Contains preprocess_for_ocr(frame_bgr) which converts BGR camera frames to a preprocessed image suitable for OCR (PaddleOCR). Includes grayscale conversion, bilateral filtering, sharpening, adaptive thresholding, deskew and returns BGR image for PaddleOCR.

Function

def preprocess_for_ocr(frame_bgr):
    # returns processed_frame_bgr

Processing steps (detailed)

  1. Convert to grayscale.
  2. Bilateral filter with reduced strength to keep edges.
  3. Sharpen with a mild kernel.
  4. Adaptive threshold (Gaussian) with smaller block size for smaller text.
  5. Deskew using cv2.minAreaRect on non-zero pixels.
  6. Convert back to BGR before returning (PaddleOCR expects BGR).

Why these steps? Improves text contrast, reduces noise, and normalizes rotation so OCR accuracy increases on small printed labels.

Example

from camera.capture import Camera
from camera.preprocess import preprocess_for_ocr
cam = Camera(0)
cam.start()
time.sleep(1)
frame = cam.get_frame()
proc = preprocess_for_ocr(frame)  # now feed proc into OCR

ocr/engine.py — OCR Engine (PaddleOCR wrapper)

Class: OCREngine

Purpose Wraps the PaddleOCR library, provides preprocess() and run_ocr() helpers and returns normalized output shape (list of rec_texts and det_polygons).

Constructor

def __init__(self, model=None):
    self.ocr = PaddleOCR(use_angle_cls=True, lang='en')

Methods

  • preprocess(self, frame) — delegator to camera.preprocess.preprocess_for_ocr.
  • run_ocr(self, frame) — runs OCR and adapts PaddleOCR output variations into canonical structure:

Return:

{
  "processed_frame": <preprocessed frame>,
  "det_polygons": [ list of boxes ],
  "rec_texts": [ {"text": "...", "score": <float> }, ... ]
}

Notes

PaddleOCR outputs have had several API versions; run_ocr() contains fallback handling for both old [(box, (text, score)), ...] format and newer dictionary format.

Usage

ocr = OCREngine()
result = ocr.run_ocr(frame)
texts = [t["text"] for t in result["rec_texts"]]

ocr/compare.py — Text comparison helpers

Contains: TextComparator class

Purpose Utilities for comparing recognized OCR text to benchmark strings (part numbers). Provides cleaning, similarity, and best-match logic that can combine adjacent recognized fragments to handle split OCR outputs.

Key methods

  • clean_text(s) — lowercases and strips non-alphanumeric chars.
  • best_match(benchmark_texts, component_texts, threshold=0.6) — tries single string matching and pairwise concatenation of component fragments. Returns (match_bool, (benchmark,component), score).
  • compare_texts(a,b) — wrapper using difflib.SequenceMatcher.
  • similarity(a,b) — same as compare_texts (convenience).

Usage

ok, (bench, comp), score = TextComparator.best_match(["G8NB-17SR"], ["g8nb","17sr"])
if ok:
    # treat as matched

Notes

  • Designed to handle OCR split tokens, hyphens, whitespace and OCR noise.

utils/draw.py — Drawing OCR overlays

Function: draw_ocr_overlays(frame, result)

Purpose Annotate a frame (BGR) with polygons, bounding boxes and text labels from OCR output. Used to produce the overlay image sent to the frontend on benchmark_set / component_captured.

Input forms handled

  • result can be:
    • dict with det_polygons and rec_texts, where rec_texts is a list of dicts with text and score.
    • list of {text, score} dicts.

For each polygon, draws a green polyline and red text label with score.

Returns: the annotated frame.

Usage

overlay = draw_ocr_overlays(frame.copy(), result)
_, jpeg = cv2.imencode('.jpg', overlay)
b64 = base64.b64encode(jpeg.tobytes()).decode()
socketio.emit('component_captured', { 'image': b64, ... })

test_workflow.py — Integration test harness

Purpose Script used to exercise the OCR + robo hub flow locally without the full UI. Useful for development and CI.

Typical content / behavior

  • Start camera or read a sample image.
  • Run the OCR engine on the image.
  • Optionally mock RoboHubClient interactions (emulate ack).
  • Print results and scores, save overlay images to disk for inspection.

How to use

python backend/test_workflow.py
# Inspect console messages and generated debug images

requirements.txt & backend_installation_packages.txt

requirements.txt(important Python packages likely included)

  • Flask
  • Flask-SocketIO
  • python-socketio
  • eventlet or gevent (your code uses eventlet as async_mode)
  • opencv-python
  • paddleocr (heavy; may require paddlepaddle; see next)
  • paddlepaddle(CPU/GPU install varies — consult Paddle docs)
  • numpy
  • requests
  • pillow (if used)
  • other utilities

backend_installation_packages.txt

  • may contain platform/system packages required for opencv display, paddlepaddle or other native libs (e.g. libglib2.0-0, libsm6, libxrender1).

Important: paddleocr/paddlepaddle have special install instructions depending on CPU/GPU. Follow their installation docs — on Windows, GPU support is non-trivial.

Integration, debugging & troubleshooting guide

Below are practical steps and checks for common problems you described (table not updating, reset failing, frontend not printing logs, inventory disappearing after refresh):

Common checks (quick)

1. Ports and URLs

  • Backend listens on 5001 (per backend/app.py).
  • Robo Hub listens on 5002.
  • Frontend connects to backend at http://localhost:5001 — make sure frontend JS matches.

2. Namespace mismatch errors

  • Error "/ is not a connected namespace" means the socketio.Client.connect(..., namespaces=['/']) call was not successful or the server didn't accept the namespace.
  • Ensure Robo Hub server and client both use the root namespace '/' (or change both to use a specific namespace).

3. No frontend logs & table disappears after refresh

  • Verify browser console: any socket connection errors, CORS blocked, 404 for socket.io or scripts.js?
  • On page load, your scripts.js should call socket.emit('get_inventory') — backend should respond by forwarding to Robo Hub or by returning cached last_inventory. If Robo Hub client disconnected, backend should fallback to last_inventory. Your backend @socketio.on('connect') emits cached inventory if present — ensure last_inventory is populated.
  • Use console.log and log() (existing functions) in scripts.js to see events.

4. Reset doesn't work — logs show Failed to forward reset_inventory to Robo Hub: / is not a connected namespace

  • Occurs when robohub_client.sio.connected is False. Fixes:
    • Ensure robohub_client.connect() succeeded on backend startup. Check backend startup logs for Connected at http://127.0.0.1:5002.
    • Add fallback in backend handle_reset_inventory() that, if robohub not connected, directly resets inventory.json (or emits a reset_inventory_result event) so frontend won't be stuck.

Example fallback:

if not robohub_client.sio.connected:
    inv = load_inventory()
    inv["current"] = inv["default"].copy()
    save_inventory(inv)
    socketio.emit("update_inventory", { "inventory": normalized_from(inv["current"]) }, namespace="/")
else:
    robohub_client.sio.emit("reset_inventory")

5. Inventory only displays once and then disappears

  • Ensure update_inventory events sent by backend are in the array-normalized form your frontend expects ({inventory: [...]}).
  • Make frontend robust: accept either mapping {PN: [..], ...} or normalized { inventory: [...] }. Your scripts.js already contains robust handling; ensure the backend actually sends {inventory: normalized} consistently.
  • Also ensure on socket.connect event the backend emits or the frontend requests get_inventory.

6. Latency / lag in updates

  • Inventory watcher polls inventory.json every 0.5s. If Robo Hub updates inventory.json less frequently, there will be apparent lag.
  • If Robo Hub is writing big files synchronously, reads from backend may block — use small locks or send inventory updates over Socket.IO from Robo Hub instead of file watching.

Step-by-step debug checklist

1. Confirm Robo Hub is up

  • Terminal with Robo Hub should show Client connected — sending current inventory: and printed map.

2. Confirm backend connected to Robo Hub

  • Backend logs should have [Backend → Robo Hub] Connected at http://127.0.0.1:5002 (or similar).

3.Confirm frontend connects to backend

  • Browser console: ✅ Pong: ... or log messages from scripts.js.
  • Backend console should show Ping received from front-end: ... or Frontend connected.

4. Manually test inventory flow

  • In Robo Hub terminal, force an update_inventory by touching inventory.json or using its file watcher.
  • Check backend logs for update_inventory from Robo Hub and frontend logs for update_inventory received.

5. Test reset

  • Click Reset on frontend; ensure Robo Hub prints Resetting inventory to defaults and Emitted update_inventory.
  • If not, check Robo Hub and backend connected states.

Example expected logs (happy path)

[Robo Hub] Client connected — sending current inventory: {...}
[Backend] update_inventory from Robo Hub: {...}
[Backend] Forwarded normalized inventory to frontend: [ {...}, ... ]
[Frontend] update_inventory received: { inventory: [ ... ] }
[Frontend] Inventory updated (array)

If you see No change in inventory, skipping emit, that's because the mapping equals last_inventory — expected behavior to avoid spam.

Design notes & suggested improvements

  • Robust reconnects: Add retry logic and exponential backoff for RoboHubClient.connect(). Already present but can be hardened.
  • Better fallback for offline Robo Hub: If Robo Hub is temporarily down, backend should still be able to serve last_inventory to frontend on connect. app.py already caches last_inventory — ensure @socketio.on('connect') always emits cached data (it does in your later revisions).
  • Atomic file operations for inventory.json writes to avoid race conditions (write to tmp file + rename).
  • Event tracing: Add correlation IDs to triggers so logs can match send → ack flows.
  • Persist last_inventory to disk if you need state to survive backend restart.
  • Decouple camera processing: run heavy OCR in a worker process to avoid blocking SocketIO; use a Queue.

Appendix — quick reference snippets

Emit normalized inventory (backend)

normalized = [
  {"part_number": pn, "tray1": c[0], "tray2": c[1], "tray3": c[2], "tray4": c[3]}
  for pn, c in curr.items()
]
socketio.emit("update_inventory", {"inventory": normalized}, namespace="/")

Frontend update_inventory handler (robust)

socket.on("update_inventory", (data) => {
  if (data && Array.isArray(data.inventory)) {
    renderInventoryTable(data.inventory)
  } else if (data && typeof data === 'object') {
    // mapping -> convert
    const firstKey = Object.keys(data)[0];
    if (firstKey && Array.isArray(data[firstKey])) {
      const normalized = Object.entries(data).map(([pn, arr]) => ({
        part_number: pn, tray1: arr[0]||0, tray2: arr[1]||0, tray3: arr[2]||0, tray4: arr[3]||0
      }));
      renderInventoryTable(normalized);
    }
  }
});

Safe-forwarding reset in backend (fallback pattern suggestion)

@socketio.on("reset_inventory")
def handle_reset_inventory():
    if robohub_client.sio and robohub_client.sio.connected:
        robohub_client.sio.emit("reset_inventory")
    else:
        inv = load_inventory()
        inv["current"] = {k: v.copy() for k,v in inv["default"].items()}
        inv["job"] = {"part_number":"—", "quantity":0}
        save_inventory(inv)
        normalized = [...]
        socketio.emit("update_inventory", {"inventory": normalized})

About

No description, website, or topics provided.

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

1 tags

Packages

No packages published

Languages