ML-03: On-Device Inference with ONNX / TFLite

[jpdevhub]

The Idea

Right now, the freshness ML model runs on the server. We want to move it to the edge! Running the model directly in the browser using ONNX Runtime Web means zero latency, full offline capability, and better privacy.

What needs to be done

• Convert our PyTorch model to ONNX format.
• Integrate onnxruntime-web in the frontend.
• Update the PWA so it doesn't need to hit the /api/v1/scan-auto endpoint when offline.

This is a hardcore ML and Frontend task that will make the app extremely robust. Perfect for someone looking for a serious challenge!

[SHRESTHA-012]

HI @jpdevhub I would like to work on this issue under SSoC '26. Please assign it to me.

Approach:
Start by exporting the trained PyTorch freshness model to ONNX format using torch.onnx.export, then validate the converted model with onnxruntime in Python to confirm output parity. On the frontend, install onnxruntime-web and load the .onnx file either bundled with the app or fetched once and cached via the Service Worker in IndexedDB or the Cache API. Wire up the inference pipeline to preprocess camera frames (resize, normalize, tensorize) using canvas APIs, run InferenceSession.run(), and post-process the output logits into freshness scores. Finally, update the PWA's offline logic so that when /api/v1/scan-auto is unreachable, it falls back seamlessly to the on-device ONNX session — giving users full scan capability with zero network dependency.

[jpdevhub]

Your proposal is on the spot what we required. One blocker is the model weights are deployed on hugging face and is too large for github repo. Before i assign you let me know how will you recieve the weights, you can train the model as the notebooks are uploaded as of now will be removed in next update or i can also share the drive link of that.

[SHRESTHA-012]

@jpdevhub , I can work with the Drive link you mentioned , once I have the weights I'll validate the ONNX export locally before integrating.

[jpdevhub]

https://drive.google.com/drive/folders/1PPxASxSzHF-xFQvSMPCwNOOBSQDiG0Wx?usp=sharing
wait for tomorrow as the official SSoC website https://portal.socialsummerofcode.com is not live and fetching the project issues and prs so i will nt be able to merge the pr in the meantime. You can star and fork the repo and start see the overall structure and start working. Feel free to ask any issues.

[SHRESTHA-012]

Ok sure