Record Thing

Overview

A comprehensive solution for recording, recognizing, and organizing physical items using computer vision and machine learning. It presents the user with a iOS/Android/macOS app to record things and events in the world and organise the recordings so they can be used for various purposes. Recorded things can be used to prove claims, be shared with other users, and be used to create a community showcase. As a user you can request a recording from another user.

Record Thing helps users:

• Scan and catalogue physical items
• Organize items by categories
• Collect evidence about items (receipts, photos, etc.)
• Manage requests related to items
• Make a recording based on a pre-defined workflow
• Share content from other apps (YouTube, web pages) into strategic focus areas
• Organize knowledge around strategic interests through the Thepia Strategist system
• Sync data automatically across devices using iCloud Documents

Project Structure

The project consists of both client applications and backend tools:

• iOS and macOS Apps - Swift/SwiftUI applications in apps/RecordThing
• Android App - TBD
• Python Backend - Data management, ML processing, and sync in libs/record_thing

Technology Stack

• Swift/SwiftUI: Main iOS and macOS app implementation
• Python: Backend tools, database setup, and ML processing
• SQLite/Blackbird: Database with Blackbird as Swift ORM
• MLX: Apple's machine learning framework for Apple Silicon
• DINO v2: Self-supervised learning model for computer vision
• B2/Bunny CDN: Asset synchronization and storage

Getting Started

Setting Up the Database

# Initialize a new database or view existing one
uv run -m record_thing.cli init-db

# Force reset an existing database
uv run -m record_thing.cli init-db --force

# Update database schema without losing data
uv run -m record_thing.cli update-db

# Create tables only (no sample data)
uv run -m record_thing.cli tables-db

# Populate database with sample data
uv run -m record_thing.cli populate-db

# Test database connection with verbose output
uv run -m record_thing.cli test-db -v

For more detailed CLI commands, see CLI Documentation.

iCloud Syncing

The iOS/macOS app automatically syncs data across devices using iCloud Documents. See iCloud Sync Documentation for details on setup, monitoring, and troubleshooting.

Alternatively, open the Jupyter notebook: docs/record-thing.ipynb

Sync with Server

The server storage is a Storage Bucket (Bunny CDN) that syncs with the local database and local recording files.

uv run buckia --config ./assets
uv run buckia sync --folder=13434535345

Building the iOS/macOS App

iOS:

cd apps/RecordThing
xcodebuild -scheme "RecordThing iOS" -configuration Debug -derivedDataPath ./DerivedData -destination "platform=iOS Simulator,name=iPhone 14 Pro"

macOS:

cd apps/RecordThing
xcodebuild -scheme "RecordThing macOS" -configuration Debug -derivedDataPath ./DerivedData

Swift Development Environment

brew install xcode-build-server xcbeautify swiftformat

Python Environment Setup

Using pip:

uv run -m venv .venv
source .venv/bin/activate
pip install -e .

Using uv (faster):

uv venv
source .venv/bin/activate
uv pip install -e .

Documentation

For more detailed information about specific components:

• CLI Commands - Command-line interface documentation
• Navigation - UI navigation and component structure
• Database Schema - Database structure and relationships
• Branding Guidelines - Visual identity specifications
• QA Process - Testing and quality assurance
• Apple Enrollment - Apple Developer Program enrollment

Architecture

The Record Thing ecosystem consists of these main components:

Record Thing Ecosystem
├── RecordThing App (Swift/SwiftUI) - iOS/macOS client
├── RecordLib Library (Python) - Core data management functionality
└── Backoffice Library (Python) - Server-side and admin tools
    └── Buckia - Synchronization with Storage Buckets

Dataset

The project uses the ICDAR2019 SROIE dataset for receipt recognition:

• ICDAR-2019-SROIE
• Papers with Code: SROIE

iOS Sharing Implementation

RecordThing includes an iOS Share Extension that allows users to share content from other apps directly into the app:

• YouTube Video Sharing: Share videos from the YouTube app into strategic focus areas
• Web Content Sharing: Share any web page from Safari or other browsers
• Thepia Strategist: Organize shared content around strategic interests and projects
• Evidence Linking: Connect shared content to physical items or strategic focus areas

For detailed implementation information, see docs/SHARING_IMPLEMENTATION.md.

Services and Accounts

• Apple Developer (apps@thepia.com)
• together.ai for ImageGen

License

See LICENSE for details.

Auth prompt

I want to use keychain/keyring functionality on my website to authenticate users and protect access tokens and other secrets for accessing backend services. This should enable a single registration with touchid/faceid that can be used across apple devices for the same user. It should be possible to replicate the authentication approach in other contexts such as python cli or swiftui