Developer Documentations

Okytes developer documentation

This guide explains the implementation in this repository: where the code lives, how a download moves through the system, and how to build, test and extend it.

The implemented milestones are M1: the HTTP engine and CLI and M2: the desktop application. The desktop uses the real engine and SQLite store. Browser extensions, a daemon, BitTorrent, FTP/SFTP, media manifests, device sharing and release packaging are outside the current implementation. The product README describes a broader product scope; use this guide and the source when working on existing behavior.

Contents

• Architecture
• Source file guide
• Download lifecycle
• Persistence and recovery
• Desktop application
• CLI
• Build and development
• Tests and verification
• Making changes
• Troubleshooting

Architecture

Okytes separates download policy from HTTP, storage and presentation. Rust owns the authoritative queue and transfer state. React renders snapshots and sends management requests; it does not download files itself.

flowchart TD
    UI[React desktop UI] --> IPC[Tauri commands]
    IPC --> Session[Embedded desktop Session]
    Session --> Engine[Core Engine]
    CLI[CLI] --> Engine
    Session --> Store[SQLite store]
    Engine --> Transport[Transport trait]
    Engine --> Persistence[JobStore trait]
    Engine --> Files[PartialFiles trait]
    Transport --> HTTP[reqwest and rustls]
    Persistence --> Store
    Files --> Local[Native file adapter]

Loading

crates/core defines the traits and transfer rules. transport-http and persistence depend on core and implement its boundaries. Applications compose these implementations. Core has no Tauri, React or platform-specific dependency.

The desktop service is a separate Rust crate so that queue admission, controls and settings can be tested without starting a webview. Tauri handles native integration and delegates management to that service.

Source file guide

Paths below are relative to the repository root. Library Cargo.toml files declare their crate dependencies; the root manifest defines workspace membership and shared versions.

Core engine — crates/core

File	Responsibility
src/lib.rs	Exports core modules and documents the dependency direction and transfer invariants.
src/types.rs	Shared requests, job/segment records, remote identity, IDs, conflict strategies and validated engine configuration.
src/state.rs	Explicit job and segment state transitions. Keeps replay and repeated controls predictable.
src/engine.rs	Runs jobs: probing, recovery, range allocation, workers, retries, durable checkpoints, hashing and publication. Implements DownloadService.
src/range.rs	Byte-range allocation and splitting rules, including property tests for coverage and non-overlap.
src/scheduler.rs	Throughput EWMA, adaptive concurrency, origin connection tracking and retry timing utilities.
src/transport.rs	HTTP-independent transport contract and response metadata used by the engine.
src/store.rs	JobStore contract for jobs, segments, diagnostics and checkpoints.
src/files.rs	PartialFiles contract for creating and reopening partial files safely.
src/service.rs	DownloadService, progress records and bounded engine event types.
src/desktop.rs	GUI-independent DesktopService, management actions and versioned desktop DTOs.
src/error.rs	Typed engine, transport and persistence errors.
bindings/*.ts	Generated TypeScript definitions for Action, AddInput, Job, Segment, Settings and Snapshot. Regenerate from Rust rather than editing them by hand.

Engine::new receives an EngineConfig and shared implementations of Transport, JobStore and PartialFiles. Adding a job persists it as queued; an application runner must call execute_job to start work.

HTTP — crates/transport-http

File	Responsibility
src/lib.rs	HttpTransport: reqwest/rustls client, probes, manual redirects, origin admission, response validation and streamed response bodies.
tests/protocol.rs	Exercises real HTTP protocol behavior against the fault server, including validators and malformed range responses.

Origin limits belong to the shared transport instance. Two jobs downloading from the same origin share the cap. A response holds its permit while its body is consumed; cancellation releases it. Redirects also pass through origin admission.

Storage and files — crates/persistence

File	Responsibility
src/lib.rs	SqliteStore, schema initialization, serialized database operations, paged history, runnable queue queries, preferences and exclusive engine ownership.
src/files.rs	LocalFiles: restrictive partial creation and native no-follow/reparse protections when opening files.
tests/recovery.rs	Store recovery and persistence behavior.
tests/files.rs	Native partial-file safety behavior.

CLI — apps/cli

File	Responsibility
src/main.rs	Parses commands, validates configuration, composes the engine and adapters, runs foreground transfers and formats output.
tests/engine.rs	End-to-end engine behavior using the production adapters and fault server.
tests/process.rs	Process-level CLI recovery and the explicitly invoked release memory stress test.

Desktop backend — apps/desktop

File	Responsibility
service/src/lib.rs	Session: owns the embedded engine, transfer lease, running task handles, queue admission, settings, controls and snapshot rendering.
service/tests/session.rs	Real transfers, controls, settings, validation and full-stream progress without unsafe resume checkpoints.
service/tests/races.rs	Pause/cancel races, lag recovery and bulk pause behavior.
src-tauri/src/main.rs	Native entry point, typed command delegation, app directories, tray and completion/failure notifications.
src-tauri/build.rs	Tauri build integration.
src-tauri/tauri.conf.json	Window configuration, local assets, CSP and build commands. Native drag handling is disabled to allow HTML5 drops on Windows.
src-tauri/capabilities/main.json	Webview capability configuration. The current permission list is empty; native features are handled in Rust through the application commands.
src-tauri/icons/	Native PNG, ICO and raw RGBA icon assets.

Desktop frontend and tooling

File	Responsibility
src/main.tsx	Mounts React and imports styles.
src/App.tsx	Dashboard, bounded snapshot refresh, virtualized rows, add/import flows, controls, diagnostics, settings, shortcuts and speed graph.
src/api.ts	Typed Tauri invocation wrapper, URL import parsing, byte formatting and extension-based file categories.
src/style.css	Layout, themes, focus treatment and reduced-motion styling.
src/App.test.tsx	Frontend interaction, accessibility and virtualization tests. Unit tests isolate the command bridge; production uses native commands.
index.html	Local web entry document.
vite.config.ts	React/Tailwind build and frontend test configuration.
tsconfig.json	Strict TypeScript configuration.
eslint.config.js	Frontend lint rules.
package.json	Dependencies and development/check scripts.
pnpm-lock.yaml, pnpm-workspace.yaml	Locked frontend dependency graph and pnpm workspace settings.
scripts/icons.mjs	Reproducibly generates native icon assets.
scripts/native-e2e.mjs	Windows native WebView2 integration test, isolated profile, real transfers, forced termination and evidence capture.
README.md	Desktop-specific source-running instructions.

Test infrastructure and repository configuration

File	Responsibility
crates/test-support/src/lib.rs	Exposes dev-only test utilities.
crates/test-support/src/server.rs	Deterministic TCP HTTP server with configurable latency, throttling, disconnects, validators, ranges and corrupt responses.
crates/test-support/examples/desktop_fixture.rs	Long-lived deterministic HTTP fixture for the native desktop test. Prints its URL and expected hash.
Cargo.toml, Cargo.lock	Rust workspace and locked dependency graph.
rust-toolchain.toml	Pins Rust 1.95.0.
deny.toml	Dependency license/advisory policy, with narrowly documented Tauri dependency exceptions.
.github/workflows/ci.yml	Windows/Linux/macOS checks, generated-binding checks, frontend checks and a separate Linux release stress job.
.gitignore	Excludes build output, local databases/partials and private tooling/instructions.
docs/adr/0005-desktop-service-and-bounded-snapshots.md	Explains the desktop service boundary and bounded snapshot design.
docs/adr/0006-tauri-transitive-dependencies.md	Records Tauri dependency license and maintenance decisions.

Download lifecycle

1. Validate and queue

The caller submits a DownloadRequest, or a desktop AddInput batch. Rust validates HTTP/HTTPS URLs, destination paths, filenames and limits. Embedded URL credentials are rejected. The desktop rejects duplicate canonical URLs unless the user explicitly allows them.

The engine assigns a UUID and saves a queued record. The partial filename is associated with that job. Adding a job does not execute downloaded content.

2. Admit and probe

The CLI foreground runner or desktop queue runner starts execute_job. Engine guards prevent duplicate execution of a job in the same process. A database-scoped OS lease prevents two transfer-owning processes from writing through the same profile.

The transport probes the resource and returns size, range support and remote identity. Before resuming, the engine compares stored validators with the new response. A changed source must not contribute bytes to the existing partial file.

3. Choose a transfer mode

For a safely resumable source, workers receive disjoint inclusive byte ranges. Allocation begins conservatively and adapts using measured durable throughput. Tail splitting transfers ownership before another worker writes; it does not create duplicate writers.

Responses are checked for status, Content-Range, length and identity. Weak ETags are never sent as If-Range. A server ignoring a range request cannot cause its full response to be written at an arbitrary segment offset.

Sources without safe range resume use a full-stream path. This includes relevant weak/unvalidated or chunked responses. Interrupted full streams restart from zero. Their live progress is displayed, but it is not converted into resumable segment checkpoints.

4. Write and checkpoint

The durability barrier in engine.rs, commit_durable, enforces this order:

write bytes into the partial file
             ↓
         sync_data
             ↓
commit the segment checkpoint to SQLite

Only synced bytes become committed segment progress. A crash between syncing and committing can cause those bytes to be downloaded again; the checkpoint cannot promise bytes that were never synced.

Retries use exponential backoff with jitter. HTTP 429 and 503 respect Retry-After. Pause/cancel checks interrupt transfer work and admission waits; cleanup waits until active workers relinquish ownership.

5. Verify and publish

After transfer, the engine streams the file through SHA-256. If an expected hash was supplied, a mismatch fails the job. Without an expected publisher hash, computing a hash gives a local integrity fingerprint; it does not independently authenticate the publisher's content.

The verified hash is persisted before publication. The final filename is exposed only after verification, using the conflict strategy: numbered rename, fail or explicit overwrite. Preserve-existing publication uses a no-clobber operation; overwrite uses replacement semantics.

A crash during publication is recovered by checking the persisted verified hash against an already-published output. File existence alone is insufficient evidence of successful completion.

State and control semantics

The usual successful path is Queued → Probing → Downloading → Verifying → Completed. Paused, Failed and Cancelled represent interruption or failure paths. The exact allowed transitions are in state.rs.

• Resume requeues a paused or failed job; remote identity is checked again during execution.
• Cancel stops work and removes partial data after workers stop. History is retained.
• Download again creates a fresh job ID and preserves the original history/output.
• Remove from history deletes the record and any partial data, while preserving completed files.
• Rename is restricted to idle jobs; priority changes apply to queued, inactive jobs.
• Once verified publication is committed to proceed, controls cannot interrupt that protected publication phase.

Persistence and recovery

SQLite runs in WAL mode with synchronous=FULL. A dedicated thread owns the connection, receiving operations through a bounded mailbox of 64 commands. Callers receive acknowledgements after their database operation finishes.

Schema version 1 contains:

Table	Data
jobs	UUID, serialized job record, indexed/queryable priority and state.
segments	Segment ID, owning job ID and serialized range/checkpoint record.
settings	Named JSON/configuration preferences. Desktop preferences use desktop-v1.

History queries return at most 256 records. Queue admission reads at most 64 runnable records using a partial index, excluding terminal history. State-aware updates prevent stale worker metadata from overwriting concurrent pause/cancel requests. Segment ownership changes are committed atomically.

The .engine-lock lease is scoped to the canonical database path and released by the OS after a process crash. CLI control commands can access a shared database while its runner owns the transfer lease, but another transfer runner cannot take ownership simultaneously. CLI and desktop do not automatically share a profile: the CLI defaults to okytes.db in its working directory, while desktop uses the platform app-data directory.

URLs, destinations, history and settings are local database data. Source URLs can contain sensitive query parameters, so treat databases and diagnostic exports accordingly. Partial opens reject unsafe symlink/reparse targets; Unix-created private files use restrictive permissions. TLS certificate validation remains enabled.

Desktop application

Request path

For example, adding links follows:

App.tsx → api.add(input) → Tauri add command
        → DesktopService::add → Session → Engine::add → SQLite

The session's background runner checks admission every 100 ms and starts jobs according to priority and the configured simultaneous-download count. It tracks task handles and waits for workers before destructive controls. Bulk pause takes one admission lock, pauses the runnable queue and drains workers without a separate webview call for every row.

Snapshots and progress

The frontend refreshes authoritative snapshots approximately twice per second, with one refresh in flight. Each backend page contains at most 256 jobs; the UI bounds a history page to 2,048 entries and offers paging. Search and category/state filters operate on that loaded history page.

TanStack Virtual mounts only the visible rows plus overscan. A large history therefore does not create a DOM node for every persisted job. Rust engine events also use a bounded broadcast channel. A lagged Rust subscriber must reload authoritative state; the webview uses replacement snapshots instead of a stream of every worker update.

Display DTOs use floating-point byte values for TypeScript presentation. Engine range ownership and persisted offsets remain integer values; never feed display values back into checkpoint calculations.

Diagnostics have specific meanings:

• Bytes: committed range bytes, or synced session progress for a full stream.
• Speed: measured change in durable bytes over time; sync boundaries can make updates uneven.
• Workers: live transfer tasks, including origin waits and retry backoff, rather than merely persisted active segments.
• Response latency: time to headers including origin admission and redirects, not a network RTT measurement.
• Response status/latency: session diagnostics, not durable history guarantees.
• Unknown total: indeterminate progress; the UI does not invent a percentage.

Features and native boundaries

The UI implements URL/batch entry, explicit clipboard paste, text-file import, HTML5 drops, duplicate approval, search, category/state filters, controls, priority, history and per-job details. Settings persist destination, concurrency (1–32), light/dark/system theme and notifications. Keyboard shortcuts include Ctrl/Cmd+N, Ctrl/Cmd+F, F1 and Escape.

Radix dialogs provide focus handling. Styles include visible focus and reduced-motion support. The frontend loads local assets under a restrictive CSP and has no direct filesystem or shell plugin permissions. Tray actions and native notifications run in Rust. Clipboard access uses an explicit browser API gesture and reports permission failures.

CLI

Run commands from the repository root. These examples use Cargo so they work without installing a separate executable:

# Queue a download without starting it.
cargo run -p okytes-cli -- add https://example.com/file.zip --output ./downloads

# Run queued and crash-interrupted jobs in the foreground.
cargo run -p okytes-cli -- run

# Add and run immediately; preserve an existing destination by default.
cargo run -p okytes-cli -- add https://example.com/file.zip --output ./downloads --run

# Read a bounded history page as JSON.
cargo run -p okytes-cli -- --json list --offset 0

# Inspect or control a job using its UUID.
cargo run -p okytes-cli -- status <job-id>
cargo run -p okytes-cli -- pause <job-id>
cargo run -p okytes-cli -- resume <job-id> --run
cargo run -p okytes-cli -- cancel <job-id>
cargo run -p okytes-cli -- queue <job-id> 10

# Read configuration, or replace it with validated JSON.
cargo run -p okytes-cli -- config
cargo run -p okytes-cli -- config --set engine-config.json

--db <path> selects a profile. add also accepts --filename, --sha256 and --conflict rename|fail|overwrite. The CLI is a foreground application: exiting stops its transfer process, and a later run recovers eligible persisted jobs. It is not a background daemon.

Build and development

Use Rust 1.95.0, Node.js 24 and pnpm 11.11.0. Native desktop builds require platform libraries: MSVC/WebView2 on Windows, WebKitGTK and app-indicator libraries on Linux, and the macOS native build tools. The CI workflow records the Linux packages used by automation.

From the repository root:

cargo build -p okytes-cli --locked

From apps/desktop:

pnpm install --frozen-lockfile
pnpm tauri dev

# Build an executable without an installer.
pnpm tauri build --no-bundle

# Development executable for native verification.
pnpm tauri build --debug --no-bundle

pnpm dev alone starts Vite; it does not start the Rust engine or provide Tauri commands. Use pnpm tauri dev for a functioning desktop application. pnpm build checks TypeScript and generates web assets; it is not a native executable build.

For an isolated desktop profile, set absolute OKYTES_DATA_DIR and OKYTES_DOWNLOAD_DIR paths before launching. These are trusted process environment settings, not webview filesystem capabilities.

Regenerate shared TypeScript DTOs from the repository root:

cargo test -p okytes-core --lib --locked desktop::export_bindings
git diff -- crates/core/bindings

Tests and verification

Standard checks

Run from the repository root:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked
cargo deny check

Run frontend checks from apps/desktop:

pnpm format:check
pnpm typecheck
pnpm lint
pnpm test
pnpm build

The CI matrix runs these checks on Windows, Linux and macOS. Configuration of a CI job is not evidence that its latest execution has passed.

What the tests cover

Property tests check byte-range coverage/non-overlap and state behavior. The real HTTP fault server exercises throttling, latency, disconnections, incorrect lengths, malformed/ignored ranges, identity changes, redirects, expired URLs, retry responses and corruption. Persistence tests cover recovery and file access; service tests cover actual controls and queue races. The checkpoint ordering regression test protects the write/sync/commit barrier.

The explicitly invoked release stress test uses a multi-gigabyte transfer and hundreds of queued jobs:

cargo test -p okytes-cli --test process --release --locked -- --ignored --nocapture

For Windows native desktop verification, build the fixture from the repository root:

cargo build -p okytes-test-support --example desktop_fixture

Then, from apps/desktop:

pnpm tauri build --debug --no-bundle
pnpm test:native

The native test attaches Playwright to the actual WebView2 instance and downloads from the TCP fixture. Its loopback debugging port is enabled only for the test process. It checks force-quit recovery, hashes, UI controls, large lists, imports, persisted settings and native-rendered accessibility. Local evidence is written under an ignored tooling directory.

Recorded verification boundaries

M2 verification recorded 66 Rust tests and five frontend tests passing, plus a Windows x86_64 native run. That run recovered three 8 MiB files with matching SHA-256 after forced termination, exercised a 504-job list with 13 rendered rows, and reported no axe violations on the checked light/dark main views. These fixture results are not general performance or accessibility guarantees.

M1 separately recorded Windows x86_64 and Ubuntu/WSL x86_64 engine verification, including actual SIGKILL recovery on Linux and an explicit release memory stress run. That stress run was not repeated as part of M2.

M2 native builds/runs on other platforms, manual tray gestures, OS clipboard permissions, visible toast delivery and physical OS file dragging remain unverified. Installers, signing and notarization are not provided by the current local executable build.

Making changes

Change	Start here	Verify with
Transfer/recovery behavior	core/src/engine.rs and relevant boundary traits	Fault-server integration tests, checkpoint/recovery regressions.
Range allocation	core/src/range.rs	Coverage/non-overlap property tests and transfer tests.
HTTP response handling	transport-http/src/lib.rs	transport-http/tests/protocol.rs, extending the shared server for new faults.
Persistent record/schema	core/src/types.rs, persistence/src/lib.rs	Migration/recovery tests; explicitly handle existing databases.
Desktop control	core/src/desktop.rs, desktop/service/src/lib.rs	Service integration/race tests; regenerate bindings.
Native feature	desktop/src-tauri/src/main.rs and capabilities/config	Native verification and permission/CSP review.
UI behavior	desktop/src/App.tsx, api.ts, style.css	Typecheck, lint, frontend interaction/accessibility tests.

Keep correctness and validation in Rust, even when the UI also validates input. Preserve integer offsets, bounded resource use, remote identity checks and the durability barrier. Never remove partial data while workers can still write to it, or treat a persisted active segment as proof of a live task.

Update generated bindings whenever a DTO changes. For non-obvious architecture or dependency decisions, add an ADR under docs/adr/. When adding persistent fields, consider old serialized records and database versions; a successful fresh-profile test does not prove upgrade safety.

Troubleshooting

Symptom	Where to look
Job stays queued in CLI	add only queues unless --run is supplied. Start run, and inspect priority/configuration.
Another engine owns the database	Check for a desktop/CLI runner using the same profile. The OS lease prevents simultaneous transfer writers.
Source identity changed	Inspect validators and the job error. Use a fresh download for changed content rather than forcing an old checkpoint.
Full-stream progress disappears after restart	Full-stream bytes are session progress, not resumable checkpoints; restarting from zero is expected.
UI is running but commands fail	Start with pnpm tauri dev, not the Vite-only server. Check native startup errors and profile paths.
History item is absent from search	Search applies to the loaded bounded history page; navigate history pages.
Windows executable cannot be rebuilt	Stop running instances of that executable or fixture before rebuilding; Windows can lock the output file.
Native notification is missing	Check saved notification settings, OS permission/delivery settings and structured Rust warnings.

Rust logging uses tracing; RUST_LOG controls the subscriber filter. Use the job details, segment checkpoints and typed errors to distinguish a transport failure from a storage or control failure. Avoid publishing private source URLs or profile databases in bug reports.