v2.3.0 — Bounded TikTok retries and operation progress

TikTok downloads now recover from stalled connections and response bodies within a shared operation budget instead of letting one stalled source consume the entire deadline.

• Add the opt-in NDJSON operation protocol with up to three real attempts, remaining-time propagation, cancellation, and conclusive delivery-safety results.
• Preserve streaming upload commit protection: ambiguous or partially committed Telegram sends are never blindly retried.
• Bound header/body stalls and preserve identity recovery through media inspection, with sanitized phase diagnostics.
• Prevent failed optional thumbnails from consuming retry attempts or repeating successful media preparation.
• Keep existing JSON clients compatible. Document the bot's one-second terminal-response reserve inside its 15-second total limit.

Deployment order: release and verify this backend before tt-bot v6.0.18. No database migration or new production environment variable is required for this rollout.

Included: #32

v2.2.1 — Instagram session continuity and account health

Instagram account health and session continuity fixes:

• Stop the current extraction or import batch on explicit challenge, checkpoint, feedback, or account restrictions.
• Persist account cooldowns and honor Retry-After through restarts, renewal, and credential updates.
• Preserve isolated server-issued cookies and session headers across login, metadata requests, and restarts.
• Use a consistent Android TLS/HTTP/2 transport for private API requests and recover closed connections without replaying credential submissions.
• Add safe health diagnostics and regression coverage for concurrency, persistence, proxy routing, and session isolation.

Database migration adds nullable cooldown_until and session_state columns. Existing disabled accounts are not automatically reactivated. Older versions ignore the new cooldown/session-state guarantees.

Validation: complete race-enabled tests with PostgreSQL, vet, static analysis, vulnerability checks, build, and live saved-session media extraction/downloads. This release does not guarantee avoidance of Instagram verification or bot-detection restrictions.

Changes: #28

v2.2.0 — Instagram cookie imports and reliable account states

Instagram accounts can now use browser cookies with saved password/2FA recovery. Ambiguous failures no longer permanently exclude working accounts.

Changes

• Import browser cookie exports directly, or include cookies with username, password and TOTP credentials in JSON/CSV. Working cookies take priority over password login.
• Update existing accounts through either credential format. A verified replacement cookie session can reactivate an account; login-only updates retain its existing session and dead status.
• Recover explicitly expired sessions using saved password/2FA credentials on the same device and proxy. Concurrent requests share one recovery attempt, and replacement sessions are saved before use.
• Retry bare HTTP 401/403 responses, network errors, malformed metadata and unknown API failures on another eligible account without marking the failed account dead. Retries do not revisit accounts already tried for that request.
• Continue disabling accounts for explicit restrictions/challenges or unrecoverable session/credential failures, with specific persisted reasons. Rate limits use temporary cooldowns; ambiguous login/bootstrap failures preserve the account.
• Prevent late failures from an old session from disabling its verified replacement.

Upgrade

Startup adds cookie and login-credential columns and makes stored Android authorization nullable. Keep the existing database/schema and proxy configuration; stop the previous instance before starting its replacement. Older releases cannot load cookie-only rows, so account for this before rolling back after cookie imports.

Password/TOTP backups are retained privately for recovery and are not returned by the account API. Existing dead flags are not automatically cleared: import a verified replacement cookie session to reactivate an account. Adding login credentials alone does not reactivate it.

Account import and update guide · Configuration and failure handling

Validation

The release tree matches the reviewed and tested PR #27 head. Regression tests reproduce the original failures on the previous implementation and pass with the fix. Local make check and make lint passed, including PostgreSQL integration and race tests. PR CI passed tests, Docker and Railpack checks, with no unresolved review findings.

PR #26 · PR #27 · Full changes since v2.1.0

v2.1.0 — Native Instagram accounts and extraction

Native Instagram extraction and persistent account imports are now available. The Android app method is the default; the previous RapidAPI method remains available directly or as an optional fallback.

Changes

• Import Instagram accounts through POST /v1/instagram/accounts/imports using CSV or JSON. Imports select and check a proxy, perform password/TOTP login, verify the session, and save it in PostgreSQL for immediate use and restart recovery.
• Generate and persist a Pixel or Samsung device identity per imported account. No phone, ADB, Python, or instagrapi is required at runtime.
• Keep each account on its assigned proxy route; multiple accounts can share one IP. Use static/sticky proxies to preserve the actual egress IP.
• Use the oldest healthy accounts first, balance requests across them, and promote reserves when an active account is marked dead. Metadata and download concurrency are controlled independently per account.
• Return richer Instagram metadata, including method, resolved_url, likes, views, comments, dimensions, duration, covers, and music metadata. Unavailable values remain null. Instagram music downloads are not included: music.audio is null.
• Record direct legacy and fallback attempts, including failures, in scrap_instagram_attempts.legacy_used.
• Stop remaining import rows after upstream rate limiting, preserve the saved account's user agent for CDN downloads, and allow pinned downloads to continue during account-death persistence failures.

Upgrade

Deploy this tt-scrap version before the updated tt-bot client, which requires method and resolved_url in Instagram extraction responses.

For an initial rollout with the old provider available as fallback:

INSTAGRAM_METHOD=app
INSTAGRAM_ACTIVE_ACCOUNTS=10
INSTAGRAM_ACCOUNT_CONCURRENCY=1
INSTAGRAM_ACCOUNT_DOWNLOAD_CONCURRENCY=1
INSTAGRAM_LEGACY_FALLBACK=true

Keep the existing RAPIDAPI_KEY when enabling fallback. Fallback is disabled by default; it applies to extraction failures, not subsequent download failures. Set INSTAGRAM_METHOD=legacy to use only the previous provider while retaining the updated response format.

Keep the existing database/schema, API key, and proxy configuration. Startup automatically creates the Instagram account and attempt tables and adds persisted device state; the database role needs create/alter permissions. Run only one tt-scrap instance per schema and stop the previous instance before starting its replacement.

Import accounts into the production database after deployment. CSV First Secret Question contains the authenticator TOTP secret; a leading apostrophe is removed from usernames. Session seed files are optional when using API imports. Accounts become available without restarting. Without usable accounts or configured legacy fallback, readiness returns 503; account administration remains accessible through a reachable service origin.

After import, check the account list, extract a fresh post, confirm method: "app", and download its returned asset. Readiness alone can succeed through fallback.

Account import guide · Instagram configuration and telemetry

Validation

• CI passed on release commit 3a02f62: tests with the race detector, vet, build, staticcheck, govulncheck, Docker, and Railpack checks.
• Live verification on 2026-10-30 successfully imported the development account through the HTTP API in an isolated database schema and downloaded a Reel and a photo using a saved session. This does not verify every account or the production deployment.

Pull request #25 · Full changes since v2.0.5

v2.0.5

Reduce CPU work and temporary memory during Telegram downloads and delivery.

• Skip unused SHA-256 work for Telegram while preserving checksums and ETags on raw downloads.
• Copy ready bytes directly from fixed RAM spools, with bounded writes and memory reservations retained until borrowed bytes are released.
• Reuse decoded image pixels, scale into one padded canvas, and release converted sources and completed album batches earlier.
• Give optional covers up to 50 ms of selection grace once media is upload-ready, within the existing total cover timeout. Slower covers may be omitted; cancellation still joins the cover worker before identity release.
• Preserve media validation, identity recovery, cancellation, the final 64 KiB upload gate, and existing resource limits.
• Update Go and the Docker build image to 1.27.3 to resolve GO-2026-6809.

Upgrade: no new configuration or database migration is required. Network-limited transfers may see little latency change. Tiny converted albums retain a tradeoff of lower memory use with higher CPU usage.

Validation: PR CI passed race-enabled tests, vet, build, staticcheck, govulncheck, Docker, and Railpack checks. The optimized version also passed a test-bot delivery smoke check. No production bot was used for testing.

Pull request #22 · Full changes since v2.0.4

v2.0.4

Improve cancellation diagnostics, accept videos with inaccurate dimension metadata, and harden deployments that share a single identity pool.

• Report caller cancellations as 499 warnings and shutdown cancellations as 503 service_shutting_down errors. Preserve cancellation through request-body parsing and Telegram response reads.
• Report interrupted response streams with their actual outcome, transferred byte count, and the status already sent as wire_status. Keep delivery outcome details consistent with shutdown attribution.
• Accept valid MP4s whose dimensions differ from upstream metadata. Buffered Telegram uploads use measured dimensions; streamed uploads omit automatic dimensions, and explicit caller values are preserved.
• Add persistent Dokploy settings for stop-first updates and rollbacks, readiness checks, a 60-second shutdown grace period, and failure monitoring. Add bounded Compose logs and deployment verification helpers.

Upgrade: allow 60 seconds for shutdown and run one active process per database schema. Follow the first-rollout procedure before applying the new Dokploy settings to an existing task: it retains its original shutdown limit. That procedure requires a maintenance window and briefly pauses scheduling on the single Swarm node. Host-wide Docker log rotation requires a separate Docker restart.

Validation: make check and make lint passed, including full race-enabled PostgreSQL tests, build, staticcheck, and govulncheck. All three PR CI checks passed. Deployment helper checks passed, and isolated Docker 29.7.2 Swarm tests verified readiness monitoring, graceful first rollout, lock release, recovery, and subsequent stop-first updates.

Pull request #21 · Full changes since v2.0.3

v2.0.3

Fix identity-pool failures caused by stalled PostgreSQL saves and prevent stale processes from overwriting newer state after reconnecting.

• Keep identity state in RAM and save finished operations in the background. Requests continue during stalled saves while the service retains database ownership.
• Pause new identity work when the database connection is lost. Recover the connection and verify a persisted ownership token before resuming, including when another process has already started and stopped.
• Preserve pending identity and proxy-health updates across recoverable errors. Schema and permission errors pause new work until persistence succeeds.
• Bound shutdown persistence to ten seconds after request drain, including stalled saves and connection cleanup. Skip reconnecting when no state remains unsaved.
• Register missing devices at startup without fetching signing material. Fetch signing material on demand and refresh it after one hour.

Upgrade: startup creates the scrap_ownership table in DB_SCHEMA; the database role needs permission to create it. Run one active service process per schema and allow 45 seconds for container shutdown. A hard crash can still lose changes that have not reached PostgreSQL.

Validation: make check and make lint passed, including race-enabled PostgreSQL tests, the CGO build, staticcheck, and govulncheck. All three PR CI checks passed. Regression coverage reproduces stale identity and proxy-health overwrites before the fix and verifies safe reconnect behavior afterward.

Pull request #20 · Full changes since v2.0.2

v2.0.2

Fix Telegram document delivery for WebP images from TikTok slideshows and Instagram carousels.

• Convert WebP documents to full-size JPEG before uploading, preserving image dimensions and flattening transparency onto white. Animated WebP uses its first frame.
• Wait for WebP conversion before starting a single-document upload. Other document formats retain their original bytes and streaming behavior.
• Add X-Telegram-Method to responses that contain one Telegram API call, including a single album call, while preserving Telegram's response body and status.
• Add tests for document albums, single-document uploads, original-file preservation, image dimensions, transparency, and buffer cleanup. Update the API documentation.

Validation: make check and make lint passed, including race-enabled tests, the CGO build, staticcheck, and govulncheck. No vulnerabilities found. Telegram delivery tests use local fixtures.

Full changes since v2.0.1

v2.0.1

Fix Dokploy build and startup failures with Railpack 0.15.4.

• Remove the literal ... from buildAptPackages, which caused unrelated APT dependency conflicts.
• Use structured build commands so Railpack's shell quoting preserves the linker flags, application package, and output path. Require an executable out before the build succeeds.
• Explicitly copy the Go source directories and embedded OpenAPI files. Railpack's handling of .dockerignore negations otherwise removed nested sources and failed with stat /app/cmd/tt-scrap: directory not found.
• Add a CI job using the same Railpack 0.15.4 BuildKit frontend as Dokploy, followed by a check of the image's default startup command.
• Preserve the native HEIF codecs and Docker build allowlist, and document the Railpack configuration requirements.

Validation: reproduced the missing-source failure in a complete Railpack container build, then successfully built the corrected image through the Railpack 0.15.4 frontend. The resulting image's default startup command launched the application and reached its expected missing-API-key validation in an isolated, network-disabled test. The image contains the executable and all linked native libraries, without local .env or Git metadata. Earlier validation also passed the CGO application build and media tests. Production database initialization and upstream services were not exercised by the image smoke test.

This release and tag were recreated to include all three deployment fixes and the CI check.

Full changes since v2.0.0

v2.0.0

tt-scrap now runs as one native Go service with TikTok Android app extraction, Instagram extraction, authenticated downloads, and Telegram delivery. Existing API contracts are preserved.

Changes

• Replace the Python server and app worker with one Go binary. TikTok registration, signing, identity management, and MP4 validation run in Go. Native libheif handles HEIC, HEIF, and AVIF conversion.
• Make the TikTok app method the only backend. Add optional account fallback through TIKTOK_SESSION_ID for content restrictions.
• Add PostgreSQL identity preparation and authenticated progress endpoints. Reuse saved identities, refresh signing material, and prepare ten identities per proxy before accepting requests.
• Support https://www.tt.site/t/<code>/ and https://tt.site/t/<code>/ share links, with strict host and path validation.
• Download media in parallel when the CDN provides a strong ETag. Bind every range to the same media representation and restart if it changes.
• Upload single videos, audio files, and documents to Telegram while downloading. Withhold the final bytes until media validation and the identity checkpoint succeed.
• Keep eligible media up to 250 MB in RAM by default, with a shared memory budget and automatic disk fallback. Add memory admission for image conversion and release unused buffer capacity.
• Raise general pipeline limits while keeping Instagram concurrency at four. Refresh routing, tooling, CI, and container builds.

Upgrade from v1

Read the migration guide before replacing your deployment.

• Supply DB_URL and an existing DB_SCHEMA. The database role must be able to manage the service's scrap_proxies and scrap_identities tables.
• Import proxies through PROXY_FILE, reuse saved database proxies, or explicitly enable direct connections with PROXY_INCLUDE_HOST=true.
• Remove TIKTOK_METHOD. Replace cookie-file configuration with TIKTOK_SESSION_ID if account fallback is needed.
• Stop the old process before starting the new service against the same schema. Each schema permits one active app process.
• Allow time for identity preparation before the HTTP listener opens. Review concurrency and memory settings for your host; media and image memory budgets each default to 1024 MB.
• Replace Python startup commands with the Go binary or Docker image. Native libheif and its HEIC and AVIF codec plugins are required; Python, Node, Bun, and FFmpeg are no longer runtime requirements.

The tag triggers amd64 and arm64 container publication to ghcr.io/karilaa-dev/tt-scrap:v2.0.0 and ghcr.io/karilaa-dev/tt-scrap:2.0.0.

Validation

• Race tests, PostgreSQL integration tests, formatting, vet, staticcheck, vulnerability checks, and Docker build and runtime checks passed on the implementation included in this release.
• Live tests verified TikTok share links, extraction, downloads, caching, music, native image conversion, and Telegram video, document, audio, and album delivery. Instagram image and carousel extraction and delivery also passed.
• Live Instagram reel verification was blocked by an upstream timeout. Very large uploads and memory-pressure scenarios were covered by automated tests rather than live traffic.

Full changes since v1.3.5