Skip to content

goopdl 2.0.0

Choose a tag to compare

@github-actions github-actions released this 05 Nov 20:49
· 16 commits to main since this release
v2.0.0
bc6fe86

goopdl is now usable as a library from other Python code, including backends that run several accounts and cache directories in one process. This is a major release: some things behave differently for code that calls goopdl.api directly, and the account commands of the CLI moved under goopdl auth. Downloads, info, search, latest and the rest of the CLI keep their commands and options.

from goopdl import Client, Config

client = Client(Config(config_dir="/var/lib/svc/goopdl/tenant-a", country="US"))
details = client.details("com.whatsapp")
result = await client.adownload("com.whatsapp", "/data/apks")

Guide with the whole API, the error hierarchy, async use and a migration table: https://villoh.github.io/goopdl/docs/library/

Upgrading

pip install -U goopdl (Python 3.10 or newer; typer must be 0.16 or newer). keyring is now a required dependency, see Signing in.

Commands that moved

Before Now
goopdl auth [--arch ...] goopdl auth token [--arch ...]
goopdl auth --clear goopdl auth clear-cache (goopdl auth logout does the same and also signs out a saved account)
goopdl aastoken --browser goopdl auth login
goopdl aastoken --oauth goopdl auth login --manual
goopdl aastoken (printed AASToken: ...) goopdl auth login --print
goopdl aastoken EMAIL PASSWORD removed: Google rejects passwords and app passwords
goopdl auth status (new)

goopdl aastoken still exists as a hidden command that warns and runs auth login --print; it accepts only --browser and --oauth. goopdl auth alone now shows its help.

Library behaviour

Existing function signatures are kept and new parameters are optional. These behaviours changed:

What Before Now
Corrupt protobuf in a Play response raw ValueError / EOFError MalformedResponseError, a PlayAPIError
Package names sent to Google as given InvalidPackageError (a ValueError) if there is no dot or a bad character
Unknown arch used in cache file names InvalidArchError (a ValueError)
Transport failures in goopdl.api raw httpx exceptions NetworkError, which is still an httpx.TransportError
CDN and network failures while downloading IntegrityError DownloadError, a subclass of IntegrityError (status_code, retryable, retry_after)
purchase on HTTP 429 returned "" raises RateLimitedError
download_app with the token pool busy PlayAPIError PoolBusyError (a TimeoutError)
Cache directory always ~/.config/goopdl $GOOPDL_CONFIG_DIR, then $XDG_CONFIG_HOME/goopdl, then ~/.config/goopdl; with XDG_CONFIG_HOME set, tokens are fetched again once (export GOOPDL_CONFIG_DIR=~/.config/goopdl keeps the old ones)
Dispenser HTTP 429 with no usable token exit code 1 or 3 exit code 5, with retry_after
CDN redirects followed anywhere at most 5, HTTPS on 443, Google hosts only
goopdl.auth.console, goopdl.download.Progress and the other Rich names Rich objects deprecated (DeprecationWarning), they still work; token functions log instead of printing
Account mode and files wrote nothing writes a device registration file; goopdl auth login also writes account.json
goopdl.aastoken.fetch_aas_token took a password or a one-time oauth_token takes only an oauth2_4/... token (ValueError otherwise); encrypt_password is gone

Signing in with a Google account

goopdl auth login opens a Chromium-based browser (Chrome, Edge, Chromium or Brave; set GOOPDL_BROWSER if goopdl can't find it) with a temporary profile, captures the one-time oauth_token after you sign in, exchanges it for an AAS token and saves the account. From then on every command uses that account.

  • The token goes to the system keyring (Secret Service, Keychain or Credential Manager) under the service goopdl. login reads it back to check the keyring really stored it. If there is no keyring (servers, containers, SSH) or it returns nothing, the token is saved in account.json instead and goopdl says so. --insecure-storage skips the keyring.
  • account.json is 0600 in the 0700 config directory and holds only the email and where the token lives. On Linux and macOS goopdl refuses it if it is a symlink, not yours, or readable by others.
  • goopdl auth status shows the mode (environment, saved account, anonymous), the account and where the token is. It never prints a token. goopdl auth logout removes the keyring entry, the file and every cached token; goopdl auth clear-cache removes the cached tokens and device registrations and keeps the account.
  • GOOPDL_ACCOUNT_EMAIL and GOOPDL_AAS_TOKEN still work and win over a saved account.
  • login --print prints only the AAS token, on one line, and saves nothing: TOKEN=$(goopdl auth login --print). login --manual opens your usual browser and asks you to paste the oauth_token cookie, for when the browser capture doesn't work.
  • From Python, a Client never reads account.json, the keyring or the environment variables: without credentials it is anonymous. import goopdl does not load keyring.

The system keyring protects the token against backups and a stolen disk, not against other processes of your user on Linux and Windows. An AAS token gives full Play Store access to the account: use a throwaway one.

Added

  • Client, Config, Credentials and DownloadResult. A Client has its own cache directory, account, country, proxy, architecture and profile; its methods are details, search, list_splits, item, delivery, download, the async adownload, and clear_cache.
  • download_files(): async downloads with on_progress, max_concurrent and an injectable httpx.AsyncClient. If one download fails or the call is cancelled, the others are cancelled and their .part files removed.
  • GoopdlError as the root of every goopdl exception, plus NetworkError, MalformedResponseError, InvalidPackageError, InvalidArchError, PoolBusyError, DispenserRateLimitedError, DispenserUnavailableError and DownloadError. py.typed and __all__.
  • raise_on_error on the token functions, get_config_dir(), validate_package().
  • The library guide in English and Spanish, and syntax highlighting for code blocks on the docs site.

Account mode

With Google account credentials, every call used to sign in from scratch and register a new device on the account. It now reuses one:

  • The access token is kept in memory per process, with one sign-in shared by concurrent callers, and renewed when it expires or on the first 401.
  • The device registration (gsfId and the registration tokens) is stored under the cache directory as direct-device-<hash>-…json, mode 0600, so several processes register once. The AAS token, the access token, the toc cookie and the email are never written to it.
  • With a mocked Google Play: 8 cold threads go from 40 requests and 8 devices to 12 and 1; 50 calls in a row from 250 requests to 54; 3 cold processes make one check-in instead of 3.
  • Against Google with a throwaway account: 3 Client.details calls in a row made one check-in, one device upload, one sign-in and one toc; 8 cold threads made one check-in; a new process made none and reused the same device; Client.download produced a valid APK with the same SHA-256 as the anonymous one; the CLI run twice used the same device; clear_cache() removed the registration and the next call registered a new one.

Performance

  • Connection reuse. Synchronous requests to Google Play and to the token dispenser now reuse keep-alive connections, one HTTP client per proxy, instead of opening a new TCP and TLS connection for each call. Against Google Play, the median get_details call went from 150 to 156 ms to 89 to 92 ms once the connection is open (about 40 % less; the first call, which opens it, costs the same). Cookies stay disabled on these clients, and a stale connection is retried once only when that is safe.
  • Waiting for the token pool lock. A busy lock is polled with an exponential backoff of 1 to 50 ms and jitter instead of a fixed interval. With a warm pool and a mocked Google (no network latency), 8 threads went from 153 to 8 ms and 64 threads from 355 to 96 ms, together with the change above.

Rate limits and retries

  • After a 429 from the token dispenser, goopdl sends it nothing more until the window ends (the server's Retry-After, capped at 15 minutes, or 60 s when it does not send one). With the pool partly filled and the dispenser limiting, 50 calls used to send 50 requests; now they send 1. It is in memory, per process.
  • Transient dispenser failures (503, connection errors) get one bounded retry with a cooldown, and a persistent failure stops after two attempts; before, a 503 made fetch_token try every one of the 19 profiles. A 429 stops at the first one.
  • A delivery Retry-After above 30 s is kept on the exception (retry_after) and fails at once; before, it was dropped and the call retried after 1 s and 3 s.

Fixed

  • Two downloads of the same file at the same time could publish a truncated APK that still passed the integrity check; each download now writes to its own temporary file.
  • Truncated gzip streams and OBB or asset packs were published as complete; sizes are now enforced as the bytes arrive.
  • search could return a layout block (the Play Pass banner) as an app named fbtag;playpass....
  • goopdl aastoken wrapped the token across four lines of 80 columns when the output was not a terminal, so capturing it in a shell variable did not work.
  • The CLI crashed on a console that cannot print → or ↑ (cp1252, redirected Windows output), and on remote app titles in other scripts.
  • Failures could leave tokens, signed URLs or credentials in an exception message or in a traceback frame; the public entry points no longer do. The auth dict and the proxy you pass in yourself stay in your own frames, so turn off local-variable capture in your error reporter or scrub them.

Security

  • package, arch and split names are validated, query parameters are encoded, and output paths cannot leave the output directory.
  • CDN redirects are checked at every hop; credentials are not forwarded to another host.
  • Token cache files are written atomically with mode 0600 in a 0700 directory, and the pool lock file does not follow symlinks.
  • The saved account's token lives in the system keyring when there is one; errors from the keyring are reported by exception class only.
  • AAS and OAuth error bodies from Google are no longer echoed.
  • The publish workflow has least-privilege permissions per job and its actions are pinned by commit.

Binaries

The release binaries are built with the same script that CI runs on Linux, macOS and Windows for every change, and each one is started with a smoke test that checks keyring and its backends are bundled. They include keyring: the Linux binaries grow by about 6 MB (25.5 to 31.8 MB for x86_64) and the macOS and Windows ones by about 1.3 MB (for example 13.6 to 14.9 MB for Windows x86_64). On Linux keyring needs SecretStorage, and that brings cryptography.

Known limits

  • Anonymous mode was run end to end against Google Play (search, info, list-splits, inspect-delivery, download, fast-download, Client.download, Client.adownload, api.download_app): the base APK has the same SHA-256 on every path. Account mode was run against Google on Linux with a throwaway account (see above). What the token pool and the dispenser backoff do under a real 429, and the re-authentication after a real 401, were verified with mocks only.
  • The keyring was tried by hand with GNOME Keyring (Secret Service) on Linux only. The binary smoke test uses a keyring backend that stores nothing, so on macOS and Windows it proves the binary starts and bundles keyring, not that the Keychain or the Credential Manager work from it. goopdl does not check the permissions of account.json on Windows.
  • Not run against Google on macOS or Windows; Windows is covered by the test suite in CI.
  • While auth login waits for you, the browser's debugging port listens on 127.0.0.1 for any local process. This was already so with aastoken --browser.
  • How long a Google access token lives is not documented anywhere; the cache does not depend on it (it renews on the first 401).
  • Google can invalidate a registered device without a 401. Client.clear_cache() or goopdl auth clear-cache registers a new one.
  • The anonymous token dispenser allows only a limited number of requests per hour.