refactor: harden SDK against Cloudflare blocks and add offline test s…#102
Merged
Conversation
…uite
- Detect Cloudflare also on HTTP 403 (cf-mitigated / Server: cloudflare),
previously only 520 was mapped to CloudflareError.
- Fix TypeError in Flight.get_altitude / get_flight_level / get_ground_speed /
get_heading / get_vertical_speed (and Node camelCase equivalents) when the
underlying field arrived as the "N/A" sentinel.
- Add opt-in RetryPolicy with exponential backoff + jitter; passed via
FlightRadar24API(retry=...) / new FlightRadar24API({ retry: ... }).
- Make Chrome TLS impersonation parameter-driven (Python: impersonate="chromeNNN";
Node: { impersonate: { ciphers, sigalgs, ecdhCurve } }), so users can override
the bundled chrome136 profile without waiting for a release.
- Replace jsdom with node-html-parser: zero transitive deps and works on
Node 18 (cheerio 1.x pulls a recent undici that requires the File global).
- parse_airports_html / parseAirportsHtml no longer silently coerce invalid
coordinates to (0,0); emit a warning and return None/null. Missing <tbody>
also logs a warning.
- Add offline parser test suite with bundled HTML fixtures (11 tests per SDK).
CI now gates PRs on the offline suite and runs live integration tests under
retry without blocking pushes when FR24 changes layout.
- Dedupe Entity._get_info between Flight and Airport (Python); drop the unused
ABC declaration; introduce EARTH_RADIUS_KM constant in both SDKs.
- Switch default accept-language header from pt-BR,pt;q=0.9,... to en-US,en;q=0.9.
- Add Airport.from_basic_info / from_info / from_details classmethods (Python)
and Airport.fromBasicInfo / fromInfo / fromDetails static methods (Node) as
intention-revealing aliases. The existing constructor forms remain.
- Sync nodejs/FlightRadar24/index.d.ts with the post-#101 surface; remove the
phantom cookies field on __loginData; expose APIClient and RetryPolicy.
- CI: add pip-audit, npm audit and pytest --cov steps.
There was a problem hiding this comment.
Pull request overview
This PR hardens both the Python and Node.js FlightRadar24 SDKs against Cloudflare blocks and upstream HTML/layout changes by adding configurable TLS impersonation, opt-in retry with backoff, safer parsing behavior, and an offline fixture-based test suite that can gate PRs without relying on live FR24 availability.
Changes:
- Add
RetryPolicyand integrate retry/backoff into the shared HTTP client layer; improve Cloudflare-block detection (incl. 403 signals) and make TLS impersonation configurable. - Replace fragile/live-only parser validation with offline HTML fixture tests; make airport parsing stop coercing bad coordinates to
(0,0)and emit warnings instead. - Node.js: replace
jsdomwithnode-html-parser, update CI to gate on offline tests, and sync public TypeScript surface.
Reviewed changes
Copilot reviewed 27 out of 29 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| python/tests/test_snapshots.py | Mark live snapshot tests as integration for CI gating. |
| python/tests/test_api.py | Mark live API tests as integration for CI gating. |
| python/tests/test_parsers_offline.py | Add offline parser tests using bundled HTML fixtures. |
| python/tests/fixtures/airports_brazil.html | Add airports HTML fixture for offline parsing tests. |
| python/tests/fixtures/airlines.html | Add airlines HTML fixture for offline parsing tests. |
| python/FlightRadar24/request.py | Add RetryPolicy, retry wrapper, configurable impersonation, and broaden Cloudflare detection. |
| python/FlightRadar24/parsers.py | Add warnings for unexpected layouts; stop coercing invalid coordinates to (0,0). |
| python/FlightRadar24/entities/flight.py | Use shared _get_info sentinel handling; guard formatting helpers against non-numeric sentinels. |
| python/FlightRadar24/entities/entity.py | Centralize _get_info helper and introduce EARTH_RADIUS_KM constant. |
| python/FlightRadar24/entities/airport.py | Deduplicate _get_info; add intention-revealing from_* constructors. |
| python/FlightRadar24/core.py | Switch default accept-language to en-US,en;q=0.9. |
| python/FlightRadar24/api.py | Thread retry/impersonation options into APIClient; use client for standalone requests. |
| python/FlightRadar24/init.py | Export RetryPolicy from the package top-level. |
| nodejs/tests/testParsersOffline.js | Add offline parser tests using bundled HTML fixtures. |
| nodejs/tests/fixtures/airports_brazil.html | Add airports HTML fixture for offline parsing tests. |
| nodejs/tests/fixtures/airlines.html | Add airlines HTML fixture for offline parsing tests. |
| nodejs/package.json | Replace jsdom with node-html-parser; add offline/integration test scripts. |
| nodejs/FlightRadar24/request.js | Add retry policy, Cloudflare 403 detection, and configurable TLS impersonation via undici Agent. |
| nodejs/FlightRadar24/parsers.js | Switch HTML parsing to node-html-parser; add warnings and avoid (0,0) fallback coords. |
| nodejs/FlightRadar24/index.js | Re-export RetryPolicy/APIClient. |
| nodejs/FlightRadar24/index.d.ts | Update TS surface for retry/client, airport factory methods, and config typing. |
| nodejs/FlightRadar24/entities/flight.js | Guard formatting helpers against non-numeric sentinels. |
| nodejs/FlightRadar24/entities/entity.js | Introduce EARTH_RADIUS_KM constant and export DEFAULT_TEXT. |
| nodejs/FlightRadar24/entities/airport.js | Add intention-revealing from* constructors. |
| nodejs/FlightRadar24/core.js | Switch default accept-language to en-US,en;q=0.9. |
| nodejs/FlightRadar24/api.js | Thread retry/impersonation options into APIClient. |
| .github/workflows/python-package.yml | Gate PRs on offline tests + coverage; run live tests on schedule/push with retries; add auditing. |
| .github/workflows/node-package.yml | Gate PRs on offline tests; run live tests on schedule/push with retries; add auditing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The previous heuristic treated any 403 with `Server: cloudflare` as a CloudflareError. FR24 fronts every endpoint with Cloudflare, so legitimate plan-restricted 403s (e.g. a free account hitting get_history_data) were being misclassified as bot-mitigation blocks. Rely solely on the `cf-mitigated` header, which Cloudflare sets only when its own bot management actually mitigated the request.
- Node: timeouts were silently dropped from the retry path. fetch's native
AbortError was being re-wrapped into a generic Error("Request timed out…"),
losing the `name === "AbortError"` signal that runWithRetry relied on.
Introduce a TimeoutError class and check `err instanceof TimeoutError`
alongside the existing transient signals.
- Python: RetryPolicy validated max_attempts but accepted negative
base_delay / max_delay / jitter, which could feed time.sleep() a negative
value and crash with ValueError. Reject negatives in __init__.
- Node: mirror the validation in RetryPolicy for parity, even though
setTimeout clamps negatives to 0 rather than crashing.
- Deep-freeze CHROME136_PROFILE arrays so exported consumers cannot mutate them. - Replace assert in _run_with_retry with explicit RuntimeError to survive python -O. - Mark dependency audits as informational (continue-on-error) until baseline is clean.
Splits the request-layer tests into two suites — policy (retry math, Cloudflare detection rules, error taxonomy) and transport (adapter-shaped behavior tied to undici/curl_cffi) — so the transport can be swapped without rewriting the policy tests. Also makes Airport entity initialization defensive against missing position/code/country/region keys, replaces a pair of assert-based login checks with explicit None checks on the private __login_data, downgrades a swallowed decompression failure to a logged warning, and updates the Node docs to drop the spurious await on the synchronous getZones() helper.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why is this PR necessary, what does it do?
Checklist (complete all items):
References:
No references to be shared.
Notes:
No notes to be shared.