fix: source getAirports() from the airports JSON feed - #117
Merged
Conversation
FR24 rebuilt /data/airports/<country> as a client-rendered Inertia.js page. The airport rows are gone from the HTML, so `parseAirportsHtml` found no <tbody>, warned, and returned an empty list -- `getAirports()` had been answering 0 airports since some point between 2026-08-08 and 2026-08-12. The per-country pages cannot be scraped back into shape either: for large countries they only carry a `states` array linking to per-state pages, and the new payload dropped latitude and longitude entirely. Switch both ports to https://www.flightradar24.com/_json/airports.php, which returns every airport in a single response with the exact fields the Airport entity needs (name, iata, icao, lat, lon, country, alt). The country filter now runs in memory: slugifying the feed's country names matches all 228 values of the Countries enum, so the public signature is unchanged. `countries` became optional -- omitting it returns every airport, which the per-country scrape could not do. Brazil + United States now yields 2032 airports (was 0), keeping the existing >= 1800 integration threshold. One request replaces one per country, so the fan-out and its max_workers/mapConcurrent use are gone from this path. Also stop warning when a "gzip" Content-Encoding covers a body without the gzip magic number: curl_cffi already decompressed it, and this feed would otherwise log that on every Python call. Offline fixtures move from airports_brazil.html to airports.json, captured from the live feed plus a synthetic broken-coordinates row that keeps the "invalid coords must not become 0,0" regression covered.
JeanExtreme002
force-pushed
the
fix/get-airports-json-feed
branch
from
August 12, 2026 21:21
94f893b to
39ab7c0
Compare
The country field now carries FR24's own spelling, which uses parentheses
for four countries ("Myanmar (Burma)", "Cocos (Keeling) Islands",
"Falkland Islands (Malvinas)", "Timor-Leste (East Timor)"). Feeding that
straight back into getCountryFlag() -- the obvious way to chain the two
calls -- built `myanmar-(burma)` and returned null. Both ports now slugify
the flag URL with countryToSlug, so either spelling resolves.
get_airports() in the Python port called get_json_content(), which raises
when FR24 answers 200 with a non-JSON body. That made the parser's
"warn and return []" guard unreachable exactly when it was needed, and
diverged from Node. It now passes get_content() through instead.
Altitude disagreed between the ports: reusing the coordinate helper turned
2436 into 2436.0 in Python. The helper is now _to_number/toNumber and
preserves whole numbers. Node also switches parseFloat -> Number, so a
value like "43.30 N" is rejected in both ports rather than silently read
as 43.3 in one of them -- the feed already sends strings in that position
("alt": "-1" on 11 rows).
Remaining cleanups: brotli responses no longer log the "already
decompressed" warning that only gzip was exempt from; getAirports([])
returns without a request; Python accepts plain slug strings alongside
Countries members, matching Node; and the stale requestStandalone comment
about a per-country fan-out is gone.
Tests: the flag chaining is covered end to end in both ports (verified to
fail before the fix), plus offline coverage for the parenthesised slugs,
int-preserving altitude, strict numeric rejection, and the unfiltered path.
The previous commit swapped parseFloat for Number to reject "43.30 N", but
Number("") is 0, and the empty-string guard ran against the raw value --
so a whitespace-only or array coordinate became 0 in Node while Python
returned None. That is the Gulf of Guinea bug the helper exists to prevent,
reintroduced in the port it was meant to align.
Both ports now gate on the same decimal pattern before converting, so they
accept and reject exactly the same inputs: " ", "abc", "0x10", "1_000",
"inf", [] and {} are all None/null on both sides, while "1e3", "+5", ".5"
and "-23.4" parse on both. Bare Number would have taken the hex, and bare
float would have taken "1_000" and "inf".
Also from the review:
- getAirports accepts a single country as well as a list, in both ports. It
used to raise "countries.map is not a function" in Node and silently
return [] in Python.
- index.d.ts declared latitude and longitude as `number` while the parser
can return null for either -- altitude on the next line was already
honest about it.
- The two integration tests added for the flag chaining collapse into one,
halving the 860 KB feed downloads per run, and the tricky country is now
discovered from the response instead of hard-coding MYANMAR_BURMA, so the
guard survives FR24 renaming it.
- The offline fixture gains a real "Myanmar (Burma)" row carrying `alt` as
a string, which is what the feed does on 11 rows, so the parenthesised
spelling and string altitude are covered without network access.
Two holes left by the previous commit, both found by review: The rewrite of _to_number moved the math.isfinite() check onto the pre-parsed float branch only, so the string path let "1e999" through as inf -- a latitude that poisons distance math and makes json.dumps() emit invalid JSON. Node already returned null for it. The string path is finiteness-checked again. Both ports also disagreed in the other direction: str([43]) is "[43]" in Python but String([43]) is "43" in JavaScript, so a one-element array read as 43 in Node. Neither port stringifies non-strings now. parseAirportsJson only unwrapped strings and Buffers, but request() hands back an ArrayBuffer whenever the response carries an unexpected content-type -- so a valid JSON feed served as, say, octet-stream fell through to "no rows array" and returned [], the exact silent-empty result this PR set out to remove. ArrayBuffers and typed arrays are now decoded like Buffers. Python needed no equivalent: its client returns bytes, which json.loads already accepts.
Three review rounds in a row found a bug in this one helper, and two of them were caused by the previous round's fix. The cause is structural: the two ports are kept in agreement by hand, so a case fixed in one language stays broken in the other until someone thinks to check. testdata/numeric-coercion.json now holds the cases, and both offline suites iterate it. Adding a case there is enforced in both languages at once -- verified by flipping one expectation and watching Node and Python fail together. Neither port ships its tests (nodejs/.npmignore, the pyproject build exclude), so a path above the package roots costs nothing. The table immediately caught what round four was about: Python read "1" * 400 as a 400-digit int where Node returned null, because int() ran before any magnitude check and Python ints have no ceiling to overflow. _to_number now parses as a float first, rejects non-finite results, and only then recovers int-ness -- capped at Number.MAX_SAFE_INTEGER, past which JavaScript stops being exact and the two would drift apart regardless. Also fills the gaps around the API surface this PR widened: tsd covers the no-argument and single-country forms of getAirports, and both READMEs show that omitting the countries returns every airport.
The shared testdata/ table was more machinery than this earns: a new
top-level directory, 28 cases, and workflow path filters to keep in sync,
all for one coercion helper. The table is back inside each port's test file,
trimmed to the nine inputs that either caught a real bug or pin a stated
invariant, with a pointer to its twin so the two stay in step.
Two real divergences found while doing it, both in the Python port:
- The MAX_EXACT_INTEGER clamp only covered strings, so a native JSON
integer above 2**53 returned exactly while Node's JSON.parse had already
rounded it: {"lat": 9007199254740993} read as ...993 in Python and ...992
in Node. The int branch clamps too now.
- row.get("name", "") only defaults on a missing key, not on JSON null,
where Node's `?? ""` defaults on both. A null country therefore arrived
as None and would have made the new live test's re.search() raise
TypeError rather than fail an assertion. Now `or ""`.
The clamp added in the previous commit called float() on any int above 2**53, and float() raises OverflowError -- not inf -- once the int exceeds the double range. A feed row carrying 400 digits as a raw JSON number therefore crashed get_airports() outright, where Node returns null. This is the fourth bug in a row in this one helper, each planted by the fix before it, so the shape changed rather than another branch being patched. Every value now funnels through a single float() conversion with one OverflowError guard, instead of three type branches each needing their own. Int-ness is recovered afterwards, and only when the round trip is lossless (`exact == number`), which is also what keeps -23.4 from truncating to -23. The table's "integer past 2**53" case was vacuous in Node -- a JS number literal is already rounded before it can reach the payload -- so it is replaced by a test that writes the numbers straight into the JSON text, the only way to hand both ports a value their parsers must survive rather than one already flattened for them.
Fuzzing the text path -- the half the numeric corpus never touched --
showed the two ports disagreeing on nearly every non-string value the feed
could put in name, iata, icao or country. `?? ""` only defaults on null and
undefined, so Node stored 0, false, [] and {} verbatim; `or ""` defaults on
every falsy value, so Python stored True and 123 but blanked the rest.
Both ports now accept a string and blank anything else, which is what
Airport documents and what callers assume: getCountryFlag(airport.country)
would otherwise slugify an object into "object-object" and request that
flag over the network.
The feed has never sent a non-string here, so this is about the contract
rather than observed breakage -- but it is four lines, and it closes the
bogus-request path.
The filter compares a slug derived from the feed's display name ("United
States") against the enum's URL path slugs ("united-states"). Those are two
separate FR24 vocabularies that happen to agree today -- verified 228/228 in
both directions -- but nothing detected a divergence. A rename on either side
would silently return [] for that country behind a warning a caller may
never see, and display names move more freely than URL paths.
Both ports now assert the agreement inside the test that already downloads
the whole feed, so it costs no extra request. Verified by adding a bogus
enum entry: the test fails and names the country.
Five findings from an independent review, all verified against both ports before changing anything: - A row with one unusable coordinate kept the other, so an airport could carry a latitude and a null longitude. Anything gating on `if airport.latitude` read that as located, and the old HTML parser had nulled both. One bad coordinate now drops the whole position, which is also what the log line always claimed to do. - The "both ports accept the same strings" invariant did not hold for non-ASCII input: Python's \d matched Arabic-Indic digits, str.strip() drops U+001C-U+001F where String.trim() keeps them, and trim() drops U+FEFF where strip() keeps it. The pattern is ASCII-only now (re.ASCII on the Python side) and both ports trim one explicit space set. My earlier fuzzing missed this because the corpus was ASCII. - country_to_slug(0) was "" in Python and "0" in Node -- `or ""` versus `?? ""` again, this time in the slugifier. - Coordinate warnings were logged per row. The feed carries every airport, so a degraded response meant hundreds of lines per call; they are now one summary line with a sample. - Demoting the decompression warning by Content-Encoding also silenced genuinely undecodable brotli, leaving the parser to misreport it as invalid JSON. For text and JSON payloads the body is now checked instead of the header, so a broken one still warns; binary bodies keep the old behaviour, having no such tell. Not changed: the review's example for the country vocabulary finding does not reproduce -- the enum says "czechia" and the feed says "Czechia", which slugify alike. The reviewer could not reach the feed (403) and assumed "Czech Republic". The underlying concern was real and is already pinned by the enum-versus-feed test.
Added comment lines drop from 101 to 41. Multi-line rationale blocks become one line where a maintainer could otherwise "simplify" the code back into a bug, and disappear where the code, a docstring or a type already said it. No behaviour change: both suites and all linters pass unchanged.
Arrays are objects in JavaScript, so `typeof row !== "object"` let a row like [1, 2, 3] through and built an airport with every field blank, while Python's isinstance(row, dict) dropped it. Verified: the same payload yielded 2 airports in Node and 1 in Python. Also corrects the api.py comment about get_content(): it is a body served as text/html that reaches the parser's guard. A malformed body under a JSON Content-Type still raises from json.loads, and the previous wording read as though it did not. Not changed: the review also reported that the >= 1800 threshold for Brazil + United States cannot be met because the feed carries ~1606 for those two. The live feed has 6745 rows, 282 for Brazil and 1750 for the United States, so the threshold passes with room to spare -- as the green integration runs across seven CI jobs show. The figure came from an archived 5672-row snapshot, 1073 rows short of production.
Three findings from the latest review, all reproduced first:
- getAirports(new Set([...])) crashed in Node with "countries.map is not a
function", and only after downloading the whole feed, while Python happily
iterated the same argument and returned 282 airports. Node now takes any
iterable, as the annotation implies.
- country_to_slug ran on every row even with no filter, where the slug is
unused: an NFKD normalize plus three regex passes over 6745 rows for
nothing. It is computed inside the filter branch now.
- The enum-versus-feed test asserted equality in both directions, so a
country FR24 *adds* would fail CI even though the library handles it
fine. Only the direction that breaks users is asserted now: an enum value
the feed no longer names means that filter silently returns []. The Node
describe also fetched the feed three times; one before() hook covers all
of them.
Not changed: the review's medium finding does not reproduce. It reports
country_to_slug("Virgin Islands (U.S.)") slugifying to virgin-islands-u-s
against an enum value of virgin-islands-us -- but the feed spells these
"Virgin Islands Us" and "Virgin Islands British", and none of its 228 country
names contains a dot or an apostrophe, so every slug matches. Stripping dots
would be guessing at a spelling FR24 does not use; the enum-versus-feed test
is what catches it if that ever changes.
…e slugifier Three findings from the latest review, each reproduced first: - get_airports called len() on the caller's argument, so a generator raised "object of type 'generator' has no len()" -- the previous commit's claim of accepting any iterable was false for the port it was written about. The argument is materialised with list() now. - Array.from turns a non-iterable into [], so getAirports(Countries) (the enum object rather than its values) or getAirports(123) answered a mistake with an empty list and no request, where Python raises TypeError. Node rejects non-iterables with a TypeError naming what it expects. - country_to_slug str()ed its input, so a Countries member became "countries-brazil" and get_country_flag(Countries.BRAZIL) returned None as though the flag were missing. It unwraps `.value` now, which get_airports already did -- and which Node got for free, its enum values being strings. Left as is: the review suggests relaxing the enum-versus-feed assertion in case FR24 delists the last airport of a marginal country. That failure is worth seeing: the enum would then advertise a country whose filter returns nothing, and the message names it, so triage is quick either way.
Auditing my own review fixes turned up one: getattr(country, "value", country) unwrapped anything with a `.value` attribute, so an unrelated object slugified to whatever that attribute held. isinstance(country, Enum) is what was meant.
The step named "Offline tests (PR gate)" is the intended gate, but the live integration step was only soft-failing on push, so anything FR24 changed on its side blocked every unrelated PR. That is how #115, a one-line js-yaml lockfile bump, opened red, and how the broken getAirports() stayed invisible on main for four days: soft on push hid it there while it blocked elsewhere. Both workflows now soft-fail on push and pull_request and stay hard on the weekly cron and on manual runs, which are the runs whose job is to notice upstream drift. Also aligns the Node bounds assertion with the Python port, which has always been inclusive: FR24 returns flights sitting exactly on the boundary of the requested box, and `to.be.below(zone.tl_y)` failed them with "expected -52 to be below -52". That was an intermittent red with no bug behind it.
…ow part) Reverts the continue-on-error widening in both package workflows at the maintainer's request; live integration keeps blocking pull requests as before. The bounds assertion fix from the same commit stays.
getAirports()/get_airports() answered with an empty list until now, so this ships as a fix release. Note for TypeScript consumers: Airport.latitude and Airport.longitude are declared 'number | null' from this version on, matching what the parser has always been able to return. Code reading either field under strictNullChecks needs a null check.
JeanExtreme002
force-pushed
the
fix/get-airports-json-feed
branch
from
August 13, 2026 02:45
0de4c8d to
053c9e8
Compare
Four findings from the latest review, verified first:
- The decode-failure block assumed bytes, so a non-bytes body raised
AttributeError from inside the except clause and replaced the original
error with its own. Reproduced with a str body; nothing in that block can
raise now.
- getCountryFlag(None) built .../flags-small/.svg and spent a real round trip
(568 ms measured) to return None as though the flag were missing. It used
to raise before country_to_slug started accepting anything. An empty slug
returns None without a request in both ports.
- The _to_number docstring claimed whole numbers keep their type in both
ports, which holds for "2436" but not for "1e3" -- a float here, an
integral Number in Node. The docstring says so now; the feed never sends
exponent notation in `alt`, and forcing an int there would change latitude
too.
- The live flag test picked the first punctuated country in feed order, so
the assertion depended on how FR24 happened to sort its rows. Sorted now.
Left alone: get_airports("") returns [] but logs a warning naming an empty
country. It is cosmetic, and the message already points at the empty slug.
…ports type Four of the seven findings from the latest review; the other three are argued against below. - country_to_slug collapsed every run of non-alphanumerics into a hyphen, while the Countries values were evidently built by deleting punctuation inside words: "Virgin Islands (U.S.)" would slugify to virgin-islands-u-s against an enum saying virgin-islands-us. The feed spells it "Virgin Islands Us" today, so nothing is broken -- and measuring first showed the stricter rule leaves all 228 current mappings byte-identical, which makes it free insurance against a rename rather than a guess. - index.d.ts declared getAirports(countries?: string[] | string) while the runtime and the JSDoc take any iterable, so new Set(["Brazil"]) worked but failed to compile. Iterable<string> now, with a tsd case pinning it. - get_airports mapped its argument through str(getattr(country, "value", country)), re-adding the duck-typed unwrap that a357b0f had just narrowed to isinstance(Enum) and turning None into the slug "none". The parser already slugifies Countries members, so the whole mapping is gone. - The get_content() comment now says a malformed JSON body still raises. Four reviews read the old wording as a blanket promise; the wording was the problem. Left alone: raising instead of warning for a country that matches nothing would fail the day FR24 delists a country's last airport, and the enum-versus-feed test already covers the systematic case. bytearray/memoryview payloads are unreachable from get_airports, since the client hands back bytes. And airports_data_url is one of seven unreferenced entries in the Core URL catalogue -- five base URLs and zones_data_url predate this branch -- so removing only this one would be inconsistent with the file.
Probing the CDN shows FR24 does not follow one rule for punctuation in its
asset names:
flags-small/cote-d-ivoire.svg 200 (apostrophe hyphenated)
flags-small/cote-divoire.svg 404
flags-small/virgin-islands-us.svg 200 (dots absent)
flags-small/virgin-islands-u-s.svg 404
Deleting punctuation before hyphenating therefore fixed the dotted spelling --
which the feed does not use, since it says "Virgin Islands Us" -- at the cost of
getCountryFlag("Côte d'Ivoire"), which worked before and returned null after.
Trading a live case for a hypothetical one is the wrong side of that trade, and
"virgin-islands-us" exists because that is the display name, not because a rule
deleted the dots.
The other three fixes from that commit stand: the widened getAirports type, the
removed duck-typed unwrap, and the get_content() comment.
The docstring called exponent notation the only exception, but a decimal point does the same: "2436.0" is a float here and an integral Number in Node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
getAirports()was returning an empty list in both ports. FR24 rebuilt/data/airports/<country>as a client-rendered Inertia.js page (<div id="app" data-page="…">, Vue chunkAirportsByCountry-*.js), so there is no<tbody>left to scrape:parseAirportsHtmlhit its guard, loggedno <tbody> for …, and returned[].Two integration tests failed on every PR touching
nodejs/**:tests/testApi.js—Expected at least 1800 airports.→expected +0 to be above 1799tests/testSnapshots.js—Airport list items match expected shape.→expected +0 to be above +0The breakage window is 2026-08-08 → 2026-08-12: the scheduled run on 08-08 (
31230849810) passed all three jobs, and onschedulethe integration step is blocking. Onmainit stayed invisible because that step iscontinue-on-errorforpushevents.Patching the parser was not an option:
statesarray linking to per-state pages (Brazil 27, United States 51, so 78 requests for the two test countries).lat/lonentirely; its fields are justid, name, city, iata, icao, total.Fix
Both ports now read
https://www.flightradar24.com/_json/airports.php, which returns every airport in one response with exactly the fields theAirportentity needs:{"name":"A Coruna Airport","iata":"LCG","icao":"LECO","lat":43.302059,"lon":-8.37725,"country":"Spain","alt":326}"United States"→united-states) matches all 228 values of theCountriesenum in both directions.>= 1800threshold passes untouched. One request replaces one per country.rows, invalid JSON, or a country with no matches warns and returns[]rather than failing silently. Unusable coordinates becomenull, never0,0(the Gulf of Guinea regression stays covered).Public surface that also changed
Beyond
getAirports(), four things are observable from outside and worth knowing before merge:getCountryFlag()slugifies differently.Airport.countrynow carries FR24's own spelling, which uses parentheses for four countries (Myanmar (Burma),Cocos (Keeling) Islands,Falkland Islands (Malvinas),Timor-Leste (East Timor)). ChaininggetCountryFlag(airport.country)builtmyanmar-(burma)and returnednull, so the flag URL is now built with the same slugifier as the feed. The change is strictly more permissive: all 228 enum values are idempotent under it, no input that previously produced a working URL regresses, and even a hardcodedmyanmar-(burma)now resolves.getAirports()accepts a single country or any iterable of them, in both ports — a string, aCountriesmember, a list, a set, a generator. It used to raisecountries.map is not a functionin Node and silently return[]in Python. A non-iterable is rejected with aTypeErrorrather than answered with an empty list, and omitting the argument returns every airport, which the per-country scrape could not do.countryToSlug/country_to_slugtakes aCountriesmember too, sogetCountryFlag(Countries.BRAZIL)works. In Python it previously slugified tocountries-braziland returnedNoneas though the flag were missing — a trap opened bygetAirports()starting to accept enum members.request.pyno longer warns when agzip/brContent-Encodingcovers an already-decompressed body. curl_cffi decompresses transparently while leaving the header in place, and this feed would otherwise log that on every call. For text and JSON payloads the decision is taken from the body rather than the header, so a genuinely undecodable one still warns instead of surfacing later as invalid JSON.Both READMEs now show that omitting the countries returns every airport; the Node one no longer claims the method "requires country selection".
Three parser rules are worth stating because they differ from the old scrape or from one port to the other:
index.d.tsdeclaredAirport.latitudeandAirport.longitudeasnumber, while the parser can returnnullfor either (altitudeon the next line was already honest about it). They are nownumber | null.This is a compile-time break for TypeScript consumers under
strictNullChecks: code reading those fields needs a null check. Runtime behaviour is unchanged — the declaration was simply wrong.Verification
Both ports return identical results — Brazil 282, United States 1750, both 2032, unfiltered 6745, and the same
GRUatlat -23.429991 / lon -46.4674 / alt 2436.A live test also pins every
Countriesvalue to a country the feed still names — the enum holds FR24's URL slugs while the feed carries display names, two vocabularies that agree today but would silently empty a country's filter if either were renamed. The reverse direction is deliberately not asserted: a country FR24 adds is a gap in the enum, not a regression.The coercion layer was additionally fuzzed with 1651 JSON literals (normal numbers, exponents to ±400, 420 raw digits, dirty numeric strings, non-numeric types) fed to both ports and compared value by value: 0 exceptions, 0 non-finite results, 0 divergences, and invalid JSON payloads degrade identically. New offline coverage pins the cases that caught real bugs: whitespace and arrays as coordinates, hemisphere suffixes, exponents and plain digits no double can hold, raw-byte response bodies, text fields carrying non-strings, and the parenthesised country spellings.
Out of scope, worth flagging
tests/testApi.js"Getting Flights by Bounds" is flaky, independently of this PR: it assertsto.be.below(zone.tl_y)and FR24 occasionally returns a flight exactly on the boundary (expected -52 to be below -52).at.most/at.leastwould fix it.continue-on-error: ${{ github.event_name == 'push' }}innode-package.ymllets live-site breakage block unrelated PRs while hiding it onmain— which is why this bug went unnoticed for four days and why chore: bump js-yaml from 4.1.1 to 4.3.1 in /nodejs in the npm_and_yarn group across 1 directory #115 opened red.Core.airportsDataUrl/Core.airports_data_urlare left in the URL catalogue although nothing uses them internally now.🤖 Generated with Claude Code