Skip to content

XEarthLayer v0.4.7

Choose a tag to compare

View all tags
@github-actions github-actions released this 31 Aug 01:59
· 10 commits to main since this release
Immutable release. Only release title and notes can be modified.
v0.4.7
f81cc9b

XEarthLayer 0.4.7 is the memory release.

Available as a Linux tarball, .deb, .rpm, and an AUR package.

If you are upgrading from 0.4.6, read the two upgrade sections below before your first flight. Both describe expected behaviour that looks like a regression if you are not expecting it.

What this release is about

One symptom, three causes. XEarthLayer's memory grew steadily on long flights until the system killed it. A twelve-hour flight ended in an out-of-memory kill at 64 GB against a configured 4 GB cache, with no application error and nothing in the log to separate the candidate causes.

Nothing was leaking. Bytes actually in use stayed flat at 572 to 725 MB across an eighteen-fold increase in work, so everything was being freed. What grew was the C allocator's retention of freed memory, which is why nothing in the code ever looked wrong.

glibc gives threads separate heap arenas to avoid lock contention, and returns memory to the operating system only by trimming the top of one. Space stranded below a still-live allocation stays committed for the life of the process, so an arena grows to cover the worst burst it ever saw and never gives that space back. The total is the sum of every arena's personal high-water mark. With a ceiling of eight arenas per core, each new burst could recruit fresh arenas that had never seen a peak.

Rare traffic peaks were being converted into permanent floor, one arena at a time.

Three mechanisms, fixed together:

  • A generated texture is 11.17 MiB, and glibc's threshold for serving an allocation that size by mmap, which does return memory on free, adapts upward past that size after a single tile. The threshold is now pinned, at a measured 0.16 ms per tile.
  • The thread pool carrying encoding and disk I/O accepted the runtime's 512-thread default and reached 581 threads, each excursion ratcheting the high-water mark. It is now sized from the work that actually queues on it.
  • The arena count itself is now capped, so once each arena has seen its worst burst there are no fresh ones left to recruit and the total converges.

Measured across matched eleven-hour flights, before and after:

before after
Committed memory 11,580 MB 9,687 MB
Allocator arenas 6,091 MB 4,038 MB
Swap 1,063 MB 533 MB
Growth steps after initial sizing 18 1
Throughput 3.06 tex/sec 3.03 tex/sec

Both allocator settings defer to an explicit MALLOC_MMAP_THRESHOLD_, MALLOC_ARENA_MAX, or GLIBC_TUNABLES entry, so if you tune the allocator yourself you keep control.

These fixes are specific to Linux's GNU C library. macOS uses a different allocator and is unaffected, and for now unprotected.

Upgrading from 0.4.6: expect one slower flight

This release corrected how mipmaps are written into generated textures, which changed their size. Every texture cached by 0.4.6 is therefore the wrong size for this build.

Previously those were detected at the last moment and replaced with a magenta placeholder. Because nothing deleted them, they were served as magenta again on every subsequent request, permanently, with no way back short of clearing the cache by hand.

They are now replaced properly the first time they are read. The cost is that your first flight after upgrading regenerates a large part of the texture cache: more downloading, more encoding, and more placeholder tiles than usual while it catches up. It settles after that.

The placeholder timeout is now the documented 10 seconds

When a texture cannot be produced in time, XEarthLayer serves a magenta placeholder rather than making the simulator wait. That limit is generation.timeout, documented and reported as 10 seconds, but the setting never reached the code, which used a hardcoded 30.

It is now honoured. This is the value the documentation has always claimed, but it is a real change: the window before a placeholder appears is three times shorter than in 0.4.6. Combined with the cache refill above, the first flight after upgrading is where you are most likely to notice it. If you would rather have the old behaviour while the cache warms, set timeout = 30 under [generation] and put it back afterwards.

A corrupt cache file could stop XEarthLayer starting

Reported during the beta by a tester who could not start at all:

memory allocation of 3418906639127570548 bytes failed
Aborted (core dumped)

That number is not a quantity of memory. In hexadecimal it is 0x2f726579616c6874, which read as text is thlayer/, eight characters of the path ~/.xearthlayer/. A damaged byte in the index cache had shifted every later read along by one position, so part of a file path was read as a field saying how many bytes to allocate.

The code already intended to handle this: read the cache, and if the read fails, throw it away and rebuild. That was correct, and it had never once run. A failed allocation of that size terminates the process rather than reporting an error the surrounding code can catch.

Every cache now bounds what it reads by the size of the file it is reading, writes durably so an interrupted write cannot leave a half-finished file behind, and discards anything it cannot trust. A cache that cannot be read is regenerated, which is what a cache is for.

Settings that never did anything have been removed

Four configuration keys were reported by config list but never reached the code they named:

  • executor.network_concurrent, executor.cpu_concurrent and executor.disk_io_concurrent set resource pool sizes. Those sizes are derived from your processor and are now correctly not user-configurable. config list had compounded the problem by printing values from a second formula that disagreed with the one actually in use.
  • cache.disk_io_profile selected an HDD, SSD or NVMe profile whose result nothing consumed. Storage detection is unaffected and still appears in xearthlayer diagnostics.

Your configuration will still load if it contains these keys, and xearthlayer config upgrade removes them. Nothing about how XEarthLayer runs changes, because the values were never used.

Also in this release

  • Textures are served without copying the whole tile. X-Plane reads about 4% of each generated texture. Delivering it used to copy all 11.17 MiB, several times over. The read path now borrows and slices.
  • Terrain banding at distance is fixed. Generated textures were missing most of their mipmap chain.
  • Regions are no longer retired as having no scenery when nothing was actually consulted, and slow regions are no longer retired permanently.
  • xearthlayer diagnostics completes on large caches instead of hanging forever, which had blocked filing bug reports at all.
  • Arch and CachyOS packages build again, and the shipped PKGBUILD is now compiled in CI so one that cannot build cannot ship.
  • Prefetch telemetry, allocator telemetry, and a quieter log at default verbosity.

See the CHANGELOG for the complete list.

Reporting

xearthlayer diagnostics now completes on any cache size, so please include its output. The log at ~/.xearthlayer/xearthlayer.log is overwritten on every launch, so copy it aside before relaunching if you hit something worth reporting.

Assets 7
Loading