Skip to content

Cross-platform compatibility fixes for Windows and macOS - #43

Merged
mmacferrin merged 5 commits into
mainfrom
cross-platform-shell-deps
Jul 17, 2026
Merged

Cross-platform compatibility fixes for Windows and macOS#43
mmacferrin merged 5 commits into
mainfrom
cross-platform-shell-deps

Conversation

@mmacferrin

Copy link
Copy Markdown
Collaborator

Summary

Fixes a set of cross-platform compatibility issues so IVERT runs on modern Windows and macOS as well as Linux. Each fix preserves existing Linux behavior. No functional change on Linux.

Fixes

  • Config path resolution (configfile.py) — Path detection was gated on sys.platform: the Windows branch only recognized values containing a backslash, but the .ini files use forward slashes (e.g. ~/.ivert). On Windows those values fell through unhandled, leaving ~ unexpanded. Replaced the two platform-specific branches with one platform-agnostic branch (treats a value as a path if it contains ~, /, or \) plus an _is_absolute_path() helper that recognizes Unix, Windows drive, and UNC roots regardless of OS. On Windows ~/.ivert now resolves to C:\Users\<user>\.ivert.

  • signal.SIGKILL guard (validate_dem.py)signal.SIGKILL does not exist on Windows, so abs(exitcode) == abs(signal.SIGKILL) raised AttributeError whenever a validation subprocess exited non-zero. Look the attribute up with getattr and skip the OOM-killer recovery branch when it is absent (Windows has no SIGKILL / OOM-killer); a failed worker now surfaces as the intended RuntimeError.

  • Cross-platform terminal colors (bcolors.py) — Raw ANSI escape codes render on Linux/macOS but appear as literal garbage on Windows consoles, where virtual-terminal processing is off by default. Enable it on import via colorama.just_fix_windows_console() (no-op elsewhere; only sets the console-mode flag, does not wrap sys.stdout, so it does not interfere with loggerproc.Logger's stdout redirection). colorama added as a Windows-only dependency; the import is defensive so Linux/macOS degrade to native ANSI.

  • Multiprocessing start method (validate_dem.py, validate_dem_collection.py)set_start_method("spawn") raises RuntimeError if the start method was already set. Pass force=True so re-invocation cannot crash.

  • Explicit UTF-8 encoding — Text-mode open() defaults to the platform locale encoding (e.g. cp1252 on Windows). Pass encoding="utf-8" on all text-mode opens (empty-result markers, summary stats, user config, photon-tile lists, the AWS datasource probe, and the process logger). Binary opens are unchanged.

Testing

Verified on Linux: config paths still resolve to the correct absolute home-directory locations; the SIGKILL decision logic was unit-checked across POSIX/Windows exit-code cases; bcolors imports and renders (VT-enable is a no-op on Linux and degrades gracefully if colorama is absent); set_start_method(force=True) is idempotent; and no text-mode open() without an encoding remains. ruff check and ruff format pass on all changed files.

Not covered here (requires real Windows/macOS hardware): the shared-memory + spawn parallel-validation path in validate_dem.py — Windows frees named SharedMemory segments when the last handle closes, unlike Linux /dev/shm, so that flow should be exercised directly on the target platform.

Path detection in Config._read_option was gated on sys.platform: the
Windows branch only recognized values containing a backslash, but the
.ini config files use forward slashes (e.g. ~/.ivert). On Windows those
values fell through path handling entirely, leaving ~ unexpanded.

Replace the two platform-specific branches with a single branch that
treats a value as a path if it contains ~, /, or \, and add an
_is_absolute_path helper recognizing Unix, Windows drive, and UNC roots
regardless of the running OS.
signal.SIGKILL does not exist on Windows, so evaluating
abs(exitcode) == abs(signal.SIGKILL) raised AttributeError whenever a
validation subprocess exited non-zero. Look the attribute up with
getattr and skip the OOM-killer recovery branch when it is absent:
Windows has no SIGKILL and no equivalent OOM-killer, so a failed worker
now surfaces as the intended RuntimeError instead of crashing. POSIX
behavior is unchanged.
The bcolors utility emitted raw ANSI escape codes, which render on Linux
and macOS but not on Windows consoles, where virtual-terminal processing
is off by default and the codes appear as literal garbage.

Enable ANSI/VT processing on import via colorama.just_fix_windows_console(),
which targets modern Windows terminals and is a no-op elsewhere. It only
sets the console-mode flag and does not wrap sys.stdout, so it does not
interfere with loggerproc.Logger's stdout redirection (which already
strips these codes from log files). The import is defensive so Linux/macOS,
where colorama is unnecessary, degrade to native ANSI rendering.

Add colorama as an explicit Windows-only dependency.
mp.set_start_method('spawn') raises RuntimeError if the start method was
already set in the process. Pass force=True in both validation entry
points so re-invocation (or both entry points running in one process)
cannot crash.
Text-mode open() defaults to the platform locale encoding (e.g. cp1252
on Windows) rather than utf-8, which can corrupt non-ASCII content and
differ across platforms. Pass encoding='utf-8' explicitly on all
text-mode opens (empty-result markers, summary stats, user config,
photon-tile lists, the AWS datasource probe, and the process logger).

Binary opens in pickle_blosc.py are left unchanged.
@mmacferrin

Copy link
Copy Markdown
Collaborator Author

Issue #10 progress (refs #10 — does not close it)

This PR is part of the ongoing cross-platform work for #10. It lands the code-level fixes; the issue stays open pending on-hardware testing.

Done in this PR (5 fixes, Linux behavior unchanged):

  • Config ~/path resolution works on Windows (configfile.py no longer gates path detection on sys.platform)
  • signal.SIGKILL guarded so validate_dem.py no longer raises AttributeError on Windows
  • Terminal colors render on Windows consoles (colorama.just_fix_windows_console()), colorama added as a Windows-only dep
  • mp.set_start_method("spawn", force=True) so re-invocation can't raise RuntimeError
  • Explicit encoding="utf-8" on all text-mode open() calls (avoids cp1252 on Windows)

Still open under #10 (not in this PR):

  • Exercise the shared-memory + spawn parallel-validation path on real Windows/macOS — Windows frees named SharedMemory segments when the last handle closes (unlike Linux /dev/shm), so this needs to be run on the target platform rather than fixed blind
  • Manual testing: macOS (Intel + Apple Silicon), native Windows / WSL2, CentOS/RHEL/Rocky
  • Cross-platform GDAL/h5py install differences; AWS credential-path variations
  • CI/CD matrix across platforms; document supported platforms

Reminder: keep closing keywords away from #10 in the eventual merge commit/PR — GitHub ignores negation and auto-closed it once already.

@mmacferrin
mmacferrin merged commit d7d176b into main Jul 17, 2026
3 checks passed
@mmacferrin
mmacferrin deleted the cross-platform-shell-deps branch July 17, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant