fix: stop salt-api FD exhaustion + drive masters badge from config#705
Merged
Conversation
Changed the SALT_API_URL in multiple playbooks and documentation to reflect the new default port 4507 instead of 8080. This includes updates in install_salt_master.yml, install_salt_master_linux.yml, deploy_salt_master_mm1.yml, and the README.md for the salt_master role. Ensures consistency in configuration and avoids port collisions.
…d added new communities. Updated report to reflect changes in node and edge counts, and adjusted the date to 2026-06-21.
salt-api on macOS inherits launchd's 256 soft open-file limit, which its cheroot server exhausts under TLS connection churn — accept() then fails with [Errno 24] Too many open files, so every /login and /run returns HTTP 500 and new TLS handshakes drop as UNEXPECTED_EOF, flipping masters to "unreachable" on the dashboard. Add SoftResourceLimits/HardResourceLimits NumberOfFiles (salt_api_max_open_files=10240) to the salt-api launchd plist so the service stays serviceable. Also rewrite SaltMasterStatus to read the configured SaltMaster row — whose api_url is derived server-side from address + salt_api_port + use_tls — and show the backend-computed health, instead of probing a hardcoded port 4507 against a non-existent /settings/probe endpoint. Co-authored-by: Cursor <cursoragent@cursor.com>
master's required checks were already failing before this branch, blocking any merge. None relate to the salt-api change; all are drift between evolved code and stale assertions: - ruff: sort import block in baselines.py:create_baseline - mypy: coerce tracing resolved_service to str (was str | None in dict value) - test_promote_topology_560: master api_url is now :4507, not :8080 - test_toast_dismiss_timing: error toasts auto-dismiss at 8000ms (#688 superseded #159's never-dismiss) — assertion updated, not the code - test_workers_bugs: scope the maintenance_mode grep to the whole sync_minion_presence function (it grew past 3000 chars with #689) - test_drift_task: mock the extra Node lookup find_baseline_for_node_sync now issues to derive os_family - test_llm_assistant_history_fix: assert streamQuery(prompt, history) after the mutation->stream refactor; the #303 history-before-addMessage guarantee is unchanged Verified locally: 2770 unit tests pass, services coverage 84.11%, ruff+mypy clean. Co-authored-by: Cursor <cursoragent@cursor.com>
Coverage Report✅ 84.2% on Gate passed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
accept()fails with[Errno 24] Too many open files— every/loginand/runthen returns HTTP 500, and new TLS handshakes drop asUNEXPECTED_EOF_WHILE_READING. Masters flip to unreachable on the dashboard. AddedSoftResourceLimits/HardResourceLimitsNumberOfFiles(salt_api_max_open_files=10240, configurable) to the salt-api launchd plist.SaltMasterStatusnow reads the configuredSaltMasterrow (itsapi_urlis derived server-side fromaddress + salt_api_port + use_tls) and shows the backend-computed health, instead of probing a literal port4507against a non-existent/settings/probeendpoint.Why this came up
Diagnosed live on the
mmmaster (192.168.1.64): salt-api log showedOSError: [Errno 24] Too many open files. Restarting cleared it; raising the FD limit makes it durable. After the fix the probe reportshealthy(all checks pass, 1 minion up).Notes
192.168.1.64has already been patched + reloaded (maxfiles soft/hard = 10240); this PR makes the same change reproducible via the role for any re-provision.Test plan
launchctl procinfo <pid>showsmaxfiles => 10240.Made with Cursor