Skip to content

fix(kibana): fail-fast on permanent errors in readiness check #173

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Oddly merged 1 commit into from
Aug 4, 2026
Merged

fix(kibana): fail-fast on permanent errors in readiness check #173

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions roles/kibana/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,12 +143,20 @@
- name: Wait for Kibana to be ready
when: not ansible_check_mode
block:
- name: Wait for Kibana readiness ({{ 'HTTPS' if kibana_tls | default(false) | bool else 'HTTP' }})

Check failure on line 146 in roles/kibana/tasks/main.yml

View workflow job for this annotation

GitHub Actions / lint

risky-shell-pipe 

Shells that use pipes should set the pipefail option.

Check failure on line 146 in roles/kibana/tasks/main.yml

View workflow job for this annotation

GitHub Actions / lint_kibana / lint

risky-shell-pipe 

Shells that use pipes should set the pipefail option.
ansible.builtin.shell:
# Exit codes:
# 0 ready (200 or 401)
# 1 transient — still starting up, keep retrying
# 2 permanent — service dead or a fatal log line already surfaced
cmd: |
if ! systemctl is-active --quiet kibana; then
exit 2
fi
if journalctl -u kibana --no-pager -n 100 --since '2 minutes ago' 2>/dev/null | \
grep -Eq 'FATAL|Unable to connect to Elasticsearch|Authentication.*failed|Kibana server is not ready.*fatal'; then
exit 2
fi
HTTP_CODE=$(curl -sk -o /dev/null -w '%{http_code}' {{ 'https' if kibana_tls | default(false) | bool else 'http' }}://localhost:5601/api/status 2>/dev/null) || true
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "401" ]; then
exit 0
Expand Down Expand Up @@ -186,4 +194,4 @@
Recent log output:
{{ _kibana_wait_journal.stdout }}
when: _kibana_wait_result.rc | default(0) != 2

Check warning on line 197 in roles/kibana/tasks/main.yml

View workflow job for this annotation

GitHub Actions / lint_kibana / lint 

197:1 [empty-lines] too many blank lines (1 > 0)
15 changes: 14 additions & 1 deletion roles/kibana/tasks/restart_and_verify_kibana.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,21 @@
retries: 5
delay: 3

- name: restart_and_verify_kibana | Wait for Kibana readiness ({{ 'HTTPS' if kibana_tls | default(false) | bool else 'HTTP' }})

Check failure on line 24 in roles/kibana/tasks/restart_and_verify_kibana.yml

View workflow job for this annotation

GitHub Actions / lint

risky-shell-pipe 

Shells that use pipes should set the pipefail option.

Check failure on line 24 in roles/kibana/tasks/restart_and_verify_kibana.yml

View workflow job for this annotation

GitHub Actions / lint_kibana / lint

risky-shell-pipe 

Shells that use pipes should set the pipefail option.
ansible.builtin.shell:
# Exit codes:
# 0 ready (200 or 401)
# 1 transient — still starting up, keep retrying
# 2 permanent — service died or a fatal log line already surfaced,
# no point in waiting the remaining retries
cmd: |
if ! systemctl is-active --quiet kibana; then
exit 2
fi
if journalctl -u kibana --no-pager -n 100 --since '2 minutes ago' 2>/dev/null | \
grep -Eq 'FATAL|Unable to connect to Elasticsearch|Authentication.*failed|Kibana server is not ready.*fatal'; then
exit 2
fi
HTTP_CODE=$(curl -sk -o /dev/null -w '%{http_code}' {{ 'https' if kibana_tls | default(false) | bool else 'http' }}://localhost:5601/api/status 2>/dev/null) || true
if [ "$HTTP_CODE" = "200" ] || [ "$HTTP_CODE" = "401" ]; then
exit 0
Expand All @@ -34,6 +46,7 @@
retries: 60
delay: 5
changed_when: false
failed_when: _kibana_http_check.rc == 2

rescue:
- name: restart_and_verify_kibana | Get recent Kibana journal output
Expand All @@ -45,7 +58,7 @@
- name: restart_and_verify_kibana | Fail with Kibana startup diagnostics
ansible.builtin.fail:
msg: |
Kibana failed to start.
Kibana failed to start{% if _kibana_http_check.rc | default(0) == 2 %} (permanent error detected — see log){% else %} (HTTP did not become ready within 300s){% endif %}.

Recent log output:
{{ _kibana_journal.stdout }}
Loading