Kubespray cannot download file: FAILED - RETRYING: Download_file | Validate mirrors

[acastaneda-cv5]

Kubespray 2.23 in RHEL 8.9

TASK [container-engine/runc : Download_file | Create dest directory on node] ***********************************************************************
ok: [s00plxcxxx7]
ok: [s00plxcxxx5]
ok: [s00plxcxxx8]
Tuesday 26 December 2023 15:37:29 -0500 (0:00:03.464) 0:03:52.155 ******
Tuesday 26 December 2023 15:37:29 -0500 (0:00:00.042) 0:03:52.197 ******
Tuesday 26 December 2023 15:37:29 -0500 (0:00:00.054) 0:03:52.252 ******
FAILED - RETRYING: [s00plxcxxx7]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx5]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx8]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx7]: Download_file | Validate mirrors (1 retries left).
FAILED - RETRYING: [s00plxcxxx5]: Download_file | Validate mirrors (1 retries left).
FAILED - RETRYING: [s00plxcxxx8]: Download_file | Validate mirrors (1 retries left).

TASK [container-engine/runc : Download_file | Validate mirrors] ************************************************************************************
failed: [s00plxcxxx7] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx7 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
failed: [ss00plxcxxx8] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx8 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
failed: [s00plxcxxx5] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx5 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
FAILED - RETRYING: [s00plxcxxx7]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx5]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx8]: Download_file | Validate mirrors (4 retries left).
FAILED - RETRYING: [s00plxcxxx7]: Download_file | Validate mirrors (1 retries left).
FAILED - RETRYING: [s00plxcxxx5]: Download_file | Validate mirrors (1 retries left).
FAILED - RETRYING: [s00plxcxxx8]: Download_file | Validate mirrors (1 retries left).
failed: [s00plxcxxx7] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx7 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
failed: [s00plxcxxx8] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx8 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
failed: [s00plxak8s119] (item=None) => {"attempts": 4, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxak8s119 -> {{ download_delegate if download_force_cache else inventory_hostname }}]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring
Tuesday 26 December 2023 15:43:30 -0500 (0:06:00.193) 0:09:52.445 ******

TASK [container-engine/runc : Download_file | Get the list of working mirrors] *********************************************************************
ok: [s00plxcxxx7]
ok: [s00plxcxxx5]
ok: [s00plxcxxx8]
Tuesday 26 December 2023 15:43:31 -0500 (0:00:01.215) 0:09:53.661 ******

TASK [container-engine/runc : Download_file | Download item] ***************************************************************************************
fatal: [s00plxcxxx7]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
fatal: [s00plxcxxx8]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

[ErikJiang]

https://github.com/kubernetes-sigs/kubespray/blob/master/inventory/sample/group_vars/all/all.yml#L135-L136

you can try using the unsafe_show_logs parameter for more detailed logs.

[prairiewolf-by]

We have such kind of output when enabling debug logs:
"msg": "The conditional check ''OK' in get_url_result.msg or 'file already exists' in get_url_result.msg or get_url_result.status_code == 304' failed. The error was: error while evaluating conditional ('OK' in get_url_result.msg or 'file already exists' in get_url_result.msg or get_url_result.status_code == 304): 'dict object' has no attribute 'status_code'. 'dict object' has no attribute 'status_code'"
As per discussin here - #10592 - that is regression against kubespray 2.22.1. Could that be fixed in 2.23.2, or the revert is the only way?

[RomainMou]

I think this should be fix in master with the merge of #10613.

[VannTen]

/close
(Per comment above, the PR also has been backported to release-2.23 and will be in 2.23.2)

[k8s-ci-robot]

@VannTen: Closing this issue.

In response to this:

/close
(Per comment above, the PR also has been backported to release-2.23 and will be in 2.23.2)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.