Add timeout and retry when installing baseline dependencies.

[HanFa]

What this PR does / why we need it:

The image-builder hits the ansible issue ansible/ansible#51663 when it's trying to install baseline dependencies with apt module. Here we need lock timeout and retry unless there is a better way.

Which issue(s) this PR fixes (optional, in fixes #(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

Additional context
Add any other context for the reviewers

[k8s-ci-robot]

Welcome @HanFa!

It looks like this is your first PR to kubernetes-sigs/image-builder 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/image-builder has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @HanFa. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[HanFa]

cc @randomvariable could you help take a look?

[randomvariable]

/lgtm
/approve

[kkeshavamurthy]

/ok-to-test
/hold

[kkeshavamurthy]

Seems like Azure Ubuntu builds are failing with

vhd-ubuntu-2004: TASK [providers : Install iptables persistence] ********************************�[0m
�[0;32m    vhd-ubuntu-2004: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Connection to 127.0.0.1 closed.�[0m
�[0;32m    vhd-ubuntu-2004: fatal: [default]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.apt', init_globals=dict(_module_fqn='ansible.modules.apt', _modlib_path=modlib_path),\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_payload_es_wx2_h/ansible_apt_payload.zip/ansible/modules/apt.py\", line 327, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_apt_payload_es_wx2_h/ansible_apt_payload.zip/ansible/module_utils/urls.py\", line 115, in <module>\n  File \"/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py\", line 46, in <module>\n    import OpenSSL.SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/__init__.py\", line 8, in <module>\n    from OpenSSL import crypto, SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1553, in <module>\n    class X509StoreFlags(object):\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1573, in X509StoreFlags\n    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'\nError in sys.excepthook:\nTraceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/apport_python_hook.py\", line 72, in apport_excepthook\n    from apport.fileutils import likely_packaged, get_recent_crashes\n  File \"/usr/lib/python3/dist-packages/apport/__init__.py\", line 5, in <module>\n    from apport.report import Report\n  File \"/usr/lib/python3/dist-packages/apport/report.py\", line 32, in <module>\n    import apport.fileutils\n  File \"/usr/lib/python3/dist-packages/apport/fileutils.py\", line 12, in <module>\n    import os, glob, subprocess, os.path, time, pwd, sys, requests_unixsocket\n  File \"/usr/lib/python3/dist-packages/requests_unixsocket/__init__.py\", line 1, in <module>\n    import requests\n  File \"/usr/lib/python3/dist-packages/requests/__init__.py\", line 95, in <module>\n    from urllib3.contrib import pyopenssl\n  File \"/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py\", line 46, in <module>\n    import OpenSSL.SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/__init__.py\", line 8, in <module>\n    from OpenSSL import crypto, SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1553, in <module>\n    class X509StoreFlags(object):\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1573, in X509StoreFlags\n    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'\n\nOriginal exception was:\nTraceback (most recent call last):\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 100, in <module>\n    _ansiballz_main()\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 92, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/tmp/.ansible/ansible-tmp-1662575750.0617676-3134-222205226284473/AnsiballZ_apt.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.apt', init_globals=dict(_module_fqn='ansible.modules.apt', _modlib_path=modlib_path),\n  File \"/usr/lib/python3.8/runpy.py\", line 207, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.8/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.8/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_payload_es_wx2_h/ansible_apt_payload.zip/ansible/modules/apt.py\", line 327, in <module>\n  File \"<frozen importlib._bootstrap>\", line 991, in _find_and_load\n  File \"<frozen importlib._bootstrap>\", line 975, in _find_and_load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 655, in _load_unlocked\n  File \"<frozen importlib._bootstrap>\", line 618, in _load_backward_compatible\n  File \"<frozen zipimport>\", line 259, in load_module\n  File \"/tmp/ansible_apt_payload_es_wx2_h/ansible_apt_payload.zip/ansible/module_utils/urls.py\", line 115, in <module>\n  File \"/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py\", line 46, in <module>\n    import OpenSSL.SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/__init__.py\", line 8, in <module>\n    from OpenSSL import crypto, SSL\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1553, in <module>\n    class X509StoreFlags(object):\n  File \"/usr/lib/python3/dist-packages/OpenSSL/crypto.py\", line 1573, in X509StoreFlags\n    CB_ISSUER_CHECK = _lib.X509_V_FLAG_CB_ISSUER_CHECK\nAttributeError: module 'lib' has no attribute 'X509_V_FLAG_CB_ISSUER_CHECK'\nConnection to 127.0.0.1 closed.\r\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}�[0m
�[0;32m    vhd-ubuntu-2004:�[0m
�[0;32m    vhd-ubuntu-2004: PLAY RECAP *********************************************************************�[0m
�[0;32m    vhd-ubuntu-2004: default                    : ok=21   changed=15   unreachable=0    failed=1    skipped=124  rescued=0    ignored=0�[0m
�[0;32m    vhd-ubuntu-2004:�[0m

@jsturtevant any ideas?

[jsturtevant]

Is it failing consistently? This is adding a package via aptget:

image-builder/images/capi/ansible/roles/providers/tasks/azure.yml

Lines 33 to 41 in 2e84251

	- name: Install iptables persistence
	apt:
	name: "{{ packages }}"
	state: present
	force_apt_get: yes
	vars:
	packages:
	- iptables-persistent
	when: ansible_os_family == "Debian"

Not sure if it was a flake or something change with that package?

[jsturtevant]

/retest

[jsturtevant]

do we need these retrys in that task too?

[kkeshavamurthy]

Is it failing consistently? This is adding a package via aptget:

image-builder/images/capi/ansible/roles/providers/tasks/azure.yml

Lines 33 to 41 in 2e84251

	- name: Install iptables persistence
	apt:
	name: "{{ packages }}"
	state: present
	force_apt_get: yes
	vars:
	packages:
	- iptables-persistent
	when: ansible_os_family == "Debian"

Not sure if it was a flake or something change with that package?

Yes, this is failing very consistently.

[kkeshavamurthy]

do we need these retrys in that task too?

I suspect this is related to something in ansible/python (I might be wrong) rather than apt repo

[kkeshavamurthy]

@HanFa Can you pls rebase, the azure issue should be fixed.

[kkeshavamurthy]

/test pull-ova-all

[kkeshavamurthy]

/hold cancel

[kkeshavamurthy]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: HanFa, kkeshavamurthy, randomvariable

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• images/capi/OWNERS [kkeshavamurthy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment