[CI][Python] Broken PyArrow install on some crossbow jobs

[asfimport]

See example here:
https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=18133&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181

============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/.hypothesis/examples')
rootdir: /
plugins: lazy-fixture-0.6.3, hypothesis-6.34.1
collecting ... collected 0 items

============================ no tests ran in 0.48s =============================
ERROR: module or package not found: pyarrow (missing __init__.py?)

Reporter: Antoine Pitrou / @pitrou
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #12069

_{Note: This issue was originally created as ARROW-15243. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
Issue resolved by pull request 12069
#12069