Boto3 hangs under mod_wsgi on Fedora when awscli2 RPM is installed

[karlcz]

Describe the bug

We encountered a very confusing malfunction in boto3 when used inside a web app hosted behind Apache httpd + mod_wsgi. On a Fedora 38 system where the awscli2 RPM was installed, boto3 would hang and never send http requests when called from within the web service, but it would work as expected when called in a Python REPL from the same daemon account.

Even this short STS test would hang:

import boto3

sts = boto3.Session().client('sts') # this works
sts.get_caller_identity() # this hangs

and the same was true of S3 calls.

Expected Behavior

We expect boto3 client operations to communicate with AWS as per documentation.

Current Behavior

When awscli2 is installed, the boto3 calls would hang before sending an http request. With DEBUG logging enabled, the last event before the hang would be like:

botocore.hooks [DEBUG] Event choose-signer.sts.GetCallerIdentity: calling handler <function set_operation_specific_signer at 0x7f0739bb67a0>

It never proceeded to the "sending http request" log event.

Reproduction Steps

On a Fedora 38 x86_64 system, doing yum install awscli2 python3-boto3 python3-mod_wsgi httpd will install the broken parts:

• awscli2- 2.11.18-1.fc38
• python3-awscrt-0.16.19-1.fc38
• python3-botocore-1.29.111-1.fc38
• python3-boto3-1.26.111-1.fc38
• httpd-2.4.57-1.fc38
• python3-mod_wsgi-4.9.4-2.fc38

Via pip freeze we would see these versions:

awscli==2.11.18
awscrt==0.16.19
...
boto3==1.26.111
botocore==1.29.111
...
mod-wsgi==4.9.4

We run mod_wsgi in daemon mode with a configuration like:

WSGIPythonOptimize 1
WSGIDaemonProcess foobar processes=4 threads=1 user=foobar maximum-requests=2000
WSGIScriptAlias /foobar /usr/local/lib/python3.11/site-packages/foobar/foobar.wsgi
WSGIPassAuthorization On
    
WSGISocketPrefix /var/run/httpd/wsgi
    
<Location /foobar>
    
    WSGIProcessGroup foobar
        
</Location>

Possible Solution

As a workaround, merely uninstalling with yum erase awscli2 removed both the awscli and awscrt RPMs. After that, the web service code would work as expected and boto3 calls would initiate http requests to AWS and succeed.

Additional Information/Context

We tried su --group apache - foobar to switch to the daemon account and run manual tests. From there, awscli calls worked and a manual test of boto3 via a python REPL also worked.

We also tried with the boto3 library from PyPI instead of from Fedora's python3-boto3 RPM. The same hanging failure mode was observed with both when run inside our mod_wsgi service environment but not when run via REPL.

In conclusion, we imagine that the way the Python interpreter is initialized inside mod_wsgi daemon processes must be different than when launched at the CLI, exposing this broken interaction between the different installed libraries. But, we're not familiar enough with the internals of boto3 to help diagnose this further...

SDK version used

boto3==1.26.111

Environment details (OS name and version, etc.)

Fedora 38 x86_64

[indrora]

Does this replicate in a virtualenv?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[karlcz]

Hmm, I'm not sure how to test with a virtualenv as it is in the Python interpreter environment under mod_wsgi.

[karlcz]

This wasn't with a virtualenv, but I did just to a test where I removed the Fedora RPMs and installed everything via pip. This does not seem to exhibit the same hangs as with the RPM versions I listed in the original issue

That may be because "awscli" from PyPI is the older version 1 branch (1.27.156). On looking into this, now I realize that version 2 isn't even published there. So perhaps the issue is that Fedora has packaged awscli2 and boto into RPMs in a way that is fragile...?

[indrora]

I don't know how Fedora is packaging their version of the CLIv2. Packaging in Python is a bit of a mess.

And you're correct: There is no official AWS CLIv2 package in PyPI for reasons (version pinning, ease of use, etc). The official (binary) distribution of the CLIv2 is distributed with an independent version of Python 3 and Boto3 that is unaffected by Pip or local Python installations.

I would suggest filing an issue with the Fedora bug tracker on this.

Closing since it appears to be an issue with Fedora.