Sms voice fix #3640
Sms voice fix #3640
Conversation
| @@ -12,6 +12,8 @@ | |||
| # language governing permissions and limitations under the License. | |||
| import os | |||
|
|
|||
| from botocore.docs import DEPRECATED_SERVICE_NAMES | |||
There was a problem hiding this comment.
I'm slightly leery about this if the user somehow ends up with an older version of Botocore on their system, but because it's constrained to the docs module we're probably alright.
There was a problem hiding this comment.
My thought process was that it would be better to keep track of one list of services. I don't expect this list to grow much or often so it shouldn't be an issue maintaining two lists if we think thats a better approach.
There was a problem hiding this comment.
Yeah, it's the right impulse. We've just had issues in the past where uses managed to get "compatible" versions that were outside of our compatibility range. So this may not exist in their version of Botocore. It's not something we can really control, these types of changes just tend to generate some noise.
There was a problem hiding this comment.
This caused things to blow up for us today - even though botocore 1.29.102 was installed
The relevant parts of the log for where boto3 was installed
Collecting boto3
Downloading boto3-1.26.102-py3-none-any.whl (135 kB)
...
Collecting botocore<1.30.0,>=1.29.102
Downloading botocore-1.29.102-py3-none-any.whl (10.6 MB)
...
Successfully installed boto3-1.26.102 botocore-1.29.102 python-dateutil-2.8.2 s3transfer-0.6.0 urllib3-1.26.15
But then
Traceback (most recent call last):
File "/mnt/tmp/spark-11c69845-a7ea-422f-a23d-8b07fb776b40/spark_sql_submit_template.py", line 79, in <module>
run_query(args.query_location, args.cluster_id, args.hash_key, args.metastore_uris, args.s3_bucket)
File "/mnt/tmp/spark-11c69845-a7ea-422f-a23d-8b07fb776b40/spark_sql_submit_template.py", line 26, in run_query
obj = get_obj_s3(s3_url_parsed.netloc, s3_url_parsed.path.lstrip('/'))
File "/mnt/tmp/spark-11c69845-a7ea-422f-a23d-8b07fb776b40/spark_sql_submit_template.py", line 20, in get_obj_s3
import boto3
File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 17, in <module>
from boto3.session import Session
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 25, in <module>
from .resources.factory import ResourceFactory
File "/usr/local/lib/python3.7/site-packages/boto3/resources/factory.py", line 17, in <module>
from ..docs import docstring
File "/usr/local/lib/python3.7/site-packages/boto3/docs/__init__.py", line 15, in <module>
from botocore.docs import DEPRECATED_SERVICE_NAMES
ImportError: cannot import name 'DEPRECATED_SERVICE_NAMES' from 'botocore.docs' (/usr/local/lib/python3.7/site-packages/botocore/docs/__init__.py)
There was a problem hiding this comment.
Hey @snolan-amount,
Would you be able to print your botocore version before the import/call here so we can verify this is using the correct version of botocore? There may be an installation somewhere else that overwrites the one initially installed. I cant reproduce this with botocore 1.29.102.
Can you confirm the botocore version imported in your script by adding:
import botocore
print(botocore.__version__)
There was a problem hiding this comment.
My team also has some Spark jobs with this ImportError in production today. Similar stack trace as @snolan-amount documented. I wonder if that's coincidence or if Spark is specifically impacted by this change in some way.
[Follow up after investigating]
Looks like we will resolve this by pinning a boto3 version in the script that bootstraps our spark workers. A later pip package install (s3fs in our case) brings an old version of botocore that breaks boto3.
# Old: results in an incompatible combination of boto3 and botocore
pip install boto3
pip install s3fs
# New command that appears to leave a working environment
pip install "boto3<=1.26.100"
pip install s3fs
There was a problem hiding this comment.
@DavidWAbrahams I'm not very familiar with Spark but I don't think it's specifically impacted by this change. There may be a package that is commonly used with Spark that forces the install of an older/incompatible version of botocore resulting in this issue. As mentioned before, you should check your botocore version to verify which one you have installed at the time you import it.
There was a problem hiding this comment.
As others have said in the issue regarding this #3648 the problem seems to be with a dependency in aiobotocore, which is used by pandas to write to S3 (among others I'm sure) - in our case we had the problem in both Airflow and EMR, it presented a little differently in each.
We have been able to get things working by pinning older versions that aiobotocore works with (which in our case is much older than 1.26.101 - we're stuck on 1.21.29 on Airflow right now but it's working
/usr/local/lib/python3.7/site-packages/boto3
/usr/local/lib/python3.7/site-packages/boto3-1.21.29.dist-info
/usr/local/lib/python3.7/site-packages/botocore
/usr/local/lib/python3.7/site-packages/botocore-1.24.29.dist-info
On EMR we pinned to 1.26.101 and it worked.
Companion to boto/botocore#2894