Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workaround for _convert_to_request_dict change #1083

Merged
merged 4 commits into from Oct 25, 2022

Conversation

ikonst
Copy link
Contributor

@ikonst ikonst commented Oct 25, 2022

botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.

P.S. we would probably need to backport this to 5.x.

@ikonst ikonst changed the title Workaround for botocore change Workaround for _convert_to_request_dict change Oct 25, 2022
@garrettheel
Copy link
Member

Worth noting that #1079 will also fix this particular issue (by relying on less private APIs, but not all)

Copy link
Member

@garrettheel garrettheel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, and +1 to backporting to a 5.x release (I'd rather not ship 6.0 yet)

@DeepSpace2
Copy link

@ikonst @garrettheel

Thanks for the quick fix.

I'd love to get this as a 5.2.2 hotfix rather than waiting to 6.0.0.

@ikonst ikonst merged commit 95eb24e into master Oct 25, 2022
@ikonst ikonst deleted the 2022-10-25-_convert_to_request_dict branch October 25, 2022 15:26
ikonst added a commit that referenced this pull request Oct 25, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Oct 25, 2022
In #1083 we've started passing an `endpoint_url` parameter to _convert_to_request_dict due to changes made in botocore 1.28.

When a model does not specify a `host`, the `endpoint_url` would be `None`. To determine the actual `endpoint_url` in botocore ≥1.28, we must call another private method, `_resolve_endpoint_ruleset`.
ikonst added a commit that referenced this pull request Oct 25, 2022
In #1083 we've started passing an `endpoint_url` parameter to _convert_to_request_dict due to changes made in botocore 1.28.

When a model does not specify a `host`, the `endpoint_url` would be `None`. To determine the actual `endpoint_url` in botocore ≥1.28, we must call another private method, `_resolve_endpoint_ruleset`.
ikonst added a commit that referenced this pull request Oct 25, 2022
In #1083 we've started passing an `endpoint_url` parameter to _convert_to_request_dict due to changes made in botocore 1.28.

When a model does not specify a `host`, the `endpoint_url` would be `None`. To determine the actual `endpoint_url` in botocore ≥1.28, we must call another private method, `_resolve_endpoint_ruleset`.
@ikonst ikonst mentioned this pull request Nov 2, 2022
@dmulter
Copy link

dmulter commented Nov 2, 2022

Would love to see this backported to 4.x as a 4.3.4 release.

archlinux-github pushed a commit to archlinux/svntogit-community that referenced this pull request Nov 8, 2022
As a side note, 5.2.3 fixes compatibility with python-botocore >= 1.28.0 [1]

[1] pynamodb/PynamoDB#1083

git-svn-id: file:///srv/repos/svn-community/svn@1345925 9fca08f4-af9d-4005-b8df-a31f2cc04f65
archlinux-github pushed a commit to archlinux/svntogit-community that referenced this pull request Nov 8, 2022
As a side note, 5.2.3 fixes compatibility with python-botocore >= 1.28.0 [1]

[1] pynamodb/PynamoDB#1083


git-svn-id: file:///srv/repos/svn-community/svn@1345925 9fca08f4-af9d-4005-b8df-a31f2cc04f65
@ikonst ikonst mentioned this pull request Nov 25, 2022
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
@ikonst
Copy link
Contributor Author

ikonst commented Dec 7, 2022

@dmulter see #1130. Had to dust off a bunch of things for this, but at least now we have a 4.x branch that passes CI.

ikonst added a commit that referenced this pull request Dec 7, 2022
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
@ikonst
Copy link
Contributor Author

ikonst commented Dec 7, 2022

4.x backport should be available at https://pypi.org/project/pynamodb/4.4.0/

@ikonst ikonst mentioned this pull request Jan 27, 2023
@MirzaBaig715
Copy link

MirzaBaig715 commented Apr 17, 2023

@ikonst I am getting this error _convert_to_request_dict() missing 1 required positional argument: 'endpoint_url' with dynamodb==5.0.3 botocore==1.20.112.

@dobeerman
Copy link

@ikonst I am getting this error _convert_to_request_dict() missing 1 required positional argument: 'endpoint_url' with dynamodb==5.0.3 botocore==1.20.112.

I had the same issue with one lambda containerized. I used aws public image for the lambda with the tag public.ecr.aws/lambda/python:3.8. Recently AWS rolled out the update which also includes update for botocore to v1.29.xxx.
I simply changed the image to public.ecr.aws/lambda/python:3.8.2023.02.20.11 and now it works.

Also, if you do install requirements in this container, make sure you use target in the command RUN pip3 install -r requirements.txt --target ${LAMBDA_TASK_ROOT} which will also override the preinstalled botocore (if its included into the requirements)

@ikonst
Copy link
Contributor Author

ikonst commented Apr 17, 2023

Are you suggesting that recent pynamodb doesn't work well with old botocore 1.20?

@MirzaBaig715
Copy link

MirzaBaig715 commented Apr 17, 2023

@ikonst I am getting this error _convert_to_request_dict() missing 1 required positional argument: 'endpoint_url' with dynamodb==5.0.3 botocore==1.20.112.

I had the same issue with one lambda containerized. I used aws public image for the lambda with the tag public.ecr.aws/lambda/python:3.8. Recently AWS rolled out the update which also includes update for botocore to v1.29.xxx. I simply changed the image to public.ecr.aws/lambda/python:3.8.2023.02.20.11 and now it works.

Also, if you do install requirements in this container, make sure you use target in the command RUN pip3 install -r requirements.txt --target ${LAMBDA_TASK_ROOT} which will also override the preinstalled botocore (if its included into the requirements)

@dobeerman I am using lambda containerization with this image lambci/lambda:build-python3.8 and there is no update in image. I had to upgrade the pynamodb 5.0.3 => 5.2.3 and now it's working fine.

@bersek36
Copy link

@ikonst I am getting this error _convert_to_request_dict() missing 1 required positional argument: 'endpoint_url' with dynamodb==5.0.3 botocore==1.20.112.

I had the same issue with one lambda containerized. I used aws public image for the lambda with the tag public.ecr.aws/lambda/python:3.8. Recently AWS rolled out the update which also includes update for botocore to v1.29.xxx. I simply changed the image to public.ecr.aws/lambda/python:3.8.2023.02.20.11 and now it works.

Also, if you do install requirements in this container, make sure you use target in the command RUN pip3 install -r requirements.txt --target ${LAMBDA_TASK_ROOT} which will also override the preinstalled botocore (if its included into the requirements)

Thanks a lot. This fixed my bug.

thanakijwanavit pushed a commit to thanakijwanavit/PynamoDB that referenced this pull request May 11, 2023
botocore 1.28 changed the signature of private method botocore.client.BaseClient._convert_to_request_dict adding an endpoint_url parameter. We are updating pynamodb to inspect the signature and add this parameter as needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

boto3 update broke pynamodb botocore 1.28.0 expects endpoint_url param to _convert_to_request_dict()
7 participants